Measurement and The Paper Factory

When I was a schoolboy, I used to catch the train to school. Every day I walked past a paper factory, or it might have been a warehouse (I don’t really know exactly what they made, I assume it was exercise books, big rolls of paper and stuff).

There was a big sign next to the front gate that proudly displayed the number of days since the last injury had occurred in the factory. It was actively maintained, so I would walk past one day and the sign would say “185 days since last injury,” then the next day it would say “186 days since last injury.”

One day I walked past and the sign had gone back down to zero. There had obviously been an injury onsite. I remember getting to the train station and mentioning it to some other kids, who had also been talking about it.

We started by joking about what injuries you could get in a paper factory and how different paper cuts might hurt people. We thought it was quite funny.

Then one kid asked:

Why would they need to have a big sign saying not to get injured anyway? Do people turn up at work deciding whether they want to get injured that day? Does someone really need to tell them not to get injured?

A kid on the train

We spoke about that on the train on the way to school but we never came to a conclusion about why adults working in a factory would need a big sign that reported on when people got injured.

Why not report number of pages of paper manufactured or something if you need a big sign at the front gate?

What do you think? Does it make sense to have a big sign reporting on the number of days since the last injury? If so, does it really belong at the front gate where anyone walking past can see it?

You can still buy signs like that – so people must see value in them:

Years later I attended business school and learned more about metrics and reporting. There is a lot of reporting and measurement that is pointless, but this particular example has some value.

Measures serve one of 3 goals:

• They enable a decision – such as “where should we deploy more resources?”;
• They are a visible reminder of something to influence behaviour – such as “remember safety is important, whatever your boss or friend tells you, safety is something we take seriously.”; or
• They are for marketing (sometimes called vanity metrics) – you report something purely to make yourself look or feel good. For example you might ask people to stop getting covid tests so you can show people that detected cases of covid are dropping, or you might report a huge velocity for your team to make people think you do heaps of work.

I think in the case of “days since last injury,” the goal is to reduce injuries through influencing behaviour. The idea, I believe, is that if everyone in the factory and even people walking past see that you are actively focused on safety, then people at work are more likely to think safety. Even more importantly, everyone has permission to argue against stupid suggestions that speed things up at the expense of safety.

Don’t stop to put on the harness you wimp – real men climb walls with no harness

Factory idiot

Sorry dude – there is literally a big sign saying you are wrong. Management will be pissed if I go and get injured so I am forced to be sensible.

Apprentice knowing they can argue back

It is not an Ad to come and work there, nor a random piece of advertising. It is a strong message that safety is important and not BS.

So the first thing to take away from this, I guess, is to remember that measurement and reporting do not just collect data – they send a message about what is important.

If you claim to be a “people focused” organisation, that cares about psychological safety and joy at work (and safety) yet the only things you measure are cycle time, velocity or profit, then people believe that that you only care about speed and money.

Similarly, if you measure velocity but not defects or technical debt, you are likely to get speed at the expense of quality even though, honestly, it is not what you wanted.

Great – but I also learned that the big sign had a downside – it could actually cause injuries.

I guess it could fall on someone’s head, or it could distract a driver who crashes their car as they come through the gate, but these are unlikely scenarios.

So – how can a sign that encourages safety and discourages risky behaviour actually lead to injuries?

Well – if you haven’t guessed, all metrics have a positive impact and also a potential negative one.

The sign discourages injuries – but, without any intent on anyone’s part, people can start to lose sight of the reason for the measure (reduce injuries) and start to focus on the metric itself.

What gets measured gets managed – but if people are managing the measure rather than the goal of the measure, then the measure itself becomes the target and it ceases to be be useful. Indeed it is actually harmful.

Let’s assume we are still in the old paper factory and that one of the crew got injured. But it is only a paper cut (or actually a small cut from a loose piece of machinery moving unexpectedly). The cut is minor and the injured party stops work to stop the bleeding.

As the injured party is stopping the bleeding, a co-worker comes over to check on her. They are both relieved that it is a pretty minor cut and then they discuss whether to report it. There is no loss of work or risk of someone being off … but if they do report it then the sign out the front of the factory will stop saying “186 days since last injury” and suddenly say “0 days.”

Everyone in the factory will ask what happened and our poor injured party will be very embarrassed. In fact it will be much worse than the original injury.

Our pair of workers decide not to report the injury and they get back to work. The same thing happens, in a similar way to several other people, none of whom were really hurt and none of whom want to be embarrassed. Then one day the piece of machinery flies loose and causes a major injury. This should have been predictable, since it had almost done so a dozen times.

I don’t think anyone was deliberately trying to cause an injury, but falsely believing that the metric needed to be achieved, rather than believing that each injury mattered, caused a major, predictable and avoidable injury.

Luckily I don’t work in a dangerous paper factory with regular paper cuts and flying pieces of machinery to worry about.

However – I have seen people mistake a metric for a goal and then manage the numbers while losing sight of the actual purpose they were originally focused on.

Keeping hospital admissions low is good in theory but not if it means reclassifying sick people as “will probably live.” Going live with a project that only has minor bugs in the code is good, but not if the bugs were reclassified from critical to minor (or closed without fixing) in order to achieve a low bug count.

Even some of my favourite metrics like cycle time can lead to trouble where people start rushing work, overestimating work or measuring “done” as “code complete and ready to integrate.”

So now days I always remember the jokes on the train as we discussed the idea that adults needed to be told to avoid paper cuts – and I also remember that what gets measured sends a message.

That message can be to improve behaviour or to do stupid things to make a nice number agree with what you hoped it would look like, while forgetting why you actually wanted to measure something in the first place.

Anyway – that is my story for today.