James King

I was thinking about metrics and estimation approaches this week. Interesting topics to be sure.

For reasons that are probably clear, I have decided that we need to keep reiterating what good metrics look like.

I guess the summary is that good metrics are the ones that help us learn from them in order to get good results. Bad metrics are ones that do not help us understand whether the work we are doing can be improved or whether we hit a goal.

I think most people agree with me. But then I was thinking about estimates. I have decided that there are two types of estimates:

• The estimate (quote) that someone gives to a customer. This is an estimate of time or price that includes contingency, expected profit etc
• The estimate we give to ourselves to plan our work.

The challenge is that when we give the estimate to stakeholders we are sometimes not sure which estimate it is:

• Is it a commitment to when we will finish, that includes contingency, buffers and so forth?
• Is it a planning tool that we use to prioritise our work AND to break our work down, flush out assumptions, debate who is best placed to do what, etc?

We should be confident of our commitments but it is still short of a guarantee.

In fact, both kinds of estimate should include our understanding of risks, our understanding of assumptions and a confidence indicator.

• Risks and assumptions will not just allow us to cover ourselves but they will also allow us to identify triggers to reassess our commitment. In other words points where we will become more or less confident.
• The risks and assumptions also drive how we plan our delivery with some of the work. Especially because the work people forget to estimate is about reducing the risks and validating the assumptions rather than building features.
• The confidence indicator would either be a confidence level (I am 55% confident this is right, I can be more confident if we …) or a confidence interval (It should be between a week and 2 months, I can be more confident if we …)

But for a commitment, we probably want to be explicit about when we will change the commitment based on changed assumptions and we need to be clear on our confidence.

Estimates as a planning tool are different though. We might start work with a bunch of assumptions to test and we might have a pretty wide confidence interval at the start of our work.

So mistaking an estimate as a planning tool for a commitment can lead to over-planning and longer, more expensive adventures. While not being clear when a commitment is made will lead to arguments when things change due to what the team thinks is clarified assumptions and the stakeholder thinks is a changed commitment.

Coming back to metrics though, estimates are useful where we have a baseline estimate and then compare it to what happens. This is the basis of story points and of some agile project milestones/releases/value points.

So if we include buffers and contingency in our estimate and then comparing this inflated guess to the outcome, then our metrics are skewed. The metrics will look good but not be as much use to use in learning how to do good work.

We will also find that the team think the inflated estimate is a goal and they will consume the contingency eagerly, without pushing back or revisiting things as they validate assumptions and find things changed.

I feel like I am half way through this thought bubble, but I am going to pause there anyway. The main point so far is to think about what you want to use your metric or estimate for before creating it.

If it is for a decision or a plan then be clear on what the need is and then spend the right amount of effort on the estimate to support the decision or plan.