Ranting and raving about agile metrics

I have been talking to people at work about measuring performance lately and it got me thinking about how we use metrics to understand our work in an agile context.

In some ways, results are results and impediments are impediments, so maybe the way we measure things is based on what we measure and not how we work. In other ways though, the way we work and even our attitude to work will impact what we measure, how we measure it and what we learn from it.

I often tell people that in agile teams we try to measure the system and not the person’s output or tasks, because we are seeking to find where the system we are in is impeding us from doing good work. But, I admit, we often track tasks and outcomes on a daily level on walls and in meetings.

So is measurement really different in agile teams, or is it basically more influenced by other factors?

Is the world really different, now that we are “agile”?

In my first job, quite some time ago, I reported on my work and my manager had to assess whether I was doing good work or not. Then my manager had to report on the work the whole team did and, I guess, people decided whether we did good work or not as a team.

I still see people doing this, even after we all went agile and changed the way we work. So I guess the first thing to ponder is – have we fundamentally and permanently changed the way we work, now that we are all agile?

A while ago I published a blog article where I suggested we had won the battle against “old ways of thinking,” but that we were now in danger of becoming the old guard. I even published some articles where I noticed that some people who I respect were moving away from “agile” (Due to losing site of craftsmanship, Mistaking people for tools, and even mistaking agile for what they used to do and talking about kangaroos).

My conclusion now is that we have changed the way we work, but perhaps we have not really finished the revolution, nor created a universally recognised awesome workplace. We still have work to do.

Based on this, I guess the way we measure things and the way we act on what we learn is still evolving too. Hopefully, as we measure our work, we change the way we work and then adapt our new measurement approaches to the new world.

Since the way we work is not yet perfect, the way we measure our work is not yet perfect either. That means we should still be questioning both the value and the results of our metics and our reporting.

James King – in this blog

So rather than viewing our current approach to measuring our work as mature and stable, we should continue to question and improve it. Maybe we need to measure our effectiveness at measuring 🙂

None of these articles point to real faults in the core principles of agile thinking though. Rather, they reflect my view that our goal should never be to “go agile for the sake of going agile,” but to “decide if an agile way of working will make things better for us and then exploit what will work for us here.”

So, for example, I am all for using SAFE, SCRUM or an organisational Way Of Working (WOW), while I am against thinking you have to implement any of these “correctly” rather than thinking you have implemented them as well as you could for now. You should not even aspire to ever implementing them perfectly in their current beautifully modeled state.

Ask not how your team can be become agile, ask how agile techniques and thinking can help your team create value more quickly and easily.

Me again

Blah blah – but what about metrics and measuring stuff?

OK – so I believe that agile is good because it helps us. I started this article by talking about how we measure our work and I should get back to talking about that.

Many agile teams measure cycle time, velocity and team happiness measures. Some use burn down charts and others use OKRs. These are all useful agile measures or approaches. But I do not think any team should use any of these tools approaches “because they are agile.”

Rather they should use these (or other) measures because they help someone make a decision (or nudge the right behaviour).

Start by asking what you want to achieve

One thing that I believe has not changed in my lifetime is that we should NEVER start with the question “what should we measure?” Instead we should start by asking “who will use what we measure, for what purpose?”

Just as it is important to understand the team’s goals before planning the team’s work, we should ask what the value we will get from any measurement and agree on specifically who will get that value. Understanding these things is what should guide us to decide what, specifically, we should measure, how we should measure it and how much effort we should bother to allocate to measuring it.

Agile reinforces this concept but did not invent it. The first principle in the Agile Manifesto is that we should “satisfy the customer through continuous delivery of value.”

(OK the manifesto says valuable software, but I am applying the same concept to metrics rather than code).

Continuous delivery of valuable information

Starting with the customer or starting with the end in mind is not specific to agile approaches.

However, continuous delivery and continuous learning are core to agile approaches. This would suggest that we should have ongoing measures (and adaptation based on what we learn) rather than quarterly reporting or only reviewing how we went after we launched a new product.

This also involves a different attitude to our work

One of the biggest shifts in thinking in the agile world is to move from “Plan well and execute well” to “sense and respond.”

This is not a subtle change and it is important. I might even devote another long article to the discussion. In short though:

  • Many people think like a tailor – “measure twice, cut once.” This is often a good strategy and it means that if you spend sufficient time planning and setting yourself up for success, then success will come from good planning and good execution of the plan. There will be some changes along the way, but basically you can (in theory) anticipate many of these and better clarify what needs to be done in the next weeks and months before you start rushing about without a plan.
  • Agile thinking rejects this sensible thought process. The agile approach is “responding to change,” which is more complex than it looks at first glance. The assumption here (I believe) is that we are at our most ignorant at the beginning of any adventure (Planning a quarter, replacing a legacy system, entering a relationship). Instead, we assume that we will learn more on the journey than we know at the start. If this is true, then we should not plan every step of the journey, but instead plan how we will learn faster and how we can identify the things we are certain we need no matter what, while still keeping our minds open on other things we might do or encounter. Success will therefore come from continually reassessing our progress and our changing environment and then acting on what we learn. Our plan is not a map of where we will go, but rather a guide to how we we learn as we go.

“Sense and Respond” is not always better than “Plan the work and then work the plan” but agile practices are optimised for continual questioning and learning. This has a profound impact on how we use metrics. Progress according to our original commitments is less important, certainty is not provided and we need to know quickly when and why we should change direction or stay the course. We need to check frequently to see if our goals and assumptions are still valid and if we are confident we are heading to somewhere we want to be.

Of course, we also want to know if we are progressing fast enough for the journey to be worth the end goal and to know if we are doing the right things well enough, but we are less certain that we understood all these things when we started and we hope that we understand them all better each week. In theory the most agile journeys will involve ever decreasing ambiguity and ever increasing certainty, which is something we might want to measure, that did not really matter if we were certain of our plan to start with.

Does this mean we leap first and think later? No, it does not. It means that we want to create a hypothesis and then we look at how to validate it. We want metrics that help us understand whether our thinking was right or wrong and where we can confidently keep moving, as well as where we should stop moving.

This means we predict, sense and respond, not rush and hope

Does this mean that we cannot predict the future? Partially, but not entirely. In fact it means that prioritisation and planning become more critical than ever before.

In every agile team that I work with, some things are known (for example the law will change from 1 January, we need to patch this security flaw, we need to integrate with these API standards using Lotus Notes as a code base) and some things are not known (I think people would love to have an online guide that looks like a staple rather than a help menu).

Some things are also given, or “must have” because the stakeholders funding our endeavour would not fund it if that did not happen (the payroll system must be accurate) while other things are trivial (if we had time it would be good to simplify this step of the process). We may as well get on with delivering the must haves before we give too much thought to the nice to haves.

In an agile context we should be measuring our progress with reference to the progress on the “must haves” rather than the “everything we could do if we had time” and we should be measuring how we test and validate our assumptions rather than whether we complete the tasks we originally thought we would complete. This has some impact on the metrics we might consider using.

There are different decision makers in an agile context

Another change faced by an agile team using metrics is that the team are empowered. This is not an optional extra that you add to agile approaches later, but core to the whole thing.

Since the team are empowered, they make their own decisions about things. This is interesting because, if the metrics are “continuously delivering valuable information to support decision making” then the customer of a lot of the metrics will be members of the team and not external decision makers.

In theory, everyone can use the same information, regardless of whether they are team members or governance people sitting in another city, so we could come up with one set of metrics for both the team to use AND the other decision makers to review.

In practice though, different decision makers may be making different decisions, relying on either different information or a different presentation of the information.

This impacts our metrics too.

  • Traditional reporting might look at resource needs and milestones, both of which are important to the team, but both of which might be of more concern to a product manager, a portfolio management team or another group of stakeholders. Traditional reporting is often optimised for people sitting outside the team and, potentially, people who are managing a dozen teams. Collecting and reporting these metrics might offer little value to the team and might even be redundant if the team are making decisions locally about how they apply their available resources.
  • Agile teams often focus on stand-ups, retros and story walls (I consider all of these as measures as they are continuous information for decision making). Agile teams often also produce burn down charts or other information that makes sense to the team but may mean little to others who are not close to the team.

So we have a range of metrics available to us, some of which are better suited to different audiences making different decisions.

I think that aligns with the idea that we start with “who is using this for what decision” rather than trying to force random seeming metrics on people who do not care. However this can also lead to a lot of collecting, reporting, justifying and documenting metrics and performance measures.

That seems to suck because it is a lot of (potentially wasteful work). It is certainly not the same as “customer collaboration over comprehensive documentation” or “keeping things simple – quick and easy to get our work done.

Has anyone solved this complex dilemma?

Fortunately this problem has been solved.

The first group of people offering solutions are the EVIL agile coaches. These are people not focused on making it quick and easy to add value, they are not focused on helping to empower the team and, most importantly, they do not recognise that the team must be accountable for their actions and that stakeholders have a right to know where and if value is being created or destroyed.

Evil coaches offer one of two solutions, that are, unfortunately only useful if you want to increase suffering and carnage:

  1. The “let them rip” strategy, where the evil coach alleges that agile is all about blind trust and good luck. They suggest letting teams rush ahead with no governance and no validation of their work or performance. It is a joy to work in these environments until you learn that ignorance is not bliss and that “not sensing and not learning” is “not agile.”
  2. The pre-packaged sales team who have the world best set of agile metrics that can be air dropped onto your teams without considering the local environment. This is kind of the opposite of sense and respond and is more like “sell and be gone.”

Who else has solved it then, perhaps with more goodness and less carnage?

I have solved it several times with different teams. So have many agile consultants (who often do bring useful tools that may add value) and many agile teams who were perhaps less knowledgeable in the ways of agile reporting, but still keen to understand and optimise the creation of value and happiness. The key is to take an agile approach.

Start with a simple question:

Who cares about the success of this team?

A typical good coach

Then the next questions are things like “what matters to them?”. This then leads to seeing metrics as information, both for the team and the stakeholders. Agile frameworks do a pretty good job of offering some help to the team and for other stakeholders there is a lot of good governance out there if you apply it to “empowered, accountable teams” and “sensing and responding.” I also find it really useful to treat my team metrics for stakeholders as a communication plan rather than a set of numbers.

Since agile is about ongoing learning though, there is another group who have found an elegant solution. Teachers have had to look at assessment as a way of measuring learning and also the value of what students have learned. They start with the agile question of “who wants this information for what purpose” and then they go a lot further.

If you want to learn more, then your can google “assessing learning.” You might find articles like this one. However you can also wait a week or so and I will attempt to apply the way teachers assess learning from different stakeholder perspectives.

I also think I can show you that they are good at combining two seeming contradictory goals: “continuous learning and iteration” and “long term curriculum performance.” to measure the value and gaps in learning for their key stakeholders.

Thanks for listening to my ranting and raving. Hopefully I will be back to shorter articles next week.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.