Guessing is the absence of research

I saw a refreshing take on research recently and I thought I would capture it here.

The essential idea is that research is a way to reduce uncertainty, which then leads to 3 insights:

• When we plan for the future we are usually guessing;
• Guesses are necessary but they are also just the absence of research; and
• Research then, is about changing the odds in our favour by making it more likely that our guesses are correct.

Research is good

If we want to be more confident in our guesses, then we can do some research. Doing this research will be good because it will change the odds of our current guesses being right from, say, 1 in 10 to 1 in 2.

It follows then, that research is worthwhile, if the cost of improving the odds in our favour (the effort of doing the research) is likely to be less than the cost of a bad guess (the pain caused by guessing wrong and then recovering from our mistake).

Research is of little or no value when:

• The research is more costly than going with a wild guess and seeing what happens;
• We will ignore the research because we have already converted a guess into a commitment and we plan to proceed regardless of what we learn in our research; or
• We are not doing research that improves the odds of the most critical guesses being right, because we are too busy researching easy but unimportant things.

Features are also guesses

Building a new feature creates an output (the feature is released to our customers) and then we hope (guess) that the feature will create a good outcome. Usually that outcome is to create some value to the user or some extra revenue for the company.

Since we are only guessing that the features will create the right outcome, there is a risk that we are wrong. We can mitigate this risk by making it cheap to create and test new some elements of the potential new features (Using approaches like MVP, MOSCOW, Kano).

So, where it is cheaper to build features and see what happens, then we should do that. When, on the other hand, it is cheaper to do some research before inflicting experimental new features on people, then we should do some research first.

The choice we make is not about either always doing research before building sometime or always working on spitting out features without a lot of research. The choice we make is about how to do the right mix of research and building of partial solutions to most economically improve the odds of creating something valuable with our limited time.

Research must lead to learning

For research to be useful though, it must be used. For that to happen, people must take the research into account when they make decisions.

Based on that idea, the quality (and value) of research is not just related to how valid the research is, but also two other factors:

• The willingness of decision makers to change their minds when the research changes the odds of their existing guess being right; and
• The ability of the researchers to interpret the result and explain the insights and implications to the decision makers.

I wonder how often we do research after we have already turned our guess into a commitment. Why do any research if it will not change what we decide to do or how we will do it?

The animal farm principle

OK, so research must be both effective in changing the odds of a guess being right and also useful to the decision maker.

We are not finished yet though because there is one more element that makes research valuable.

In the book Animal Farm, there is a phrase that “All animals are equal, but some are more equal than others.”

I guess in terms of guesses, “All guesses are potentially scary, but some are a lot scarier than others. Guessing which bottle contains poison is really scary. Guessing which book might be interesting to read next is not so scary.

Good research will be of the greatest value if it is focused on the most important guesses – the ones that represent the greatest danger if they are wrong.

If we know we can build a feature but do not really know that customers will use it, researching how to build the feature seems less useful to me than researching what the customer really needs, since it is more dangerous to build the wrong features than to struggle more than expected when building them.

I wonder how often we spend time deciding how to build something rather than learning whether people will care if we build it. I wonder if that sometimes happens because it is easier to understand whether we can build something than how it will be used, so we do what is easy rather than what is needed.

Research is not just about what feature to build next

So we have some idea of WHEN to do research, but there is also the question of WHERE or ON WHAT should we do our research.

Some product and design teams report which features they have released this week or this quarter, but this is reporting not research. They might then decide to do some research on which of those features have been adopted, or liked, or hated. Maybe they will use Pirate Metrics or HEART metrics to do this. The “research” done here might therefore be data gathering, with some anecdotal input too.

Where that is the case, all the points so far in this article apply to making the research valuable.

The information we gather might tell us whether to spend more effort on new versions of the feature we released, or fixes to bad guesses we made about how the feature would work or how people would use it.

Rather than just learning about guesses we made in the past though, research is probably better focused on what guesses we might make in the future.

We should be biased toward researching what people are trying to achieve (JTBD, Pains, gains, problems) rather than just what features they are using. Understanding what people actually do when we are not watching them involves a lot of potential guesswork and changing the odds that our guesses about them are right is a big win for us.

One more thought here though. Improving our guesses about what people are asking for seems less innovative than the far riskier guesses we make. The best research would be research that improves our odds of guessing correctly what people would love to have available, even if they did not think to ask for it.

If we can improve the odds that we will make a good guess about what people don’t yet know would be really useful, then we are moving into the realm of innovation and competitive advantage.

So the biggest advantage that research might have over building and testing features cheaply is that we can gain insights that others do not yet have, but researching what is happening in customer’s experiences beyond what they are using our product for.

Research is embedded in prioritisation

So much for prioritising the right research.

The other insight was that research should strongly influence our priority of other work.

In some organisations, people use RICE to prioritise their roadmap. Research is the C part of that equation, which links directly to the question of when to do research, because “changing the odds of our guess bring right” can also be described as “increasing our confidence in our guesses” about all the other letters in the RICE anagram.

In theory, we can improve the confidence we have in our guesses through persuading each other, giving ourselves pep talks and so forth. Unfortunately doing so really just increases our hope that our guess is right, it does not actually change the odds in our favour of us being right.

So if we use something like RICE scoring, then research fits into the prioritisation process very deliberately and very neatly.

Conclusion

A guess is when we have an absence of research. Research is done when we want to improve the odds that we can rely on our guess.

Based on that, the amount of research should be a function of:

• The improvement in the odds of us being right about a guess that the research leads to;
• The importance of the guess that we we want to be more confident in;
• The likelihood that we will actually act on what we learn (change our mind or move forward confidently); and
• The effort or cost of doing the research.

Similarly, the way to measure the value of our research is to measure these same four things once the research is completed:

• The change in the odds of being right, that the research resulted in;
• Which of our most important guesses are better informed by the research;
• Whether we actually acted on the research; and
• The effort or cost of the research.