Regression testing days 3 to 7

This article is part of a series on making regression testing useful rather than painful. The most recent article was (as you would guess from the title to this one) about day 2 of our regression testing adventure.

So far we have been looking at how to do some testing, and then do some basic scripting and mapping as we do more testing. In doing so we have (hopefully) been learning more about the system we are building but our main focus is on making sure it is still performing the way we want to.

Now we are going to evolve our testing onto a proper OODA loop.  I have created another article to explain what an OODA loop is, but for our purposes it means this:

How fast and how well can the team move between the following 4 tasks?

1. Observe the world around them – the way users are operating, the way the system is performing and the way things are coming together.
2. Orient themselves or make sense of all that data (interpreting, analysing, integrating and assessing).
3. Decide what to do next with all that information.
4. Act on their decisions as a cohesive group.

To do this we need a little lesson in OODA regression testing theory

Regression testing is a series of loops not one straight line of work

With regression testing, most people think it is like a line rather than a loop:

1. Understand the system
2. Create some scripts and then make a list of them (map them)
3. Run the tests to break the system
4. Fix the bugs

But this misses two key ingredients in our OODA loop – both of which are related observing what happens when we act and then learning from it:

• By running one set of tests we should be able to learn more about the system so we can test it better next time; and
• By running one set of tests we should be able to learn more about the system so we can use that knowledge to update our requirements, create training material and pass what we have learned on  to others.

So the real process looks a little messier when you first look at it:

But that looks like a lot of work

Fortunately it is really what we have been doing in day 1 and day 2. So for the next couple of days we will simply repeat what we did in day 2, and add these questions:

• Should we improve the way we capture and track bugs?
• How do we pass potential new requirements? Who should decide whether to act on these?
• How can we improve our test scripts and maps?
  • Should they be more sophisticated?
  • Should we drop some in favour of more exploratory testing?
• How do we pass on what we are learning?

Each of these questions relates to one of the Act boxes in my diagram and you will notice that each of them then becomes another OODA loop within the model.

My suggestions for early improvement

I think it is a good idea to start writing out the most common or critical test scenarios in a “Use Case Like” format.  This turns out to be a good way to pass on information for training, a good way to automate and a good way to show people what to do when they are testing the ssystem

I also think it is a good idea to at list start collecting a list of questions and known (or expected) answers.  This is often done in the format of an FAQ.

I think it is a good idea to write down the work arounds your are finding. If these never get fixed then you have some warnings to include in the release notes. Of course, they might be fixed later but even so it will be useful to remember them as you continue testing.

Finally, I think you will realise that  your are finding several “trivial bugs” that the team don’t want to fix. Rather than tracking these as defects or “unlikely to do requriements” you could also record them in an FAQ like format so that they become known errors or, in common IT terms “Unexpected features”.

So you can see that I don’t see regression testing as breaking the system. Instead it is the engine that is driving our continual learning (and hopefully improvement) of the system.