Scenario testing the cycle of pain for regression testing

If you want to do a Phd in human psychology, then grab an  IT team and ask them to do regression testing.

Regression testing is simply the idea that when you make changes to a system you should test to make sure you haven’t broken what was already there. But for reasons unknown to psychologists, IT teams fall into a strange psychological pattern whenever asked to do regression testing.  This is the opportunity for someone to do a Phd – we understand that people consistently adopt the same four stage cycle of pain, but social scientists are at a loss to explain why:

1. Regression test denial.  People on most projects swear that there is no time for regression testing at the moment.
   • They will admit that in the last project they spent an inordinate amount of time (and frustration) near the end of the project trying to understand and fix problems that were created early in the project. They will also observe that most of these problems came to light as soon as regression testing finally started.
   • They will also complain about “the business” being indecisive and asking for new things every showcase (demonstration of the software).
   • The business will say their “alleged new change” was an obvious part of the requirements but the team will say they could not have anticipated it.
   • A lot of time will be spent fixing these new demands, but even more time will be spent arguing about them and demanding ever greater detail in the requirements, even though we still keep missing things and the business keep changing their mind. 
   • I accept that better requirements will address some of these issues and better code writing will address others … but I will also claim, even though people seem to deny it, that doing some good regression testing early on will save a lot of time and frustration through the whole project.
2. Robot dreaming. If you can get people to accept that testing is a good idea then:
   • They will start to dream about a wonderful, magical world where fairies come and do the testing for them. But not many IT people believe in fairies anymore.
   • Having accepted that magical test fairies do not exist, people will instead dream of robots doing all their testing for them. So they fire up Google and start searching for wonderful sounding robots (often called automated testing tools) to use.
   • But it takes a lot of time to Google for the robots and it takes longer to install and test them even if you do start to use one.
   • So someone still needs to set up the regression testing for the robot and someone still needs to do some manual regression testing until the robot is ready. This causes some sorry and a brief attempt to go back to step one “just for now”.
3. Tester and intern persecution.Finally we get to the real problem. We all know we should do regression testing but nobody in the team wants to be the one to do it. So now we finally start to tackle the problem.
   • The team instantly starts a search for interns.  If they find them this is fantastic because we can now inflict all the horrible regression testing on them.
   • If there are no interns then the team search for graduates, “business people who want to become BA’s” and other victims. 
   • If there is nobody inexperienced enough to trick into doing the work, then finally the team dump the whole thing on the usual victims in IT – the testers. After all regression testing is a testers role isn’t it?
4. Pointless suffering. Having been forced to do regression testing, the victim now sets out to prove that regression testing is a horrible waste of time.
   • The victim starts doing some simple testing by running the same test on the same functionality multiple times.  This finds few if any bugs.
   • The victim then starts running some functional tests on new functionality until reminded that this is not regression testing.
   • The victim finally, grudgingly finds some simple test cases and starts to run them each week for an hour or two.
   • Running the same simple tests for a couple of hours results in very few defects since what is being tested is usually the unimportant stuff or the stable stuff that nobody is breaking.
   • So the victim concludes that they are spending a lot of time for very little return and then finds other more important work to do.
   • The team return to step 1.

But rather than the “cycle of pain approach” I think it might be better to use scenario testing as a way to do regression testing.

The main reason is that this is the first approach anyone taught me and it worked. So I never really learned any other approaches. Not the best reason I guess, but a good enough one for me.

But the process I learned process involves more than just writing some test cases and running them. So it seems like more work, but in fact it is not. I hope to show that the approach I learned is actually quicker, more effective and more fun than either the “cycle of pain”.

The whole approach consists of four steps:

1. Understanding the context. Regression testing involves testing whether you have broken the functionality that was already there. So it stands to reason that you will test better if you understand what is already there, if and why it matters and who uses or cares about it.
   • Don’t panic.  This can be done easily. You can read the project mandate (or project charter) and you can talk to people.
   • In addition you can use some simple techniques to improve your understanding. I would suggest these as a pretty good starter pack
     • PAC
     • Using the international standard for being scared (FMEA- asking what could go wrong)
     • Creating a product summary statement
2. Scripting. You can do effective regression testing using only exploratory testing. But if you are doing the same tests repeatedly then it is probably worth writing down what you are going to do to save time when you are doing it.
   • Don’t panic. Some people think you need to build an entire library of scripts before you start testing. But you can start with a few basic ideas and then build on them each week as you go.  I will publish a guide to creating both slack and good scenario tests soon.
3. Making a map. You can run a lot of manual tests, but if all the tests are covering the same thing then you will be wasting a lot of time re-testing what you have already tested. You will also be consistently not testing the same things every week, which seems a little dumb.
   • Don’t panic. Your first go at this can be a simple list of the 3 or 4 scenarios you are testing to start with.  But as time goes on you will find it more and more beneficial to have some kind of map linking your scenarios to the things you should be testing.
4. Running some tests. Of course the point of the other 3 steps is to help you run some tests, so this step is pretty obvious.
   • Don’t panic.  You can run some tests this week and some next week.  As you go you will update your scenarios and your map so that you get more and more effective at testing in less and less time.
   • As you are running your tests though, you will learn more about the system and you will find that the context in which you are testing is changing. So go back to step one.

I guess the choice is up to you and the team.  The traditional approach (the cycle of pain) has been used for years by many experienced teams, so it is tried and tested. On the other hand, the creative testing cycle I have described here sounds a little radical and may not be as widely used. But I have found the creative cycle to be less painful, more interesting and more effective for me.