But then something significant happened. We discovered that work was not just about performing a set of predictable tasks. We started to realise that:

• Work was about adding value rather than destroying value rather than just delivering stuff; and
• We started to see value as something delivered through the interaction of entire teams and not through individual achievement.

Yet you will still find performance agreements in almost any company that are based on individual achievement, sometimes even ranking people against their peers to see who is the best and worst in the team.

Doesn’t it seem a little inconsistent to say that teamwork is the main ingredient for success and then assess people on individual delivery?

But assessing teams instead of individuals is a huge mind-shift and it is a mind-shift that I have not been able to make. Try as I have done to embrace the agile (and modern management) concept that it is the team that delivers value and not the individual, I still think there are people who contribute more than others and I can’t shake the feeling that these are the people I want to attract, retain and reward, even if I have to do so at the expense of others.

Perhaps I am still evolving as a manager and still learning what I know is true::

• Intrinsic motivation is better than extrinsic rewards – ie motivation comes from within;
• It is the diversity of the team and the mix of skills and styles that leads to success – you need different types of people in the team, not clones of some alleged paragon of excellence; and
• Teams are increasingly virtual (temporary) and distributed (spread around the globe) so most managers are now managing people they don’t actually work with directly anymore.

Unfortunately that leaves me with an inherent contradiction – I want the best people and I want to reward and motivate them … but the best people is actually the best MIX of people, and it is the people in the team who will motivate themselves and each other.

One solution is to wimp out (ie avoid the problem). This works well if you are not too passionate about leadership. You can either defer all performance discussions indefinitely or you can use the old fashioned performance agreements and apologise (while adding that it is not your fault – the company inflicts this on everyone). But I don’t find those approaches very motivating.

One surprisingly easy and accurate approach is the “net promoter score” that is used in ITIL and other methodologies to evaluate the performance of an overall team or set of services.

Essentially you just ask you stakeholders one question – Would you recommend us to others? But rather than getting a yes/no answer you ask for a score out of 10. And rather than seeing 5 out of 10 as a pass mark you take anything under 8 as bad and 8 or above as good enough.

So to rate myself as a manager, I could ask my team (if they can be honest) to answer the following question:

If you were talking to your friends, would you recommend to them that they work for me if they get a chance? Please give me a rating out of 10.

In a similar way, if I was managing a group of testers who worked on projects, I could ask the project manager, the developers they were working with and the other testers on the project:

IF a friend of yours was going to work in a project, would you recommend that they have X in their team as a tester? Can you please give me a rating out of 10, where 1 means “no way” and 10 means “Absolutely- every time, even if they had to cancel something else in the project to find the extra budget”.

I think you would probably get a fairly consistent rating from different people in the team and that you will actually get a pretty good assessment of the performance of the tester.

So, with this simple question in hand, could we simply abolish the traditional performance agreement and appraisal process?  Sadly, I don’t think we can.

It would be great if we could allocate a standard score to all our team members and then we could buy and sell them on a “tester commodity exchange”. If I needed adequate testers, I could pay less and get some 5.5er’s. But if I needed some really challenging testing done I could pay more than my competitors and buy in some 9.5’ers.

Giving people a score out of 10 (or ranking them against their peers) should never have been the real purpose of the performance agreement.

The real purpose of performance management discussions is just that – to have a discussion about current performance, possible improvement areas and possible growth for the future.

The real discussion is therefore about where and how the team member is adding (or destroying) value in the organisation. It is about how their work suits them at the moment and where they should head from here – to continue (or start) to find work that is challenging, motivating and valuable to the team member and also valuable to the wider team and organisation. In fact in some cases it might even be a discussion about how someone who is adding great value but has outgrown their role can find a more fulfilling role somewhere else.

So I think we need to improve the performance discussion we have. The problem is – how can we do that?  How can we measure the value of someone who provides intangible value (such as quality that a tester helps build into the system) and how can we talk about growth and development in a world that changes so fast my new mobile phone is old before I finish my first phone call?

It would be great if you could let me know where to find some of the answers that others have come up with. In the meantime I am going to work though my own ideas and publish some of them here. So let me know what you think as I go, and I would really appreciate any suggestions or ideas that you think might help.

And they are all related to the managers in the team not having enough time to talk about succession planning.

Managers spend a lot of time talking about the people in their team – who is good, who is not so good, who is happy and even how people are getting on with each other. But people are not talking about

• What to do when people move on from the roles they are doing;
• Wether people are still growing in their role and if the role is growing with them;
• Whether people need more support to learn how to do their existing role better; and
• Whether we need to start developing people now to be ready to fill the roles we will have in the future (2 years from now, 6 months from now or even in a month’s time when the big project goes live).

These are the topics that are meant to be covered under the heading of succession planning, and without them the following problems seem almost inevitable:

• People get stuck in a role that no longer really challenges them because they do it so well and it is so important to us (eg they are the only one who can look after the backend production database. This is robbing the good people we have from the potential growth they could be getting in a different or extended role;
• Companies are going outside to hire managers above a certain level because they lack the talent within the company to fill the most important and most rewarding roles in the organisation; and
• Teams are becoming increasingly single point sensitive (the can’t afford to lose one particular person) and then go into crisis mode if that person moves on or even tries to go on holidays.

So, is there a sophisticated and powerful way to address all of this?

No, but there is a simple conversation that the management team can have on a regular basis. And if the management team have this conversation then it can support the performance and development discussions they are having with their team members – so those team members are aware of potential opportunities within the company in the future and have the chance to develop the skills (and experience) to fill those roles.

The management discussion

I think the succession planning discussion deserves to have its own meeting, otherwise it always gets left to the side while we address “urgent issues”. And I think it needs to involve the whole management team.

The meeting should start as all other meetings do:

• People turn up and talk about the weather. Then the manager arrives 5 minutes late and apologises. he or she asks what has already been covered and the team reveal that they didn’t do anything because he or she was not there yet. (OK this part of the meeting can be improved).
• We remind ourselves why we are here – The purpose of this meeting is to discuss the critical roles that we have in the team and the people either in them now or potentially in them in the future.

Cool – but how do we discuss all this? Well, here is a form to use:

As  a word document: Succession planning table

As a PDF: Succession planning table

And here is a possible agenda:

1. What roles are critical in our team?  Of those, which ones are the most critical?
2. Which roles have only one person in the team capable of doing them? How much would it hurt if that person won the lottery and retired?
3. Now the hard part – for each critical role:
   • What is the name of the role?
   • Who is doing it at the moment?
   • How experienced are they in the role (draw a scale from [new – still learning – comfortable – mastering the role – ready for something new]?
   • How might the role be different 2 years from now?  How might it be different in 6 months?
   • Is there a development plan in place to keep the current incumbent (person in the role) growing?
4. For each role
   • Who is ready to take over the role in an emergency? Or if the current incumbent is on holidays?
   • Who is ready to take over the role right now?
   • Who might be ready in 6 months?
   • Who might be ready in 18 months?
   • What outside options exist for filling the role, temporarily or permanently? For example are their consultants who could help in the short term, are their people in other parts of the organisation we could bring across, are there people we know of in other companies who might be able to fill the role?
5. In addition to the roles we have today, are their any emerging roles that we might want to have in the future?
6. For the people in the team who might be able to fill our critical roles in the future?
   • What development plans should we have in place to ensure they are ready for the role?
   • Do they actually have any interest in doing the role?

All that discussion will take some time and might take quite a bit of thinking to do properly. But if you have a go at it a few times I think it should all come together.  Of course you can also get your managers to do some preparation before the meeting … or you could cut the meeting back to only talk about agenda item 3 and 4 in the meeting and let each manager do the rest on their own.

Hopefully, even knowing that the management team think about this stuff will help motivation a little. And if you actually implement improved performance and development discussions as a result of this meeting then you should definitely see an improvement in the performance of your team.

The key benefit of the approach is that you do NOT try to understand and prioritise every thing at once against everything else. Instead you break the problem down into very small pieces and move through each one methodically. You then come up with a ranking of the importance of each variable from top to bottom.

And the approach scales well. You can do it on your own or you can get a lot of people to use the process at once to form a consensus about what issues really matter the most. I have used the approach for everything from setting priorities in retrospectives to ranking the capabilities needed in a team to structure more effective training.

An example of where you can use it

For example, lets say you are working with a team of testers who work on projects across the whole organisation. They have been complaining about being unloved and their stakeholders are questioning whether they actually add value to projects.

You do a retrospective with the team to reflect on where they think their could be improvement and you have some meetings with your key stakeholders. As a result you find out that the team could get better at communication, team building, consistent tool use, work quality, knowledge of the systems being tested, working with the developers instead of against them, finding critical defects instead of just cosmetic things, problem solving, collaboration, good manners and a range of other things.

If you decide on one thing to fix then everyone will complain you are not fixing the rest, but if you try to fix everything you will not get anything done. Even worse, if the team don’t see value in what you are fixing then they will not help and you will fail. Even worse again, if your customers don’t see value then they will keep escalating and complaining about everything and you will be too busy apologising (or hiding from them) to focus on fixing things.

So you need to prioritise and here is where you can use stepwise programming.

Applying stepwise programming

The approach is simple. First you create a matrix with the items you want to prioritise listed in both the left hand column and the top row:

Next you start on the first row and compare the heading you put in the left-hand column with the heading you have put in the top row.

• If the variable in the left hand column is higher priority to the the one in the top row, then enter a 1. Otherwise enter  -.

For example I think it is important for the team to get better at both root cause analysis and collaboration. But if I had to choose between them, then I would choose root cause analysis. So I would enter a 1 in the appropriate field. I then continue through the first row doing the same thing:

Similarly, if instead of doing the exercise on my own I had my whole team of 6 people do the exercise I would get each person to complete the row separately and then update the table with the total votes that people gave to Root cause analysis compared to everything else:

So I could say that the team think getter better at root cause analysis is more important than getting better at collaboration and (unlike what I think) the team as a group believe that getting better at root cause analysis is even more important than using automation to improve testing.

But the real benefit comes when you have completed the whole table:

So you can see from the table that the team (as a group) believe the biggest benefit would come from getting better at root cause analysis. And if we could pick a top three to focus on then we would pick Root cause analysis, Collaboration and Use of automation.

The numbers may not add up in this table because I just made them up quickly. But even in the real world some people seem to vote that (say) root cause analysis is more important than collaboration the first time you compare them and the vote the opposite way the second time. To counter this, some teams only allow comparison once and then simply put the number (6 – 5) in the field that has collaboration as the left-hand heading and root-cause analysis as the top. But I don’t do this, because I think even that flip-flopping can help you get a better understanding of where the team feels the value is.

In the real world, I often use this technique when there are between 10 and 20 variables to compare. I find it a really good way to prioritise when I am faced with both intangible stuff (values and preferences) and great complexity (too many variables for my little brain).

Of course it does not tell you how to fix the problems, only which ones the group really think you should focus on first.

But where do you start if you want to do effective regression testing, but you are already busy and don’t want it to be huge burden?

I hope this article and a couple that follow will turn out to be a guide to one painless way to get regression testing up and running. Let me know if it helps.

Day one – first thing in the morning

Start by getting some context before you expend a lot of energy testing what you don’t understand. I have put some tips in this article, but you can also just do the obvious things:

• Read the project charter if there is one, otherwise sit down and create one
• Sit down and discuss the users of the system you are building and the product you are building
• Document what you found, even if it is just on the whiteboard or on napkins

This can all be done in under one hour and it is worth getting the regression testing team to sit down together to do it. Of course you can feel free to take 2 hours if you want to fully understand what you are testing

Day one – 20 minutes of scripting

Create some scenarios that you will use for testing.  This does not mean that you have to fully document well written test scenarios, since that comes later.

The easiest way is to simple use the PAC table you created and come up with some examples of what some users might do.

I am using the example project from a previous blog where a cafe owner wants to :

This is our initial map for our regression testing. It won’t last long but it gives us a place to start.

Spend an hour or two doing exploratory testing

Now that you have some rough scenarios, log into a test version of the system and try to run some tests.

The tests you run will be “exploratory” because you have not created any instructions yet. But for each scenario, don’t just log in and make things work, instead try this:

• Try to run the scenario successfully from end to end and then record the outcome
  • Write any bugs or “strange things” on a piece of paper or whiteboard
  • Write down how long it took to run the scenario
• Stop and ponder for a moment, Use the international standard for being scared to come up with things that might go wrong. Then try to run the scenario again seeing if it will fail.
• As you run the scenario a second or third time, try using the international standard for being stupid to come up with mistakes or “strange things” the users might do at each step during the scenario.
• Document your findings again
  • Scenario number
  • Potential errors or questions I found
  • Rough steps I followed

Go and report the potential errors you found and if they turn out to be errors then record them in whatever bug tracking system the project uses.

Discuss the context again for 20 minutes

Sit down with the other people doing regression testing and discuss how things went

Provide the following “management report”:

Tests pass if you found no errors that went into the bug tracking system. They fail if you did record errors. Passing or failing is not good or bad, it just shows which scenarios found something.

Now sit down with the group again and discuss the context your are testing in

• What led to errors?
• What did you learn?
• Should you change the PAC or Product statement? Should you change the scenarios you are running.

There is more to do tomorrow, but you have already done more thorough regression testing than a lot of teams do. Well done.

Then I worked on a project with a mining company and one of the consultants explained human factor analysis to me in simple terms. He told me that mine sites can be dangerous and part of his job was to “stop people killing themselves when they are stupid”.

I suggested he stop hiring stupid people but he told me that they tried that and it didn’t work. Apparently you can be really intelligent on a mine site 800 days in a row but then be stupid for 10 minutes one day and be in an accident and then be killed.

“Luckily we have a standard for being stupid though” he said

I was surprised that he could get people to comply with a standard for making mistakes, but then he showed me “iso90210 (not a real standard) – the standard mistakes you can make when using a machine”.

1. Action omitted
2. Action too early
3. Action too late
4. Action too much/too often
5. Action too little/not often enough
6. Action too long
7. Action too short
8. Action in wrong direction
9. Right action on wrong object
10. Wrong action on right object
11. Wrong action on wrong object
12. Information not obtained/transmitted
13. Wrong information obtained/transmitted

It seemed like a long list of obvious things, which is what it turned out to be. I found when I was analysing processes (which I was doing at the time) that many errors could occur in the process when one of the things happened in the list. So I combined this with the “international standard for being scared” (FMEA) to find I could tear most processes apart and find errors really quickly.

Even if you can’t break the process instantly (which you usually can), you can almost always break it if you:

• Apply one standard mistake; and
• See what happens and then apply a second one rather than simply moving on.

This also applies to computer applications. For example

• The user gets to question 5 and realises they do not have the information they need to complete the form.
• The system gives them a polite error message.
  • Rather than completing the form or hitting cancel (the right thing to do) they exit by hitting the “back button” or closing the browser (right action wrong object); or
  • They hit cancel and nothing happens so they hit it again 3 times (too often) and third time they hit the button the browser decides to reload and process the click on the wrong button (something from the next screen).

I find that even though I am not a tester, I can often break applications in regression testing by assuming my users will comply with the standard for being stupid at least once or twice.

You can take things a lot further if you study “human factors” because it turns out that “human factor analysts” (I am not sure what they call themselves) have spent a lot of time trying to find common reasons for all kinds of things. For example:

• Common reasons why people do the wrong thing deliberately (peer pressure, not enough time in the day, rewarded for wrong behaviour etc),
• Common reasons people make mistakes (information overload, falling back on how the old system worked, lack of knowledge, lack of caring etc).

So there seems to be a whole field of study about how to make mistakes properly. I guess if you are going to do something you may as well do it properly … and since we make mistakes so often we may as well get really good a making them

But what is of more use to IT projects where people want to build good systems:

• If your system could kill people, probably get an expert in safety to test it.
• Even if your system won’t kill people … surely you want to anticipate the most obvious mistakes that users will almost certainly make.  So it follows that you would also want to at least do some simple testing to find out what happens when they do.

So I think it is fair to say that any team who want to produce good processes, good IT applications or good products will do some regular testing to see what happens when “the dumb users” make the same predictable mistakes we all tend to make.

So I would like assume every developer, BA and tester would do some simple exploratory testing  as they add features and I would assume that every team would do some regular regression testing as their system emerges.

But maybe that is the uncharted territory for human factor analysts – why do IT teams consistently make the same mistake of assuming the users will do exactly what they are meant to, when everybody knows they won’t?

Why would anyone spend$5m and 6 months of their life on a project and then only test how a user might actually use the system at the  very end of the project when there is no time left to modify the way the system behaves?

When you think about it though, shouldn’t we do a similar thing to define our project purpose when we start the project. I guess the answer is yes, but it could turn into a lot of work. So what we need is an easy, lazy approach.

This is what I think you can do on just about any project.

1. Define the problem you are meant to solve.
2. Define your scope.
3. Flavour to taste:
   1. Consider PAC to define the users
   2. Consider defining your product
   3. Consider adding an estimate
   4. Consider listing risks and other bad stuff

Paragraph one – choose one of these problem statements

• Problem statement from the book Fierce Conversations by Susan Scott (you should really read this book if you want to be both lazy and successful at the same time):
  • The problem we have is ….
  • It matters because ….
  • Ideally what we would like is …
  • So far we have done …..
  • Therefore, what we want to achieve next is ……
• Focussing question
  • How can [We] provide [Something] to [someone] so they can [do something or get some benefit or deal with some issue]
  • Eg – how can the IT team provide a database to Jenny so she can run her coffee club. OR how can Jenny provide a coffee club to her customers so they can buy coffee to take home
• The old fashioned story/legend format, which is the simplest you can use.
  • Currently [things are like this] but it sucks because [some annoyance] so we will [resolve the issue]
  • For example Jenny runs a coffee shop and she would like to sell take home coffee to customers so we want to create a coffee club to do this. Or Jenny has set up a coffee club for her customers but she is losing track of all the information she needs to track so we are gong to build her a database.

You can find a couple of other problem statements in a previous article.

Paragraph two – say what is in scope and what is not

No project ever fully solves a problem from every possible angle. If you build software, maybe you should train people and maintain the software. But even if you maintain the software for 5 years, at some point someone should replace the software with a better solution. But even if your project team agree to come back and replace the software, will they be the ones who pay for the carbon offset to make up for the extra greenhouse gases produced during the project lifecycle.

People generally understand that some things will be in scope and that others will be out of scope. Yet we often make wildly different assumptions about what actually goes into the scope.

So I think it is useful to define the following in at least some detail:

• This is in scope – deploying a database, maintaining it in production; and
• This is out of scope – ongoing training, maintaining the customer data.

This is often put into a table with the in-scope stuff on the left and the out-of-scope on the right.

But if you are a vendor or you are delivering to a demanding internal client you might choose to go one step further:

• The project team will deliver (in scope) – deploying a database
• The customer will – provide the team with access to users, supply coffee, provide testing
• The following is out of the scope of this project – ongoing training

Paragraph three onwards.

• Add constraints or priorities if you want to.  For example it needs to be in production before Friday next week (eek)
• Add PAC, Product statement or risks if you like
• Add an estimate of the effort involved if you want to
• Add the name of the sponsor, the provider and others if you really want to push the content.

If you want to you can even add a communication plan, a view of wether users will actually adopt the change we are implementing or a test strategy, but not all projects will need this much detail at the beginning.

A simple estimate

People often hesitate to provide estimates. But you can provide something as simple as this

We are 90% confident that the project will take between X and Y.  We could be more confident if [something]. For example:

• We are 90% confident the project will take between 50 and 150 years (let’s kill it now);
• We are 90% confident the project will take 3 developers between 3 and 5 days (let’s do it)’; or
• We are 90% confident the project will take between 3 weeks and 4 months. Hmm -
  • We can kill it if 3 weeks is too long;
  • We can get started if 4 months is tolerable and report back as we go. We will be more confident once we install the invergolator and get it running on windows 7; or
  • We could do more analysis.  This would add one week to the project but when we understand your needs better and review the deployment options for invergolators then we can provide a better estimate.  Let me know what you want us to do.

I have often found these estimates are enough to make informed decisions. But if you want to get far better at estimating then you should probably read “How to measure anything”.  That book even teaches you to estimate and understand the value of doing more estimates.  But the lazy approach is just to do what I did above.

What do you think?

It seems pretty basic doesn’t it.  So your project should probably have a more detailed description that everyone understands (the stakeholders and the team working on the project). But in practice I have often found a simple summary of the project is enough to make do.

In fact my quote for the day is :

Projects where everyone has the same simple understanding of the problem do better than those where some of the team have a really detailed understanding that is different to what the rest of the team understand.

I was shocked and horrified so I wandered off.

Soon after I spoke to some business analysts who were struggling with an agile project. They alleged that the agile approach they were using did not allow enough time to capture the requirements properly. But guess what, they broke down under interrogation and admitted that a large part of the problem they were having was that they did not really understand the product they were building.  Apparently the evil agile people had forced them to write stories about the system before they had time to understand it.

So I started wondering – is their any way you could actually understand a system or product before you built and tested it. This sounds like something you would want to do even if evil agile people want you to go really fast. In fact the faster you need to go in building something the less time you probably have to spend on wasted effort that is not related to what you are building.

I have already described PAC as a simple approach to user analysis but that describes what the users do and not what the system does for them.

So here is an old school approach that won’t take long.

Start by looking at the project charter/mandate and then ask people what the system does. Then do a little exploratory testing if the system already exists.

Now you are ready for a quick one hour workshop (or one hour of sitting in a cafe if you are on your own).

Step 1 – book a room for an hour or go to a cafe and order a skim milk cappuccino

If you have some friends who also want to understand the product then bring them along. Otherwise sneak off to a cafe on your own.  You will need a pen and paper and probably enough spare change to pay for you coffee.

Step 2 – create a one paragraph summary of what you think the product does

You can write you summary in any format you like, but I like the following approach which is loosely stolen from a book called “Stand back and deliver”. They use a similar approach to describe a team, but I find it works well enough for a product. It is also similar to the cereal box approach described in “Agile Project Management“.

The product provides [something] to [someone] so that they can [do something better]. Without it [someone] would have to [fall back]

If you are in a workshop setting, have each participant write this without speaking to the others. If you are on your own then I guess you should just write it out. But only allow 15 minutes for this exercise (plus 10 minutes to share and argue about the answers if in a workshop). You have a lot to do and you want to have time to drink our cappuccino.

Example

For example, lets say that Jenny owns the coffee shop you are sitting in. She wants to set up a new “wine and coffee” club so that people can learn more about their favourite beverages, order premium beverages to consume on site and even order bulk purchases for home.

Jenny decides that she wants to set up a new database system to keep track of the members of the club and if she can get the budget she will also add a web page with a blog and a social community for club members.

What does Jenny’s new database do and who is it for?  She could build it for the members or for herself … does it matter?

Jenny comes up with the following summary:

The coffee club database keeps track of coffee club membership information so that we can manage the coffee club. Without it we would need to have a manual system in place or forget the whole coffee club idea.

Optional step 2a – add some old school requirements (3-5)

With stories being all the rage these days, people sometimes often talk about the dark ages when we relied on requirements documents. But hidden in all those documents we could sometimes find some really clear and concise requirements that really did help us to build what people wanted.

First we used to list some functional requirements (what the system is meant to do) and then if you want to make good products you add some quality statements or “non-functional” requirements to clarify the style, performance and behaviour you want the system to have.

Here is my favourite format for functional requirements:

The [system or module] will [do something];

Or you can go all out and add two more two optional sections:

When [something] [the module] will [do something] if [some condition]

You can write millions of these one line requirements, but for the sake of understanding what the new product does you only really need about 3 – 5 lines. More than this will often cause people to start simply agreeing with them. Only listing the top 3 requirements will often cause people to have to think about what they mean.

Similarly, you can spend all day on this, but I would only allow 15 minutes, plus 10 minutes to share if you are in a workshop. Even 5 minutes might be enough.

So let’s see what Jenny came up with.

The coffee tracking system will store data on members of the coffee club

The coffee tracking system will store data on the coffees we have available and the promotions we are running at the moment with respect to those coffees

The coffee tracking database will store a history of promotions we have run and purchases users have made where privacy legislation and PCI compliance allow us to do so

The web page will provide static data on premium coffee and wine varieties and a blog for publishing regular coffee related news

When members join the online club the web page will provide access to a chat room where they can share coffee related views if the member has a valid login

Jenny has more work to do in order to fully scope and build the system.  But she now has a one page description that she can use to check whether she and her IT vendor (Charles who works for free coffees) understand the system.

Optional step 2b – add a quality statement or measure

Even if a system does what we want, it might not do it the way we want it.  For example

When we get home, my flatmate will cook dinner if I do the dishes after dinner

This sounds good – but I have had some flatmates who are great cooks and some who can make toast and coffee taste terrible.

So it is a good idea to define how you will know if the new system is good or not. Here is one (not too complicated approach):

The stakeholders will think [system or module] is great if [something)

The stakeholders will be really upset if [system or module does/does not … ] or [something happens]

Jenny might come up with:

I will be really annoyed if the system is hard for me and the barista to use

Customers will be upset if we leak their personal data or use it to spam them

I will be really happy if the system can run on my new galaxy tab as well as my netbook

Of course you are allowed to do better. A lot of people use iso9126 (a standard for software quality) to list the non-functional things the system should do (be reliable, be easy to learn etc). I often use FMEA (a standard way to be scared) or FAB (a sales technique I use for non-functional stuff).

Step 4 – Go and find out that you are wrong

You would think that it would be hard to get a one paragraph description of a system wrong.  But on almost every project I have tried this on I have found that people disagree on it.

So the final step is to share this paragraph with the project sponsor, the product owner or the project owner. They will correct your understanding and often surprise you.

If you have this simple summary, I can almost guarantee you will be able to do a lot of things much more easily – requirements are easier and so too is regression testing for example.

The only downside is that it can make it harder for developers and product owners who want to build without having to think about what they are doing. And of course you might need to spend up to an hour in a coffee shop to do the hard part of the work.

Regression testing is simply the idea that when you make changes to a system you should test to make sure you haven’t broken what was already there. But for reasons unknown to psychologists, IT teams fall into a strange psychological pattern whenever asked to do regression testing.  This is the opportunity for someone to do a Phd – we understand that people consistently adopt the same four stage cycle of pain, but social scientists are at a loss to explain why:

1. Regression test denial.  People on most projects swear that there is no time for regression testing at the moment.
   • They will admit that in the last project they spent an inordinate amount of time (and frustration) near the end of the project trying to understand and fix problems that were created early in the project. They will also observe that most of these problems came to light as soon as regression testing finally started.
   • They will also complain about “the business” being indecisive and asking for new things every showcase (demonstration of the software).
   • The business will say their “alleged new change” was an obvious part of the requirements but the team will say they could not have anticipated it.
   • A lot of time will be spent fixing these new demands, but even more time will be spent arguing about them and demanding ever greater detail in the requirements, even though we still keep missing things and the business keep changing their mind. 
   • I accept that better requirements will address some of these issues and better code writing will address others … but I will also claim, even though people seem to deny it, that doing some good regression testing early on will save a lot of time and frustration through the whole project.
2. Robot dreaming. If you can get people to accept that testing is a good idea then:
   • They will start to dream about a wonderful, magical world where fairies come and do the testing for them. But not many IT people believe in fairies anymore.
   • Having accepted that magical test fairies do not exist, people will instead dream of robots doing all their testing for them. So they fire up Google and start searching for wonderful sounding robots (often called automated testing tools) to use.
   • But it takes a lot of time to Google for the robots and it takes longer to install and test them even if you do start to use one.
   • So someone still needs to set up the regression testing for the robot and someone still needs to do some manual regression testing until the robot is ready. This causes some sorry and a brief attempt to go back to step one “just for now”.
3. Tester and intern persecution.Finally we get to the real problem. We all know we should do regression testing but nobody in the team wants to be the one to do it. So now we finally start to tackle the problem.
   • The team instantly starts a search for interns.  If they find them this is fantastic because we can now inflict all the horrible regression testing on them.
   • If there are no interns then the team search for graduates, “business people who want to become BA’s” and other victims. 
   • If there is nobody inexperienced enough to trick into doing the work, then finally the team dump the whole thing on the usual victims in IT – the testers. After all regression testing is a testers role isn’t it?
4. Pointless suffering. Having been forced to do regression testing, the victim now sets out to prove that regression testing is a horrible waste of time.
   • The victim starts doing some simple testing by running the same test on the same functionality multiple times.  This finds few if any bugs.
   • The victim then starts running some functional tests on new functionality until reminded that this is not regression testing.
   • The victim finally, grudgingly finds some simple test cases and starts to run them each week for an hour or two.
   • Running the same simple tests for a couple of hours results in very few defects since what is being tested is usually the unimportant stuff or the stable stuff that nobody is breaking.
   • So the victim concludes that they are spending a lot of time for very little return and then finds other more important work to do.
   • The team return to step 1.

But rather than the “cycle of pain approach” I think it might be better to use scenario testing as a way to do regression testing.

The main reason is that this is the first approach anyone taught me and it worked. So I never really learned any other approaches. Not the best reason I guess, but a good enough one for me.

But the process I learned process involves more than just writing some test cases and running them. So it seems like more work, but in fact it is not. I hope to show that the approach I learned is actually quicker, more effective and more fun than either the “cycle of pain”.

The whole approach consists of four steps:

1. Understanding the context. Regression testing involves testing whether you have broken the functionality that was already there. So it stands to reason that you will test better if you understand what is already there, if and why it matters and who uses or cares about it.
   • Don’t panic.  This can be done easily. You can read the project mandate (or project charter) and you can talk to people.
   • In addition you can use some simple techniques to improve your understanding. I would suggest these as a pretty good starter pack
     • PAC
     • Using the international standard for being scared (FMEA- asking what could go wrong)
     • Creating a product summary statement
2. Scripting. You can do effective regression testing using only exploratory testing. But if you are doing the same tests repeatedly then it is probably worth writing down what you are going to do to save time when you are doing it.
   • Don’t panic. Some people think you need to build an entire library of scripts before you start testing. But you can start with a few basic ideas and then build on them each week as you go.  I will publish a guide to creating both slack and good scenario tests soon.
3. Making a map. You can run a lot of manual tests, but if all the tests are covering the same thing then you will be wasting a lot of time re-testing what you have already tested. You will also be consistently not testing the same things every week, which seems a little dumb.
   • Don’t panic. Your first go at this can be a simple list of the 3 or 4 scenarios you are testing to start with.  But as time goes on you will find it more and more beneficial to have some kind of map linking your scenarios to the things you should be testing.
4. Running some tests. Of course the point of the other 3 steps is to help you run some tests, so this step is pretty obvious.
   • Don’t panic.  You can run some tests this week and some next week.  As you go you will update your scenarios and your map so that you get more and more effective at testing in less and less time.
   • As you are running your tests though, you will learn more about the system and you will find that the context in which you are testing is changing. So go back to step one.

I guess the choice is up to you and the team.  The traditional approach (the cycle of pain) has been used for years by many experienced teams, so it is tried and tested. On the other hand, the creative testing cycle I have described here sounds a little radical and may not be as widely used. But I have found the creative cycle to be less painful, more interesting and more effective for me.

The problem with production support is that nobody has time to ask for what they want, but it is all urgent and super critical. So the last thing you often feel like doing is to slow down and understand the context that the user is in (annoyed, relaxed, sitting in a cafe, in their most important sales meeting ever etc). Instead the focus is usually on fixing “it” before you know what “it” really is.

But this often leads to rework. So whenever I am doing enhancements I always spend a little time understanding why the enhancement is needed, who it is needed by and when/how it is likely to be used in the real world. To do this though, I think we always need to spend a little time understanding the people who will use the system and why they need something new.

There are many approaches to doing this, but one I often find useful is “PAC” or “People, Activities and Context”.

PAC refers to a series of steps to understand:

• The People using the system;
• The activities they will perform; and
• The context in which they will perform them.

The steps are quite straightforward:

P: People (Define the users)

Let’s start by breaking “the users” into three groups:

• Those who actually use the system for the purpose we built it for are the Primary users.  Most regression testing is likely to focus on these users;
• The often forgotten Supporting users are those who interact with or use the system for the purpose of helping the Primary users to use it. For example production support or on-site support engineers; and
• Other stakeholders are those who do not directly use the system but who gain a benefit from the system or have a say in what it does.

Example primary users might include:

• Experienced and inexperienced users
• Wholesale and retail customers
• Users of different versions of the software – Basic Edition, Advanced Edition and Enterprise Addition for example.
• Local users and overseas users
• Onsite users, home users and mobile users

In each case you list the types of users and then ask yourself:

• Do they use different functionality?
• Do they use the same functionality differently or need a different experience using it?

A: Activities

The next step is to list the typical activities that the user might perform, that might impact or involve the system.

The trick here is NOT to just go through the requirements. Instead the focus is on the user and “what they will do during the day”.

So we might have a requirement for a user to login and another requirement to process member enquiries. But we might actually put down the activities of “talk to customers on the phone” and “send a report to the boss”.

Sending a report to the boss might involve running a report from the system (and thus match a requirement) or it might involve collecting information from several sources, including the system we are building.  So the user might actually be logging into our system to run some general queries or to just look at some screens.

Similarly, several users might do the same activity, or one activity might involve multiple users. So don’t worry about linking the activities to a specific user yet, just put them all down in a list.

So the idea is to capture “what activities does a user get involved in”.  These can then become the basis for requirements, test scenarios, stories, use cases or whatever you want.

C: Context

The final component of PAC is context.  This is important because the same user might perform the same activity very differently at different locations, using different toolkits or even in different emotional states.

So context can cover anything about the user’s environment or situation, for example:

Physical environment:

• Will this be done at the office, at home or on the road.
• Some users might be in a major company while others are in a small office.
• Some users might be in the outback in Western Australia while others are in a major city in Japan.

Technical environment:

• Will all users be using the same operating system, browser version etc. I find in particular that customers assume we support all browsers in all versions but IT. departments only test with one browser, assuming any others are unsupported.
• What hardware will people be using?  Will this impact their response times or ability to see our graphics.

Emotional state:

• During a disaster might be a different process to relaxing at the office.
• The user might be using the system in front of a complaining client or after work.
• when there is time to wait for reports to run.

Time

• Is there a specific time when things are harder or more urgent, for example peak hour or the annual enrolment week at university where things will be both more important and more likely to fail;
• Are there blackout times like overnight backups, annual reporting periods or change freezes?

Bringing it together

Now that you have collated all the information you join the dots. So you look at each user and ask yourself whether they will do each activity and in what context they will be operating.

You can put this all in a table if desired:

Of course completing the table is not the point.  The real point is to use this information to create stories, requirements or tests.

Unfortunatley there there is often little or no automation in place at all when you join a new team and there is often not enough time to get it working properly before the next release. 

In fact though, even when there is a lot of automated testing in place, I have sometimes been surprised how quickly a good business person will find a significant bug by simply sitting down and mucking around (playing) with a system for an hour.

Exploratory testing

So I think there is a place for “mucking around” testing every week on a project. This informal approach will test for the things the team never thought of and will often reveal surprises even if it is just a developer or random business person doing the testing.  And often, those surprises will be missed by consistent and formal testing even when it is properly automated.

Of course when a real tester does “mucking around” testing they like to call it “exploratory” testing to make it sound more legetimite and scientific.

Exploratory testing is, in fact, exactly the same thing as “mucking around with the system” to see if it breaks. But the exploratory testing done by a master tester, compared to a random business person, is like the difference between the work done by a master painter and that done by an enthusiastic child.

I will not attempt to cover the numerous techniques, heuristics and instincts that a master tester (or painter) needs to learn to master their craft, but I will say that regardless of whether you are an expert tester or a rank amatuer, exploratory testing can be fun and surprisingly effective (yes – even for IT Managers like me).

I will also say is that any effective approach to testing, including regression testing, involves an allocation of time to “just mucking around”. This often opens up problems you didn’t want to find, but that is better than going live with the same problems.

A good description of exploratory testing can be found in the following article if you want a better definition http://www.satisfice.com/articles/what_is_et.shtml.

Exploratory testing can be boring if you just press the same button all the time. But it can be more fun if you just muck around with the system and try to do “the stupidest possible thing a user can do”, or “see what would happen if you tried to enter Rupert Murdoch into system and see what the process will do with a salary of $120m or so per year”.

If you allow yourself time to be inventive and creative, then you will probably stumble on a surprising number of bugs … and potentially even requests for scope changes.  So this can be a pretty effective approach and failing anything else you will find some of the bugs AND you will find it will even be enjoyable if you do it properly.

But even then I think we can do better.  So there must be another option, somewhere between using robots to do your testing (ie automating testing) and mucking around having fun breaking the system (exploratory testing).

So my next quest is to find something beyond exploratory testing that will still add value when manually testing and will still not suck too much for those doing it.