When methods can work

There are methods for almost everything you can think of, also in the field of software testing. What often is forgotten is that a method is just a representation and simplification of reality. The danger I see is using methods without questioning that method. Without knowing and checking the boundaries of a method you make unwillingly the assumption that the situation you are in is exactly as the method represents.

I hope you agree that nothing can be identically as described in the method you are tending to use. If you think so you have not only to search for the boundaries for the method you are trying to use and implement, you also have to investigate your own situation much better.

Often I see that people are familiar with methods they have been taught. Over the years they implemented and relied on them, they created some best practices for themselves. In the beginning they asked other people if they understand it correct and used it right. At that moment they were questioning themselves, the method and the environment they were into. After a while it became just a trick they performed and implemented methods the easy way, not questioning anything. I see this as a risk using methods you know. You are no longer fitting the method to the organisation; you are trying to change the organisation to fit the method so you can be successful.

At school I bothered the teachers with asking questions about the method, claiming that those are not useful. Looking back I questioned the method to understand how it works and where the boundaries are. This made me help to implement and use it when possible and triggered me when I tending to cross those borders when implementing them.

It is easy to say that a certain method is not useful; it is harder to tell what parts can be useful and what parts can be of advantage for you.

I suggest that instead of starting with methods or best practices, first start asking questions to get a picture of an organization/ project. Based on this you might be able to create a model for your own of that organization/project. Now you have two models, one model of the organisation and that one called method. Combining this with the known boundaries of the method you might be able to successfully use the method or parts of the method and become successful yourselves.

Different metrics

One of my hobbies is playing with MS Excel. I do this at home, at my assignment and in the past I learned it on Experts-Exchange.com.

The power of MS Excel is collecting data and transform it into information. Initially I tried to create a dashboard which I can use in every project. Unfortunately this was barely possible. Each time, the challenge was to translate the provided data in such a way it represents the same overview of tables and charts. Also requests for information from management differed each time.

I noticed there is a difference between information I need to control the test process and information management need to get informed.

As a project is dynamic and also data is dynamic during the process I start always easy and make the information I can provide also dynamic.

Information for me would be:
- # issues found
- # issues found in combination with system component
- # number test cases /issues open for test
- # test cases /issues in combination with location (what is the status and who should act
- percentage ready (when possible in relation with time left)

Often the difficulty is that information I need is stored in other applications. When possible I download them. Disadvantage can be here if the structure of information is changing or objects are changing in names it become hard to compare. Therefore I start always easy and let the dashboard grow when necessary, or reduce when possible.

Personally I like tables with data, because it enables you to discuss how to understand the information. This is often a complex process, not only you have to explain what information is there, you also have to explain that the data which is shown is correct. Often you can draw several different conclusions out of those tables.

The pitfall here is that this is sometimes hard to explain to management because the like charts. The risk of charts is the discussion behind cannot be shown, information might be missing, people can be mislead. In certain cases I like to keep it simple and fancy.

Here some examples I used over the years:

- the progress meter, can be used per process and for a total. For detailed explanation see: Peltier's Speedometer

Another chart I use is a traffic light to show how the testers feel about the quality, this is often supported by written remarks. Although it is not objective, you still show the feeling and possible threats.

Another daily chart I often use is to see the location of issues. This is a chart which changes on daily basis which I use in combination with available time

A chart managers often like is a chart planned vs actual. This is often hard to get as the data for planned numbers of test cases depends on the availability of testers and also if they are dedicated available for testing. Another remark can be the origin of planned figures. As planned is different then intended.

One of the powerful functions in MS Excel to obtain data unstructured sources is SUMPRODUCT. This one I use often above pivot table, as they avoid growth of workbooks with MB's. Using them wise full, it also enables you to change when data sources are changing.

My advice is start simple and avoid using standard templates as each project will be different. It cost more time to change a template based on provided data and information then start from scratch. Working this way you avoid that unnecessary data has to be maintained and management is informed with non-required information.

BVA test case generator

Recently I read a post about a online tool which can generate test cases based on boundary value (BVA). After a reader of my blog also attended me on this tool I made the free subscription.

On TestersDesk.com you have several tools which I think can be useful like:
- Pairwise TestCase Generator
- Boundary Value TestCase Generator
- N-Way/Random TestCase Generator
- and some more

For example the BVA++ generator is very easy in use, although it doesn’t cover the boundaries we don't know it can be useful .

If you know the values you want to test it is just minutes work to generate the list of cases you need. (within seconds).

I believe this tool can be useful if you have less time available and there is less risk in the functions you are testing. You can use it for:
- generate fast a list of cases you need to cover
- calculate how many time you need to perform those cases. If the generator gives you about 25 cases and you assume that each case will take you about 1/2 hour to execute, you at least know that you might need about 12,5 hour for execution. It can help you to identify in a few minutes how many time you might need to test a function
- the generated cases can also be used to check whether the already defined cases are sufficient.

One of the items I missing here is that you cannot get a list of the total cases so you are able to pick from the total list (if it is there I have missed it.)

How many documents can you create?

Often I hear the saying a good tester is a lazy tester. This doesn't mean that testers don't want to do nothing; it is just that testers just want to do just that what is necessary. Like why execute all 1000 test cases if based on test techniques these can be minimized to a number of 52?

Over the years I noticed that to control projects a lot of documentations is created. This made me curious about which types of documents I had to deal with or might have created if I follow the methods like PRINCE2, ISTQB, TMap or others.

Here a list of documents I had to face with:
- Project assignment document
- Initial test advice report (before actual testing starts)
- Master test plan
- Risk analysis report
- Product Risk Analysis Template/ checklist
- Product Risk Analysis report
- Detail Test Plan (for each phase one: SIT, FAT, UAT, PAT)
- Test strategy checklist
- Test strategy document (finally it would become part of the MTP and DTP)
- Weekly progress report
- Daily progress update report
- Daily progress templates (to gain information from for the daily progress reports)
- Deviations report
- Phase end report
- Escalation reports
- Logical test specification document
- Physical test specification document
- Test Script
- Test process dashboard
- Intake test basis checklist
- Checklist intake test tools
- Checklist test process evaluation
- Test Process Evaluation report
- Test Process Improvement Checklist
- Test process improvement report
- Checklist test techniques
- Test schedules
- Test estimation document
- Defect registration
- Defect screen print templates
- Test Execution Quick reference cards
- e-mails
- Review template
- Review report
- .....

Of course there are some more documents I had to face with. Looking at this list I just wonder: Is there still some time left after generating these documents to actually perform tests? Of course these can be all useful documents. We should continue asking the systems and the processes questions and preserving the outcome of it. Only: are we working for those documents or are those documents work for us?

Why test management tools can cause pain

I'm aware of management tools who claim to support the testing processes. Some of them can be very useful because they can provide information about the number of defects found, number of test cases created and executed, coverage of those cases related to requirements, and more.

This morning I asked myself why I start projects most of the times using MS Excel. In most of the cases it is because of the lack of test management tools. On the internet you might find all kinds of whitepapers why we should use these tools. It can be easy to convince people why to use it. It might be harder to explain people when to use it. It might be even harder to make people think why not to use it.

Here some points which came up in my mind why it wouldn't be wise to use tools:
- When management didn't decide yet what information they need to make decisions;
- When projects are successful because they have the flexibility to adapt their development process and test process based on their situations: multiple test approaches are used;
- When it takes more time to convince users to use the tool and actually using it instead of working (i.e. testing);
- When people start trusting on the information provided by the tool instead of using common sense;
- When the tool provide too much information which distracts people for the real goal of the project;
- When the tool is used to punish people instead of measuring progress and risks.

I think it is important when certain tools are used they are not dominating your test process and forcing you to make decisions how to act because it won't fit in the tool. Tools shouldn't dictate, it should guide and help. They also should leave space for creativity. Management should be aware to judge the information provided by those tools any time.

Two heroes dealing with uncertainty

Last weekend I was surprised by the persistence of my children which made me proud. Even thinking at that moment how they dealt with the situation makes me still give the proud feeling. Perhaps there are some similarities between their situation and projects we have to deal with.

A few weeks ago my son asked me to join a family event organized by his sport club. The event was a cycle tour through our region. As my children like to cycle in our neighborhood, the thought this was a good idea. So we decided to join that event without knowing what situations we would face. My son is 8 years old and my daughter 6 years old and as normal healthy children they complain whenever. I already calculated that a tour of 20km would be possible for my children to succeed without fewer complaints.

When starting the tour I asked what distance we have to cycle. The organizations claimed it would be between 30 and 40km. After a short discussion with my wife we decided that it would be acceptable although it was raining a bit.

During this day we had to face some challenges:
- not knowing what route to follow as after each track ended with a new piece of the map
- not knowing where to finish
- not knowing what whether to face with
- not knowing what quality the roads would have
- not knowing the unknowns on the route

During the day my children didn't complaint at all, although there were reasons to do. They already made me proud as at the end it turns out we cycled over 60km. Cycling this number of meters is already hard for a person who didn't touch a bicycle for a few years, imagine how hard it has to be for those children on those little bikes.

Afterwards it made me think why we succeeded and why they didn't complaint.
The following reasons came to my mind:
- They accepted the uncertainty
- They trusted me as guide
- We helped them by pushing them when wind was tougher or roads are steeper without they asked for it
- We listened and watched when we assumed they needed a short break
- We called the organization when we got lost
- We decided not to participate in the last track and go straight forward to home. Although we had to cycle still about 10 km. Cycle straight home is better then keep playing the last game with the risk to lose direction and stop more often which would cost a lot of energy
- We rewarded them by telling how good they are doing, how proud we are
- We focused on the beautiful environment instead on our burdens

When looking at these reasons I see some similarities with projects I delt with. Perhaps you also were in projects when things are getting tough people are starting to complain. To avoid this perhaps the following actions can be taken by the project manager/ test manager:
- Make people aware of a certain uncertainty so acceptance is easier
- Make it visible when you ask for help and for what reasons
- Help your project members before they start complaining, reducing the pain before they feel it can help avoiding focusing on that feeling
- Make the team focus on what is happen in the big picture instead of focusing on what goes wrong
- Don't stretch their capabilities for to long, give them a break, although there is time pressure
- Keep telling them when they do things right
- Be open, don't focus on the problems you have as management, keep eye for their situation
- Make the fun happen

There is more behind

When I am allowed to talk about testing I can get thrilled about this topic. People can see in my eyes that I love this profession. Over the years I met several people I had to work with. Sometimes it was easy to convince them about the meaning of testing and sometimes it was hard to test together.

I think there are several types of persons you will face during the job who need different information

1. People who have not any relationship to testing and are interested in what you are doing
2. People who are ordered to test, without knowledge about testing
3. People who are asked to test and want to learn about it
4. People who want to understand what you are doing, without testing
5. People who are dedicated to testing as you are and want to share knowledge
6. People who are dedicated to testing as you are and don't accept your knowledge
7. People who don't want to do anything with testing and just want you to do it
8. People who don't want to do anything with testing and who don't care

Talking about testing cost time and effort. I think it is important to spend those resources wisely. Therefore before you share information you should identify those types what type of person you are facing with. It will help you to avoid losing those people and spending the wrong effort.

If you acknowledge these types you can train your selves as the message will be different for each type. In your group of stakeholders there might be people who have something to say only don't have the time to listen.

Art, memorials and software testing

Last weekend I was in Berlin with a couple of friends. Just to celebrate our friendship. One of the things I saw there was the Jewish Holocaust Memorial. It is an impressive huge monument of pilars.





As I don't read manuals when installing software so I also didn't read the introduction of this monument. The process of experiencing I had of monument made me think about similarities within software testing.


Initially we stood there wondering what to do. We noticed standing outside the monument that it has a beginning and at the horizon we saw there was an end; although we couldn't see the last stones. It has also sides and you can enter the monument from any position. Another observation was that the pillars stood in obvious straight rows and in line; they seem to be gray and of different sizes. From the point I was standing I could only oversee a small part of what is happening in the monument as not all paths were visible.

Except of just standing there and accepting the monument as it is we tried to learn more from it. We got the idea just to step of our rented bicycles and take a walk across the field of pillars. At that moment there was no defined plan. We just walk to see what is at the end.

Initially I walked with 2 other friends telling each other what we saw which we couldn't see standing outside the monument. Some findings were like: the pillars are not straight in line at all. The bricks on the floor might also be part of the monument. The pillars don't have all the same size and same gray color. The floor is not equal, although we noticed that it had some curves from out starting point to the obvious end. It curved also sideways. Between the bricks on the floor there were lying small stones of almost equally color.

After a few minutes walking I did what most human beings would do, I start playing. While we started walking together in the same direction it was very easy just to step to the left and "loose" the friends, walk a bit faster and pop up just a few stones ahead while they didn't noticed it. This was the trigger, they started playing along. In the beginning we had no direct goal, only to see if we were able to surprise each other showing up on a different position.

In the beginning we knew in which area the other was, when stepping behind from some stones or crossing other lines and catching a short view of the shoes or jacket. After a moment we were so busy to surprise each other we lost each other. This brought me back to the monument and I made my own context what the monument should represent. I realized how easy it is to loose dear friends with just a small action. In the beginning you know were they should be, after a while you just know they are somewhere, and at the end you don't have nothing; just the believe that they are still there.

After I lost the friends in the monument I tried to find them. I figured out a plan how to approach the web of pillars, walking different routes with different speed. Assuming that they are going from beginning to end I started to walk from left to right. I also deliberately walked the same route back and also backwards. Just to get some information which should give me a clue in which part they were walking. After a while I accepted that I lost my friends in the maze and continued walking alone.

At that moment I noticed I went back to my initial goal: to learn from and experience the monument. Based on my gained knowledge I reshaped the context of the monument. It is not just a pile of stones lined up. It has also the ability to loose people (information) or to gain information (meet other people). I now walked between the pillars just to see what others are doing. I watched their behavior in the center of the monument and also at the borders. I now noticed that some people are doing the same as what I did with my friends. Others were just walking through and talking to each others. A few were using the paths to get to the end and at certain places people were just sitting thinking about the monument or enjoying the nice weather.

That day I learned a lot. Therefore I'm writing this blog. What I learned/did was:

1. approaching a object not based on best practices, instead asking myself questions what could be the purpose of it and what can I do with it. Thinking about possible value
2. approaching the object with questions
3. observing what I see while I experience and trying to redefine new questions
4. what initially wasn't there can be of importance later on like the difference of depth in the object
5. rules changing: initially we were just walking, then trying to surprise each other, losing each other in the game, accepting the lost and redefining goals
6. context is changes after a while
7. items change of importance: the little stones between the bricks. At a certain moment they had some value. Only at that moment when the thought arise they keep the bricks together
8. Different moments need different questions and lead to different observations
9. I would have missed a lot if I approached the object with a fixed plan based on best practices
10. I enjoyed being there, walking there, learning from it.

Sun Tzu Art of war and software testing

Finally I found some time to start reading Sun Tzu's The art of war.

In one of the first pages I noticed some text which might explain why it is hard to manage a test process.

"A general must see alone and know alone, meaning that he must see what others do not see and know what others do not know. Seeing what others do not see is called brilliance, knowing what others do not know is called genius. Brilliant geniuses win first, meaning that they defend in such a way as to be unassailable and attack in such a way as to be irresistible. (Thomas Cleary, page 7, Sun Tzu: The Art of War)"

Imagine that you have three generals: one general from business, one general from development and one general from software testing.

As a general from software testing you stand in between the war zone from the other generals. They expect you to provide information so they see what you see about the system and the process. For example: does the process to create the system contain enough information to meet the requirements. They also expect you to provide information so they know what you know. For example: are there some defects in the system.

If the software testing general has to give the views and knowledge away, he can barely be a brilliant genius. Although this is often what the other generals expect from him as he should give positive news about the quality and an advice to go to production. In this case he will give away his knowledge and views away.

To me it seems that we should keep providing this information which will lead that no one will see you’re a brilliant genius. Also you cannot fulfill the role of a software testing general as you will not be successful when you have to give away all information.

Magazine: Quality Matters, now with contribution of myself

Recently I send in an article with a though I had in my mind for some while about improving test processes. Sometimes it is better to neglect improvements how obvious they are.

In the 2nd edition of the magazine: Quality Matters my story is presented under the title: "Be innovative! Stop improving your test process."