Keep thinking about testing

Although it is my holiday I cannot help keep thinking about testing. Perhaps it is good to continue thinking about testing even when you don't have to work. Currently my thoughts are about the projects from the last years. Somehow during the last days of the year people over think their good, bad and ugly moments.

In this year I was able to attend several conferences like Software & Systems Quality Conference in Germany, Dusseldorf;EuroSTAR 2008 in The Netherlands, The Hague; TestNet Najaarsevent in The Netherlands, Nieuwegein. The main subject of those conferences was about the future of software testing. And that future can be challenging by using Agile approaches, Model Based Testing, more tools, Business Chain Testing, Security Testing, new test methods and much more.

During this year I also read more weblogs like those from James Bach, Paul Gerrard, Michael Bolton, Matt Heusser, Elisabeth Hendrickson and many more weblogs. I also spend time reading books related to software testing like: Testing SAP Solutions, Tmap Next, TMap Next BDTM, The Complexity of chain testing, Software testen in Nederland, SmarTest, Business Process Validation, Kwaliteit in ontwikkeling, Testen 2.0. And some are still lying there to be read.

Occasionally I spend time on the web searching for more information as well in magazines, articles as submitted videos.

Did this all make me a better tester? Do I want to be a better tester? Did it help? I like the word I learned in the workshop: Certified Scrum Master: "It depends".

And that dependency made me keep thinking about software testing although it is my holiday. Only this time not for new ideas. I made me think about this year. In the year 2008 I posted some articles on my blog to write down some of my thoughts which might help me and others to think in another perspective about software testing. Below you see a short retrospective of my year 2008.

One of the thoughts I had was using the congruence model related to open system thinking in identifying other risks and objectives in the test process. I described this model on macro, meso and micro level. Although it is not yet completely finished it led to the following postings.
Open System Thinking and Software Testing (1)
Open System Thinking and Software Testing (2)
Open System Thinking and Software Testing (3)
Open System Thinking and Software Testing (4)
Open System Thinking and Software Testing (5)
Open System Thinking and Software Testing (6)
Open System Thinking and Software Testing (7)
Open System Thinking and Software Testing (8)

As I like to play with other models to see if it can be used in the testing profession I also wrote posting like using the ServQual model: Service Quality Model (SQM) and Software Testing

Or summarizing the impact of development models in test process: Development Models and the impact on Testing.
Even the Cultural Union I used in a article: Success of improving your Test Process depends on culture. One of the recent postings was using the mining method: Rooms and pillar approach in software testing: Room and Pillar approach for software testing

I also used this weblog to express some of my ideas related to a new quality attribute called: "Ecoliability" :
The green part of development and software testing: "Ecologiability"
Introduction of the quality attribute: Ecoliability
How to deal with "Ecoliability"

or terms like Request For Knowledge (RFK):
Request For Knowledge (RFK)
The RFK process explained
RFK and knowledge processing

or Organizational Testing Board (OTB), and some other life cycles
Software testing: An organizational approach (1)
Life Cycle of a Tester
Product User Life Cycle (PULC)

And for some fun I tried to use some metaphors to explain a bit of testing in postings like:
Beware of the auto-pilot
Software testing and Astronomy
Mikado Management or Test Management
Software testing compared with making music
Testing your software like building a house
Tetris as Test Management tool
Flying in a hot air balloon
Shuffleboard

Did I like to write those postings? YES! And I will continue for the next year. Perhaps other people are willing to help me in the year 2009 by responding on my postings and discussing the items. I hope that I can continue learning and expressing my ideas to shape my thoughts. For those who read my blog occasionally I hope you enjoyed those postings. This will be my last posting this year. I wish you all the best for 2009!

Open source book: Software Testing Guide Book

Usually you see information in PDF format so it is sometimes hard to reuse. Accidentally I ran in a project were writers from over the world are creating a Software Testing Guide Book

Download the Open Office Document version here.

I have to admit I did not read the whole document and therefore I'm not able to say I support the content of this document. I think it is a project worth to mention.

Magazine: Professional Tester is Back!

One of the first magazine related to software testing I subscribed to was Professional Tester.
I enjoyed reading this magazine. Only after some while I didn't receive any issues.



I noticed that the website still exists so now and then I take a look at this site. to my surprise and joy I noticed the site has changed and there was a banner: Professional Tester is back!

Currently they didn't publish the editorial calendar. So I will keep monitoring this site waiting for the next issue.

Magazine: What is testing.com

During a short search on the internet I ran into this magazine: WhatIsTesting.com

The first issue was launched in April 2008. And after the free subscription I noticed that there are not yet other issues released.


I'm not quite sure what the value is of this magazine as on the same site there is a link to that other magazine: Experience Tester which already published their 4th issue.

Perhaps they combined forces; otherwise it might be another Magazine: What is testing.commagazine for us testers to get information from.

If something cannot be tested

So now and then I hear some sounds from testers or business when I ask them to test something that they are not able to test that something.

Somehow this sounds very strange to me. I believe that everything which can be build, can also be tested. If the business were able to find an error and the developer is able to solve that error then we should be able to test the solution. So who is right and who is wrong?

I believe in my testers. My testers are not unwilling to test. They are just not able to perform the tests. And I just want to know why.

Some reasons they gave are:
1. The test system doesn't contain that situation and I'm not able to recreate that situation;
2. I don't have sufficient rights to create useful data for the test;
3. The solution is too technical and therefore it is hard to understand the problem;
4. We don't have that problem anymore;
5. I don't have time to test and this is a minor issue;
6. I know that the solution is not what I need and therefore I'm not spending any time for testing on it;
7. This problem is not my problem; I'm not able to help.
8. Who are you?

I also believe that I should convince them to test, support them to test and during testing, facilitate them to test and make the business accept the risk when they are not able to test.

If they are not able to create the data, I should see if others are able. Mainly it is because of restrictions in the system. If giving them authorizations to the system to provide useful data without messing up existing data, then this should be the solution.
If they don't have the necessary technical knowledge needed to test this, I should help them with a person who has that knowledge and teach them or help them during the test.
If they know that the given solution is not what they need, I can bring them in contact with the developer and I also have to identify how this misfit occurred.
If they don't have the time, or it is not that important anymore I should search for other ways. A solution migh be embedding the test in another test case.
If they don't know me, I might have the wrong person and should look for another tester.

Even after all kinds of actions the tester might not be able to check whether the problem is solved. Does this mean that they should not test? NO! They should at least check if the functionality which is connected to the solution is still working. Should I close the ticket after those tests? NO, not immediately, first I have the business to accept the risk that the problem might not be solved and convince them that at least the proposed solution will not harm other processes.

Article: The Economic Impacts of Inadequate Infrastructure for Software Testing

Some now and then I find on the internet an interesting article. This is one of them:
Planning Report 02-3: The Economic Impacts of Inadequate Infrastructure for Software Testing

Prepared by: RTI for National Institute of Standards & Technology Program Office Strategic Planning and Economic Analysis Group, May 2002

This document gives some nice examples and depth on software testing.

Service Quality Model (SQM) and Software Testing

ServQual model
One of the models I cannot forget is the ServQual-model (*1) . It has been thought to me on school were the focus lies more on measuring the quality in providing services. In my opinion one of the strengths of this model is visualizing the possible Gap’s which might danger the quality of delivered services.

Wikipedia: SERVQUAL-model : "It measures the gap between customer expectations and experience."



The Five Gap’s explained
GAP 1 – difference between consumer expectations or quality determinants, and management’s perception of such consumer expectations
GAP 2 – difference between management’s perceived quality determinants and service specifications (i.e., the critical-to-quality specifications)
GAP 3 – difference between quality specifications and actual service delivery
GAP 4 – difference between actual service delivery and the company’s external communications about services delivery (e.g., word of mouth, past experience, promises, reputation, standard of care)
GAP 5 – Difference between expected service and perceived service

The difference between perceived service and expected service is a function of four different gaps GAP5 = f{GAP1, GAP2, GAP3, GAP4}

ServQual model in Software testing
To me this seems similar what we are doing during testing. Normally in software testing we refer to development models to fit our process to the development process. If we approach software testing also as a service and tune the model a bit towards system delivery we might come up with a model like shown below.



The five Gap’s for software testing
Gap 1: User Expectations - Management Perceptions Gap
Gap 2: Management Perceptions - System Quality Specifications Gap
Gap 3: System Quality Specifications - System Delivery Gap
Gap 4: System Delivery - External Communications Gap
Gap 5: Expected functionality- Perceived functionality Gap (or the System Performance Gap)

Challenging the Gap’s
As in software development we have already methods and tools to measure these gap's and control those. So there is nothing new here. What might be new is that instead of keeping the information within the project you bring them to the business. You make your project more transparent by translating development into provide service from the project.
Assuming that these Gaps’s cannot be avoided, transparency can help to increase the level of acceptance by the business.

Examples of activities measuring the Gap's
Gap1: As you see that the distance between the start and ending point is large customer involvement should be allowed and increased. This will help to identify differences in expectations of users and project management. Also a process of defining SMART requirements which fit's to the development approach should be established.
Gap2: Measuring the translation of requirements into written functionality should be one of the basic activities of testing. Only most of the time we avoid the reviewing process of requirements and designs. If these processes are in place we log issues for it when there is no match or functionality is unambiguous.
Gap3: Measuring requirements to delivered functionality is on basic activity of software testing. Here we actually executing test cases on the system to check whether it match the written expectations.
Gap4: This is one of the gaps we usually forget to manage. Only this might be one of the important ones when you check the relations to other activities in the model.
Gap5: Often we see this gap be controlled by performing the user acceptance test. In my opinion these activities will be strongly influenced by other activities from the model. Therefore this might not be sufficient.

Lessons we can learn from this model
1. Business involvement is very important and should exist during the project
2. Requirements should be reviewed and omissions should be communicated not only to project members. Also if some requirements will not be met, inform the business to change their expectations.
3. If issues are found, be honest about this towards the business. Explain what you are going to do about this and when. Every system has its failures, business can understand this. You might use the approach to solve this to build on the confidence of the users. They will be able to match the gained functionality towards their needs. It will also help to manage their perception about received functionality and by being transparent, their trust might grow and they might forget about negative experiences from the past.
4. Transparency should be their. Don't undervalue the power of word-of-mouth communication when users are involved in the project and they are not being heard.
5. Keep track of changes in functionality and also of communication. Perhaps start also maniging your workarounds (see: Start managing your Workaround's!)

For sure there are other things we can learn. Perhaps you can come up with some things. Just keep in mind that this model has its strengths and weaknesses.

(*1) Parasuraman, A. Valerie A. Zeithaml en Leonard L. Berry (1985) "A Conceptual Model of Service Quality and Its Implications for Future Research", Journal of Marketing, volume 49, Number 3 (Fall 1985), p41-50

Are you ready to deliver?

I don't think this situation is different from others. There is a business who wants a solution, there is a manager who needs the business having this solution, there is another manager who is responsible to get the good solution in production, there are business users who are made responsible for test execution and there is you. You are responsible for the test process.

What are you going to do when the solution must be tonight in production?

Some people are spending time to identify the possible risks. Others spend time to convince certain managers that it won't happen. Perhaps there are testers who just start testing.

I'm sure that all situations will fail. Normally we start identifying risks related to what should not happen in production and we will try to measure that those situations will not occur. This process of identification takes too much time of the 4 hours you have left. You will have the risk identified and just a couple of hours to check if those risks are covered. And afterwards you didn't fill in the complete picture although you created the expectation that after testing all risks are measured and identified.

Convincing the managers that you are not able to test would another approach. Only this will not help the business.

Start testing saves some time, only how will you know you did the right thing?

A previous test manager of mine told me that if business is not able to tell what should work or what risk should be avoided, ask them which errors they don't want to see.

I can imagine that you identify together with the business what value the new solution will have.
For example:
- The new screen enables the user to approach information faster which helps the user the process the customer information easier and therefore the service level increases.

I can imagine that translating this into errors the business wouldn't like is hard. You might enter the zone of negative testing.
You might start with the primary functions in the process like:
- Printing an annual bill;
- Sending a confirmation letter;
- Correct storage of data;
- ...

You might think of all kinds of variations of printing a bill. What about proofing that the user is able to print the annual bill for that client and is not able to disturb printing using that screen for other clients?

This could be translated in errors to avoid.
I don't want to see that:
- The annual bill for this group of clients is not printed
- The confirmation letters are sent more then 1 time
- The confirmation letter is send to incorrect people
- The data which is changed or created cannot be seen after returning after a next login
- The user spend more time to identify what is happening on the new screen

Will this avoid all risks? I don't think so. Will this help? I believe so as you minimized the number of test cases just by defining together with the business what should not happen. I strongly believe that in a half an hour discussion the business can tell you which disasters should not happen as they already have experience with it. It will be harder to identify all items which should work.

Will this lead to a solid advice on quality? Sure it will not. It will give the business an indication about the quality. If you tell them you are not able to cover all points within the given timeframe to make a solid advice but you might give them indication what might happen and based on what this indication is made, they might be able to accept this approach. And afterwards they are able to tell if they will accept the risks when you are finished. This might lead to a situation were you are finished within time and the business has tghere solution.

I suggest that a quick action force is available when things are put into production and monitoring if the errors defined are not happening.

Beware of the auto-pilot

Today I drove to my work and during this trip I received a text message from a friend warning me for a traffic jam. Based on this information I decided to take another road to avoid this traffic jam. Instead of sending him a message back I took the phone and called him to thank him for this warning. The coincidence was that just 500 meters before I had to drive straight forward I choose the turn I usually do. Just after 25 meters driving I noticed the mistake I made. Only there was no way turning back. I had to continue and was stopped by the traffic jam of eleven kilometers. This cost me about 40% delay.

The best part was that it made me think about this situation. I realized that this also could happen to us. When we are testing systems we know very well, we also turn on our auto-pilot. The good part is that in normal situation we are able to perform out test much faster then we did initially. The worse part of this will be when we get distracted we also might take the usual route instead the intended route based on new information.

I think the lesson learned would be to focus even harder on decision points in the system under test when things are getting a routine.

Testing Games

This morning started good. I posted an article about a possible relation between Rooms and Pillar Mining and Software testing. I found some time to play a game on the Wii and now having a short break. With my thoughts partly on testing and partly on playing games, this though brought me to Google on games related to software testing. Here some I found:

Bug Fix Bingo by K. J. Ross & Associates
Planning poker by Mountain Goat Software
Non-approved test methods by Charles K. Kincaid

Perhaps you can extend that list of test methods.
While writing this blog I hoped I would find more examples. Unfortunately, I didn't. So here some thoughts of mine to use when you want to do something with your time only there is nothing left: Held a competition with your colleagues to check:
1. who is able to find a bug in already tested software;
2. who is able to perform as fastest a number of test cases from beginning till end
3. who found the most issues and present it in a fancy chart
4. who is able to construct a data set in such a situation that your colleague must use functionality to correct the data and be able to finish the test case.
5. Make funny photo's of your colleagues impersonate test methods

Article: Acceptance Test Driven Development (ATDD)

Elisabeth Hendrickson’s posted on her weblog Test Obsessed An article related to Acceptance Test Driven Development (ATDD): an Overview

Here she links to an article she wrote about this topic which she presented at STANZ 2008 and STARWest 2008. Driving Development with Tests: ATDD and TDD In my opinion a good overview and example how this could work.

Room and Pillar approach for software testing

A journey underground
In the beginning of this month I went with a group of colleagues to "the caves" of Maastricht in The Netherlands. During this trip in the dungeons of the earth I learned that this was a result of mining and not a natural phenomenon. What I found impressive was the way they cut the stones out of the earth. To me it looks a bit like "negative building", you built something by taking away sources.

They start to work on the top and dig their way down below. Only it was not just start digging, they first search for the flint. If they found it they knew what level the ceiling would be. This was an approach they used for centuries. In that time you were owner of the ground below your property. This means that boundaries were set based on the size of property above ground.

As there were more people who were mining in that area, the chance to meet your neighbor underground was accepted. To prevent that others are digging in your area marks were set on the walls. As the mine extend the risk for collapsing increased. To avoid this they came up with a method to dig enough stone away and still maintain the supporting structure. This was done by leaving pieces of stone to act as pillars. The result was that you have now a area with pillars and rooms you can safely walk through from one side to another side.

What is Room and Pillar mining?
According to Wikipedia: Room and pillar: "Room and pillar is a mining system in which the mined material is extracted across a horizontal plane while leaving "pillars" of untouched material to support the overburden leaving open areas or "rooms" underground. It is usually used for relatively flat-lying deposits, such as those that follow a particular stratum"



William A Hustrulid and Richard L. Bullock explained the method as: "If one pillar fails and surrounding pillars are unable to support the area previously supported by the failed pillar they may in turn fail. This could lead to the collapse of the whole mine. To prevent this the mine is divided up into areas or panels" Note:*1

Mining and software testing
Somehow I see some similarity to this kind of mining and software testing. We also are trying to dig our way through the system. In my opinion we are also performing some negative building. Although we are not building the software ourselves, we try to find errors which will lead to changes in building. Also we are starting from the surface and going downwards in the system until we are called to stop. If we look at the explanation of Hustrulid and Bullock we also might phase a collapse of the system if we tested to deep or wrong. For example we perform wrong actions which leads to stopping is for further testing of that test case or data might get corrupted.

Pillar and Rooms in Organizations
Perhaps this is a bit far-fetched; processes in organizations can also be seen as pillars and rooms. If you take for granted that organizations are performing activities and those activities are assigned to processes you can create a similar map like shown in the figure below.



In level 1 you can define the departments. In level 2 the main processes and in Level 3 their sub-processes. Each block in a level is a room where activities are performed and the information is handed over to another room. To transfer these information agreements on communication and format must be defined. These can be seen as the pillars for the organization.

Mapping your test cases
To find your way through the mines you also need a map and a guide. To find you way to the system and check if the system supporting the business processes you can use a similar map of processes.

This map can be used to make the path of test scenarios visible. Those paths can be used to present certain coverage of the system towards the business processes. Initially you can show that based on the selected test cases you cover every business process at least once. Another benefit is the creation of transparency that the system is able to process data to support the processes.
Secondary use of this map is to can be as a storyboard. Let the business explain what they are doing in their processes, what their understanding is about the impact of their changes in data for next processes and also prioritize their activities. These priorities can be used to add depth in the test cases.



Test coordinator as guide
Even if you have a map of the mine, it will always be hard to find your way through the system. To speed up you journey through those mines to make sure you see the important things and find your way back to the exit it is recommended to get a guide who leads you. To lead you through the system you can assign this task to the test coordinator. He uses the map to lead you through the system and based on this map he explains in terms (processes and activities) you know what it mentionable to pay attention to. As for a guide in the mines the gas lamp, an additional flash light and his experience of the environment helps to make the journey pleasurable. The test coordinator should bring in his experience of the system, the test process and other additional tools to make it worthwhile.

Creating test depth in the room
I already mentioned to define test cases based on using every process at least once. This can be done using test cases for proving the system is supporting level 2 processes. I suggest going at least through every Level 3 processes.

As one of the main borders are activities and every process can consist out of more then one activity you can use the prioritization to decide if a certain path should be performed multiple times. Or combine activities as you are staying in a room for a longer period. As example a process can exist on: creating, maintaining and deleting data.

You could derive here three similar paths for it. You also can use it in one path. If you are using it in a same flow you have to be sure that the next process is not of the same priority and also the mutation of data has a minor effect on that process.

Mining the Business Process Chain Testing
In my opinion the approach of pillars and rooms is very useful to define a set of test cases to support Business Process Chain Testing (BPCT). You approach the system from a business point of view and measure if the system is supporting the organizational processes. Based on importance and risks you can define the depth of test cases. Also using a structure as pillars and rooms you are able to tell the business what kind of test approach you have and how you make sure that important things are done. You also are able based on the activities which have to be performed which resource you need from the business to perform the test cases.

Note: *1 William A Hustrulid and Richard L. Bullock (2001). Underground Mining Methods: Engineering Fundamentals and International Case Studies. Society of Mining Engineers, p 493-494. ISBN 0873351932.

Life Cycle of a Tester

Somehow we seem to think that for every project a tester should be available. Somehow we expect that that tester has all the experience and skills. Somehow we expect that tester to be the sheep with the five legs (I'm wondering if this is also an English expression)

I think you need to select the tester you needed for that particular moment. With this statement I assume that there is next to a product life cycle, a development life cycle also a tester's life cycle.

The basic idea for this thought is:
There are not always those testers available who can help you for a whole project. You have to make concessions about the skills of hired testers. This involves a certain risk which can be avoided by education or hiring more testers.

You can also accept the life cycle of a tester and admit that a tester should be available just for a period of time. Some simple steps could be:
1. Divide your test process in several phases;
2. Define for every phase the skills and expertise you need;
3. Prepare a time frame when you need those skills;
4. Start searching for those skills;
5. Find the testers who are able and willing to help you;
6. Adapt a process of monitoring the needed skills and needed testers.
7. Prepare some overlap and knowledge sharing

Benefits:
As you are able to find those different testers you are able to ask/ pay different tariffs. Another benefit is that you have the tester you need for that certain moment. You save time in education and you don't make concessions in quality.


I can imagine that you have several processes like:
1. Test management
2. Test preparation
3. Test specification
4. Test Execution

For these processes you might have different persons with different skills.

Of course there are other "somehow’s". There are other processes and there are some times not the testers available. It is just a thought. Feel free to leave a comment, perhaps we can share thoughts.

Link: Testing challenges by Matt Heusser

On a previous post I already mentioned about some good questions asked by Michael Bolton on Matt Heussers weblog: Creative Chaos.

Matthew create a testing challenge which is worth to pay attention to.
For more info see:
A new testing challenge
New testing challenge - II
New testing challenge - III
New testing challenge - IV
New testing challenge - V
Perhaps you can put some though in the basket?