When thinking about software testing most of the people think about executing tests. In general they are correct. To be able to start executing the desired test cases there are processes defined and in some cases no processes defined.
Only testing is more then just executing test cases. It more likely a playground we are presenting, using and offering our skills and knowledge. Doing this we find our stakeholders or they are seeking us. So besides of knowledge of the system or testing we also need to able to use soft skills.
It would be easy to visualize who is asking for our expertise. You might just draw a picture about the possible stakeholders.
It would be more challenging to visualize what request for information you might get. This is what makes testing a hard profession. You have to deal with:
- Organizational processes and procedures;
- Testing processes and procedure;
- Differences in skill levels from testers, developers, architect, managers, users and so on;
- The dynamic environment were it all happens;
- Differences in management styles;
- Differences in systems, technologies;
- Differences in cultures;
- Differences in history.
To act on a playground were this all happens just executing test cases is a wrong perception to explain what testing is about. It needs more than that. It needs skills and the ability to use those skills by the ability of asking the correct questions to collect information about the system, people and organization. It needs a certain attitude to be able and allowed to gain, interpret and present information.
Wednesday, February 11, 2009
Why is software testing hard?
Posted by
Jeroen Rosink
at
7:46 AM
0
comments
Labels: Testing in General
Tuesday, February 10, 2009
The glass is half full or half empty
Is the glass half empty or half full? is a common expression, used rhetorically to indicate that a particular situation could be a cause for optimism (half full) or pessimism (half empty).
What I'm wondering is how this should be used in projects? Should we be optimistic or pessimistic? I would be too easy to say that we should be neither of both as we are there to measure the "truth about quality". The thing here is that we are also working with people. And people have to be convinced, when you are too optimistic or too pessimistic they won't believe you. If you are neither of both, they probably won’t trust you.
I suggest being honest. Be pessimistic about the things which are going well beyond expectations and be optimistic about parts which unexpectedly are going different.
Don't use the half-filled glass metaphor, be honest and create room for the natural optimistic and pessimistic people.
Posted by
Jeroen Rosink
at
5:13 AM
1 comments
Labels: Metaphor
Monday, February 9, 2009
Risks of "definition Done"
Somehow I see some risk in agile projects working with definitions of Done. Here some risks which slips my mind.
Risks:
- Usually only the functionality which meets the "definition Done" will be shown in the demo to the customer. Undone functionality is in the code available although it is not presented in the demo. This might lead to failures in production because based on acceptance of items Done a release might be shipped in production;
- If there are minimal skills in the teams and organization the "definition Done" might also be too small. This might lead that although the software meets the definition huge errors related to fit to process and usability will occur.
- Business has too less knowledge to define a "definition Done" and the team supports the definition phase based on their current knowledge. Definition is written mainly from technical perspective due to expert knowledge of the team of coding and too less knowledge of the business processes;
- "Definition Done" is used by project management to meet their project goals and deliberately narrowing the scope and content of the definition during the project. Instead of proofing that the system fits the business it is turned into proofing that the system fits the project.
I know that there will be more risks to mention, these were just some few which slipped my mind. I certainly not am saying that Agile is not working. I certainly not am claiming that "Definition Done" is creating false trust. I just wanted to show that we have to precise on defining Done instead of stepping too easily over this important tool.
Posted by
Jeroen Rosink
at
5:33 AM
0
comments
Sunday, February 8, 2009
To do or not to do that is to measure
Somewhere there is a thin border between the type of metrics needed and metrics offered. What I saw over the years is that projects generate a huge number of data. Based on experience I noticed that most of the data was useful to provide information in a project and not desired in the other.
Translating data into information is usually time consuming. This raises the question to me: should I spend time to provide all information I'm able to deliver? I would answer with a "NO".
Should I stop collecting data? Again I say "NO".
There are several tools available to provide information about the test process. Usually I end up with creating some kind of cockpit in MS Excel. The benefit of this is the ability to combine data from several sources and turn them into information.
To get the data from other tools in MS Excel I export the data from other tools into a readable format for this spreadsheet tool. The advantage here is that I am able to conserve historical data.
Instead of turning all data into information I start with some obvious ones. Those management need to get information about the process and those I need to steer processes. I learned that after starting presenting nice columns of data and charts management is asking for more different information or the already presented information in more detail or less detail.
Based on conserving historical data I’m able to use those also instead of staring measuring from scratch.
The benefits of not presenting all information you are able to give will be immediately saving time as you don’t offer all information. Being adaptive for new information requests will give you the benefit that things are changing because of presented information.
I believe defining and creating metrics should lead to change in behavior, attitude or trust. And we should be aware of creating false trust. Keep asking what might change if metrics are collected and presented. If nothing change, the do not create metrics. Certainly avoid the attitude: I do this because it is written or I do this always.
Posted by
Jeroen Rosink
at
10:57 AM
0
comments
Labels: Metrics, Testing in General
If software testing is so simple
On his weblog Michael Bolton posted a comment in his article Getting Them To Do The Work "If we're going to have business people write the tests, why not have them write the code for the product too? Why not have the programmers make the marketing decisions?"
I think this supports my idea that offering the business templates to fill in test cases is not the way to get the business involved in the development project and the testing activity. Certainly it will not help measure the quality or asking the proper questions to the system. It will basically only test how much knowledge the user has about the system he is already using or will be using in the nearby future.
To me software testing is a skill and needs a certain attitude. The templates we are using are mainly used to structure our process and skills and explaining the business what we are doing to help them.
If you decide to give the business users the template; then also support them with sharing knowledge how to use them.
Posted by
Jeroen Rosink
at
9:11 AM
1 comments
Labels: Testing in General
Tuesday, February 3, 2009
Classification of the "Unknown Unknowns"
In my previous post Schedule the Unknown Unknowns I talked about the good post and classification by Catherine Powell about possible problems from systems in the test schedule: Problem Types. She wrote a follow-up on this: Uncertain of Your Uncertainty. She asked for ideas on this topic. This triggered me to dive in the subject of "Unknown Unkowns".
Although there are a lot of articles which can be found on the internet the one from Roger Nunn: The "Unknown Unknowns" of Plain English helped me to get a bit of light in the darkness. He posted there the "Rumsfeld's categories: the states of knowledge during communication"
1. Known knowns:
* Things I assume you know - so I do not need to mention them.
2. Known unknowns:
* Things I know you do not know - so perhaps I will tell you, if I want you to know.
* Things you know I do not know - but do you want me to know?
3. Unknown unknowns:
* Things I do not know that you do not know or that you need to know - so I do not think of telling you.
* Things I am not aware of myself - although it might be useful for you to know.
* Things you do not tell me, because you are not aware of them yourself - although I might need to know.
I think the main topic in his article is "Communication". He refers to Rumpsfeld: "Rumsfeld's winning enigma uses combinations of only two very common words that would fit into anyone's list of plain English lexis."
An article from C. David Brown, Ph.D. , PE: Testing Unmanned Systems triggered me to think in 2 types of systems: manned systems and unmanned systems. Why is this important? Acknowledging the existence of unmanned systems can help you not to forget also to those systems questions. Although I never worked with unmanned systems; I worked in projects with modules which were autonomously working parts in the whole systems. So there is a huge chance that it contains my "unknown unknowns".
C. David Brown classified Unmanned systems can generally be classified into three categories: - remotely operated;
- fully autonomous;
- leader-follower.
Combining these both stories you have to deal with human communication and system communication to find out about the "Unknown Unknowns". I think it should be possible to capture those uncertainties in a time schedule by identifying which questions on what moment to whom/what to ask for which purpose. Of course this is so obvious. Still it is hard to tell how to fill in those gaps.
What often I read on James Bach's and Michael Bolton's weblogs is to question the system, to question the business. To test the tester instead of certify the tester. If I'm correct these questions are raised to obtain mutual understanding and exchanging knowledge. The pitfall here is that questions are not asked because of one of the above 3 reasons mentioned in the "Unknown Unknowns".
Perhaps the nature of the system can help us here and also complexity.
If a system/function is running fully autonomous the chance of asking the system is much harder, the chance of getting answers from the system is even harder. Human interaction has to take place which is introducing another variant of those 3 unknown unknowns.
Perhaps you can continue now in at least 3 directions:
1. Keep asking the system and business, continue learning from each other and start adapting your test process based on this uncertainty;
2. Define an approach were you are not measuring how many time is already used, start measuring how many time you need to get things Done (a SCRUM approach could help)
3. Another option would be reviewing. I think the reviewing can decrease the occurrence of level of unknown unknowns as the individual expert is no longer the only persons who ask questions, also others. With reviews you might build up a system of numbers which can help you to define on which areas you have lesser knowledge by the number of questions raised or major issues found.
In the first 2 options, I can imagine that it will be hard to take this into your time schedule. If you will use the last option you might take some numbers from the Boehm curve. and play with it From the book (Dutch) "Reviews in de praktijk by J.J. Cannegieter, E. van Veenendaal, E. Van de Vliet and M. van der Zwan" a table they defined a simplified model of the Boehm-law:
Phase Simplified Boehm-law
Requirements 1
Functional Specifications 2
Technical Specification 4
Realization 8
Unit/Module test 16
System/Acceptance test 32
Production 64
Now you can start playing with these figures, keep in mind that this will not lead to reliable figures. At least some figures can be useful instead of no figures.
First: flip them around
Phase Simplified Boehm-law
Requirements 64
Functional Specifications 32
Technical Specification 16
Realization 8
Unit/Module test 4
System/Acceptance test 2
Production 1
Second: if 1 = 1 hour then you might calculate 20% for available unknown unknowns. Here you will have in the requirements phase and addition to the time schedule of 64 * 20% = 12,8 hrs
I think you can use such kind of calculation as in the beginning of a project the chance for unknown unknowns is much higher then at the end.
In this situation you accept that every unknown unknown will have the same impact.
You can add a third step to it:
Third: Add classification based on MoSCoW principle
Must haves = 100%,
Should Haves = 80%
Could Haves = 40%
Would Haves: = 20%
When having 10 M, 4 S, 25 C and 25 W this will lead in requirements phase to:
10*(64*20%))*100%= 128 hrs*100% + 4*(64*20%))*80%= 40.96 hrs + 25*(64*20%)*40%= 128 hrs + 25*(64*20%)*20%= total: 360.96 hrs
Fourth: Here are still a lot of assumptions made. Based on History and complexity the percentages should be adapted. The initial 20% should be defined based on historical figures. The percentages used in the MoSCoW classification can be defined on technical complexity combined with available information. I assume if a lot of questions are raised the chance something is missing will be higher if reviews are executed in a proven state.
Fifth: The numbers used from the simplified Boehm-law can be adapted based on level of clear communication between people. Perhaps the level of interactions between people can be used to minimize the chance that information is not shared in an early stage.
Posted by
Jeroen Rosink
at
9:04 PM
0
comments
Sunday, February 1, 2009
Schedule the Unknown Unknowns
Catherine Powell posted a very good and clear article about dealing with possible problems from systems in the test schedule: Problem Types.
She classifies them into:
- Fully understood problems;
- Partially understood problems that can be worked in parallel;
- Partially understood problems that are linear.
What I missed here is considering the risk of problems which we are not yet aware of: the so called unknown unkowns.
Donald Rumsfeld said: "There are known knowns. There are things we know that we know. There are known unknowns. That is to say, there are things that we now know we don’t know. But there are also unknown unknowns. There are things we do not know we don’t know."
I think Catherine covered with her classification almost the whole saying of D. Rumpsfeld:
* Fully understood problems -> things we know that we know
* Partially understood problems -> the known unknowns and the now unknowns
As she mentioned that this can be used to refine the time schedule, the known knowns are already in place, and the fully and partially understood problems have to be considered to estimate some additional time for it.
The unknown unknowns I still miss in the classification. If I understood her correctly she gives the advice to keep investigating and then add them to one of the classifications.
I can imagine that in the time schedule you already reserve time for this. For time reservation you need some solid ground. As we are talking about the unknown unknowns this is certainly missing. Perhaps you can justify that solid ground by investigating how complex the system is and how many questions you still might ask to the system. Based on this you can visualize the field of uncertainty.
Explicitly reserving time for this can help project management to identify the risk that the schedule will exceed. The big challenge is not only to move the unknown unkowns towards the unknown knows section; it will also be a challenge to make this activity accepted by project management.
Posted by
Jeroen Rosink
at
8:35 AM
0
comments
Saturday, January 31, 2009
Shooting with hail or just one test case?
Almost every one wants to setup a controlled test process.
It is if we are playing the song "Proud Mary" by Ike & Tina Turner: "Take the beginning of this song and do it easy. Then we're gonna do the finish rough."
We always start simple with a controlled set of test cases. Often these test cases are based on the outcome of using test techniques. So far so good.
Still we might become lazy over time. If errors are found in production, we re-test the solutions and add the test case to the test set. There is less time to question the system and our test process what went wrong and what can go better.
In manual testing after some short time you will see that keep adding those test cases to the test set will make our process "rough" as we don't have time to execute all test cases or select a common sense set of cases. Business is already relying on us and demanding that all is done.
In automated testing this time pressure is almost minimized. We can keep adding new cases, let the number exceed over the thousands and still continue and deliver.
This reminds me about the phrase: "A bird in the hand is worth two in the bush"
If we accept the meaning of this phrase we are we not able to continue controlling that bird in our hand and instead sewing our own bush?
It is easier to take care of a bird we know then nurse the birds in the bush. If we feel we need another bird, just let the bird free and pick another out of the bush. Not just the first one which fell in your arms, go into the bush and see which sounds the best to your needs.
Keep it simple and easy, enjoy the roughness and prevent to get stucked on numbers. Don't create a process where you are shooting with hail on the system.
Posted by
Jeroen Rosink
at
6:48 AM
0
comments
Labels: Metaphor, Testing in General
Friday, January 30, 2009
Just one development method
Over the years I learned that in most companies a development method is chosen. In the Netherlands it depends in which area you work. Most of the organizations I saw or heard about classified RUP as development method. In SAP related projects they claimed they were working according ASAP.
How often is not asked to the tester if he/she has experience with that certain development method? At least it is a pre when you have some experience. The conclusion is often made that if a person has experience working with a certain development method he/she is able to fit that experience towards testing in those projects.
Besides that experience of development method also experience is asked about testing methods and the ability to adapt that method to those development methods. In plain theory it is all possible and is also written in several books.
Most of the time those organizations are not able to use the development method as it was intended. And they also did not write down or monitor which adaptations they have made. The risk here is that it will be even harder to make the test method fit to the process. I think we should never forget that testing is just a part of the development process and if that process is not clear then the chance that the test process is optimized is very small and improvements can only be made towards the testing process. A strong link in a weak chain is created.
I wonder why organizations keep sticking to one development method although they already know it doesn't fit for all projects and still demanding to work accordingly as it is default standard.
What I suggest to evaluate in your organization what need you have and how that need can be turned into value. Sometimes you have large projects were RUP can work very well. Sometimes you have smaller projects were an Agile approach fits better. The main thing is to define multiple approaches and select them based on the benefits and check if the disadvantages are acceptable. Also keep in mind if the chosen approach fits in the business culture and supports the organizations vision, strategy and goals.
If the development method is clear a test method can be fit much better and the benefits of both approaches can be reached and the disadvantages controlled better.
Posted by
Jeroen Rosink
at
7:32 PM
0
comments
Labels: development, Testing in General
Some bugs missed
In testing we are trying to avoid bugs to be shipped to production environment. As tool we use test cases. The good part of using test cases you can control the depth of testing. If a tester is less experience you can write detailed test scripts. If the tester is very experienced and skilled you can do without some details. Using these test scripts some nice metrics can be created like: how many of what in which time period and is it according plan, schedule and expectations.
A bad thing is that we often instruct the lesser experienced testers to test only those things which are written down. And we give the same instruction also to the experienced testers as the time schedule must not slip.
What I saw is that for small parts this is not yet an issue as test cases are defined based on using test techniques and this gives us the ability of measuring some coverage. In large tests like chain tests a risk occurs that defects are missed as the focus is defined on high level to perform certain actions. Obviously those defects don’t hurt the system and process immediately and therefore accepted as intended functionality.
If the testers are also the key-users there is a chance that this "wrong" behavior is explained to other users and also documented in work instructions and manuals. People will continue working with the process on production ignoring these errors. Sometimes it you will see that unintentional these actions will become part of a workaround without knowing it is a workaround. It can also happen that those errors keep unnoticed.
Since testers try to re-use testware the next time testers will perform the same actions. During their usage on production their knowledge of the system is increased.
In the past I saw situations that this gained knowledge lead to finding the errors during testing which were "previously" not there.
This will lead to a discussion if the functionality is as expected as in the past it worked that way, so why should it be now an error? Should it be registered as an error in the "new" functionality or should it be logged as a production error?
Logging it as an error in the test environment it will have impact on the advice for release. If it is a production error it should be decided when to solve it. A situation is created if it is allowed to disturb the project or keep disturbing the production.
What will you decide and for what reason?
Posted by
Jeroen Rosink
at
7:17 AM
0
comments
Labels: Testing in General
