Do you change your testing strategies?

Just out of curiosity I'm wondering if you change your testing strategy. Is there room for change? Are you willing to change? Are your brave enough to change?

Deliberately I'm not asking for the need for change. Because of the current recession I think there is an urgent need to change. The question is what should be changed?

In many projects, people are working with proven technologies and methods. Some of the methods are very useful to cover all risks and measure the quality proved by those methods. Some methods are useful as they rely on best practices. Some methods are necessary as organizations are forcing to use those kinds of project management methods.

In all cases those methods are used to solve problems for organizations. I'm just wondering if there is time enough to keep using those solid methods. Is there budget enough to measure all quality? Are resources and skills available to make it a success using those methods?

Perhaps it is time to change. With change I'm more thinking of a context driven test approach as James Bach, Cem Kaner, Michael Bolton and others are suggesting. If I understood that approach good enough the basis of that approach is defining the context first instead of defining the framework for the project first and the "proven" test method.

When thinking about the context and considering the time we live in now we might understand that resources are not available to make it a success. Because of a clear context decisions can be made to accept risks.

Under current conditions risks have to be taken. Accepting risks might lead to new opportunities since the focus is just there were the organization sees it challenges, opportunities and problems which have to be solved.

I'm just curious:
- Do you dare?
- Do you care?
- Do you have enough resources, budget and skills to continue as you did in the past?
- Or do you change?

How to question errors?

Recently I was in a discussion with a tester/user who wanted more information about the cause of an error and the provided solution. It took quite some time to understand each other. I have to say that I made a huge mistake here. I listen to the tester why he needed more detailed information only draw the wrong conclusion.

The conclusion was merely: Why do you need all that information if you can check if the symptom is solved, it is a minor change and has minimal impact on your process. Demanding more information although no agreements with development are made to provide this will extend the cost of the solution.

Basically the problem was not the level of detail of the documentation and the provided solution. The tester just wanted to know if the developer understood his problem enough and wanted to check until which level the analysis was done.

It took more time to understand each other: the tester wanted more information so he could decide the impact and define the test depth. I tried to convince him that if there is less chance of damage, it might not me wortth to go into this detail.

This made me think how we could talk the same language. for this I created a picture, perhaps this will help in the nearby future to discuss.

The first picture is to identify the type of error and how it is solved. The columns at the end can be used together with the developer to get a mutual understanding about the error, cause and the solution.

The second picture can be used to visualize the questions we might ask and to monitor if the effort for gaining information is worth in relation with the solution and impact. I also left here the possible questions as they have to be defined in their context.

Although I know it is not covering everything, I think it still can be used to visualize our thoughts.

Understanding the problem: the skilled helper

How often are we offering solutions to help people based on our experience? Somehow we are trained to identify possible problems and judging them against our best-practices. Because we are always in some kind of time pressure we tending to help claiming our approach is proven and has some successes.

For example: we are asked to structure the test process and our best practice is the TMap approach. Instead of identifying the actual problems we are defining the test process based on their phases and to see what context we have to work in we perform a so called Product Risk Analysis (PRA).

For some time I'm trying to understand the Context-Driven-Testing (CDT) approach. I think the main idea here is to understand the context first before you come with solutions.

To understand the context you first have to listen. This reminded me about a book I read some years ago: The Skilled Helper: A Systematic Approach to Effective Helping by G. Egan.

I think this approach fits well in our profession also. Aren't we also "helpers"?

Egan's approach consists of three stages which might prevent us jumping into conclusions/solutions. Every stage consists of three steps with continuous evaluation (E).

Stage 1 - The present scenario: Help clients identify, explore, and clarify their problem situations and unused opportunities.




Stage 2 - The Preferred Scenario: Help clients develop goals, objectives or agendas based on an action oriented understanding of the problem situation



Stage 3 - Formulating Strategies and Plans: Help clients develop action strategies for accomplishing goals, that is, for getting from the current to the preferred scenario


I think these stages and steps can be combined with Heuristic Test Strategy Model can help define the correct context and choosing the best strategies and plans for that certain context.

For more information about Context Driven Testing see:
The Seven Basic Principles of the Context-Driven School
Weblogs:
James Bach
Michael Bolton
Cem Kaner

For more information about Egan's approach you also might check: Introduction to ‘The Skilled Helper’ A Systematic Approach to Helping

Problems and responsibilities

Headache
During a master class SCRUM Jens Østergaard and Bas Vodde gave us the phrase "If a problem gives you a headache then probable it is not your problem."

The idea behind this was that the headache is initiated because you are not able to solve it immediately which made you think without reaching the solution.

Getting headaches is sometimes inevitable in software testing. It is easy to handover the problem to some one else. Will this be the correct way? I think you have to check if it lies in your field of responsibilities and your field of influence.

Responsibility
I always try to think in the number three-approach. Based on this I came to the following classification of problems and responsibilities:
- problems in your area of responsibility, only you don't know how to deal with it
- problems just beyond your area of responsibilities, only it has to do with your field of profession
- problems beyond you area of responsibilities, only you were eager to notice them

In your area
If problems can be assigned directly to activities you are responsible for, then you should not hand over the problem. As you got already the headache, try to find a person who is able to help you. This help can be given in support or advice.

Just beyond your area
If the problem is not related directly to your activities you should check if it is related to your profession. It would help the person you handed the problem needs your support, your expertise and your commitment to help him.

Outside your area
This is an area you have to be careful because you noticed things which should not be your case. Here you have to decide if you are approaching it in a formal way or informal way. Formally could be by escalation or documentation (e.a. e-mail). Informal would be by identifying the person in your field of influence that might be able to guide the information.

Problem solving the headache
To deal with your headache, you might:
- Ignore it, and hope it will go away over a while;
- Take an aspirin;
- Identify your responsibility and take that.

How many bugs do you need to make a metric?

Last year I attended a presentation by Markus Schumacher about metrics. His presentation can be summarized by his saying: "Metrics should change behavior".

I cannot forget about this saying while I'm busy collecting and presenting data. Often I see presentations of figures how many bugs we already have found during testing. Discussions are held how many bugs we didn't find. A lot of time is spent to make nice charts from this data.

Michael Bolton posted a very good blog posting about metrics: Meaningful Metrics

In his blog he also suggests to create metrics which rising questions instead of creating metrics to be used as a control mechanism.

I don't think you can expect from management that you skip presenting control-metrics from the start. It is also not appreciated you don't present any metrics. What I often do is start simple by presenting the number of issues found in a certain period and what their severity is. Although this won't change behavior or raising questions about quality as it should do, it creates the room to discuss what should be measured. Collecting data and creating metrics is also a dynamic process.

I would summarize this process in the following steps:
1. Ask what metrics are needed immediately, let management prioritize because it cost time and effort;
2. Ask what their goals are for those metrics;
3. Define the goal of the selected metrics together with the customer and define criteria how to use them and to monitor the benefits of those;
4. Add some 1 or 2 other metrics you think which are needed to meet their goal;
5. If they got used to those metrics you might even visualize it in charts;
6. After an iteration or a weekly meeting check if it met their goals;
7. Adapt your metrics based on requests, this also can mean you add or remove metrics;
8. Sometimes you have to dive into your collected to see if questions can be defined based on historical data;
9. If people got punished because of the metrics, remove that information as interpreting metrics is a profession on its own and often the metrics we are dealing with can barely be used as indications;
10. Do this every time again, don't rely on the believe if things were good enough the last tie then it should be good enough for now;

Coming back to the title: "How many bugs do you need to create a metric?" I would say: None. You will need at least one question where you have to find an answer to. This answer should lead to change in behavior, make people move or raise more new questions.

Numbers of bugs can be used to create fancy charts. As long as those charts doesn't move people I would say that charts presenting >1000 bugs, 10 categories over a period of 10 weeks are very useful if you want to exercise with colors and chart types.

Why is software testing hard?

When thinking about software testing most of the people think about executing tests. In general they are correct. To be able to start executing the desired test cases there are processes defined and in some cases no processes defined.

Only testing is more then just executing test cases. It more likely a playground we are presenting, using and offering our skills and knowledge. Doing this we find our stakeholders or they are seeking us. So besides of knowledge of the system or testing we also need to able to use soft skills.
It would be easy to visualize who is asking for our expertise. You might just draw a picture about the possible stakeholders.

It would be more challenging to visualize what request for information you might get. This is what makes testing a hard profession. You have to deal with:
- Organizational processes and procedures;
- Testing processes and procedure;
- Differences in skill levels from testers, developers, architect, managers, users and so on;
- The dynamic environment were it all happens;
- Differences in management styles;
- Differences in systems, technologies;
- Differences in cultures;
- Differences in history.

To act on a playground were this all happens just executing test cases is a wrong perception to explain what testing is about. It needs more than that. It needs skills and the ability to use those skills by the ability of asking the correct questions to collect information about the system, people and organization. It needs a certain attitude to be able and allowed to gain, interpret and present information.

The glass is half full or half empty

Is the glass half empty or half full? is a common expression, used rhetorically to indicate that a particular situation could be a cause for optimism (half full) or pessimism (half empty).



What I'm wondering is how this should be used in projects? Should we be optimistic or pessimistic? I would be too easy to say that we should be neither of both as we are there to measure the "truth about quality". The thing here is that we are also working with people. And people have to be convinced, when you are too optimistic or too pessimistic they won't believe you. If you are neither of both, they probably won’t trust you.

I suggest being honest. Be pessimistic about the things which are going well beyond expectations and be optimistic about parts which unexpectedly are going different.

Don't use the half-filled glass metaphor, be honest and create room for the natural optimistic and pessimistic people.

Risks of "definition Done"

Somehow I see some risk in agile projects working with definitions of Done. Here some risks which slips my mind.


Risks:

- Usually only the functionality which meets the "definition Done" will be shown in the demo to the customer. Undone functionality is in the code available although it is not presented in the demo. This might lead to failures in production because based on acceptance of items Done a release might be shipped in production;
- If there are minimal skills in the teams and organization the "definition Done" might also be too small. This might lead that although the software meets the definition huge errors related to fit to process and usability will occur.
- Business has too less knowledge to define a "definition Done" and the team supports the definition phase based on their current knowledge. Definition is written mainly from technical perspective due to expert knowledge of the team of coding and too less knowledge of the business processes;
- "Definition Done" is used by project management to meet their project goals and deliberately narrowing the scope and content of the definition during the project. Instead of proofing that the system fits the business it is turned into proofing that the system fits the project.


I know that there will be more risks to mention, these were just some few which slipped my mind. I certainly not am saying that Agile is not working. I certainly not am claiming that "Definition Done" is creating false trust. I just wanted to show that we have to precise on defining Done instead of stepping too easily over this important tool.

To do or not to do that is to measure

Somewhere there is a thin border between the type of metrics needed and metrics offered. What I saw over the years is that projects generate a huge number of data. Based on experience I noticed that most of the data was useful to provide information in a project and not desired in the other.

Translating data into information is usually time consuming. This raises the question to me: should I spend time to provide all information I'm able to deliver? I would answer with a "NO".
Should I stop collecting data? Again I say "NO".

There are several tools available to provide information about the test process. Usually I end up with creating some kind of cockpit in MS Excel. The benefit of this is the ability to combine data from several sources and turn them into information.

To get the data from other tools in MS Excel I export the data from other tools into a readable format for this spreadsheet tool. The advantage here is that I am able to conserve historical data.

Instead of turning all data into information I start with some obvious ones. Those management need to get information about the process and those I need to steer processes. I learned that after starting presenting nice columns of data and charts management is asking for more different information or the already presented information in more detail or less detail.

Based on conserving historical data I’m able to use those also instead of staring measuring from scratch.

The benefits of not presenting all information you are able to give will be immediately saving time as you don’t offer all information. Being adaptive for new information requests will give you the benefit that things are changing because of presented information.

I believe defining and creating metrics should lead to change in behavior, attitude or trust. And we should be aware of creating false trust. Keep asking what might change if metrics are collected and presented. If nothing change, the do not create metrics. Certainly avoid the attitude: I do this because it is written or I do this always.

If software testing is so simple

On his weblog Michael Bolton posted a comment in his article Getting Them To Do The Work "If we're going to have business people write the tests, why not have them write the code for the product too? Why not have the programmers make the marketing decisions?"


I think this supports my idea that offering the business templates to fill in test cases is not the way to get the business involved in the development project and the testing activity. Certainly it will not help measure the quality or asking the proper questions to the system. It will basically only test how much knowledge the user has about the system he is already using or will be using in the nearby future.

To me software testing is a skill and needs a certain attitude. The templates we are using are mainly used to structure our process and skills and explaining the business what we are doing to help them.

If you decide to give the business users the template; then also support them with sharing knowledge how to use them.

Classification of the "Unknown Unknowns"

In my previous post Schedule the Unknown Unknowns I talked about the good post and classification by Catherine Powell about possible problems from systems in the test schedule: Problem Types. She wrote a follow-up on this: Uncertain of Your Uncertainty. She asked for ideas on this topic. This triggered me to dive in the subject of "Unknown Unkowns".

Although there are a lot of articles which can be found on the internet the one from Roger Nunn: The "Unknown Unknowns" of Plain English helped me to get a bit of light in the darkness. He posted there the "Rumsfeld's categories: the states of knowledge during communication"

1. Known knowns:
* Things I assume you know - so I do not need to mention them.
2. Known unknowns:
* Things I know you do not know - so perhaps I will tell you, if I want you to know.
* Things you know I do not know - but do you want me to know?
3. Unknown unknowns:
* Things I do not know that you do not know or that you need to know - so I do not think of telling you.
* Things I am not aware of myself - although it might be useful for you to know.
* Things you do not tell me, because you are not aware of them yourself - although I might need to know.

I think the main topic in his article is "Communication". He refers to Rumpsfeld: "Rumsfeld's winning enigma uses combinations of only two very common words that would fit into anyone's list of plain English lexis."

An article from C. David Brown, Ph.D. , PE: Testing Unmanned Systems triggered me to think in 2 types of systems: manned systems and unmanned systems. Why is this important? Acknowledging the existence of unmanned systems can help you not to forget also to those systems questions. Although I never worked with unmanned systems; I worked in projects with modules which were autonomously working parts in the whole systems. So there is a huge chance that it contains my "unknown unknowns".

C. David Brown classified Unmanned systems can generally be classified into three categories: - remotely operated;
- fully autonomous;
- leader-follower.

Combining these both stories you have to deal with human communication and system communication to find out about the "Unknown Unknowns". I think it should be possible to capture those uncertainties in a time schedule by identifying which questions on what moment to whom/what to ask for which purpose. Of course this is so obvious. Still it is hard to tell how to fill in those gaps.

What often I read on James Bach's and Michael Bolton's weblogs is to question the system, to question the business. To test the tester instead of certify the tester. If I'm correct these questions are raised to obtain mutual understanding and exchanging knowledge. The pitfall here is that questions are not asked because of one of the above 3 reasons mentioned in the "Unknown Unknowns".

Perhaps the nature of the system can help us here and also complexity.
If a system/function is running fully autonomous the chance of asking the system is much harder, the chance of getting answers from the system is even harder. Human interaction has to take place which is introducing another variant of those 3 unknown unknowns.

Perhaps you can continue now in at least 3 directions:
1. Keep asking the system and business, continue learning from each other and start adapting your test process based on this uncertainty;
2. Define an approach were you are not measuring how many time is already used, start measuring how many time you need to get things Done (a SCRUM approach could help)
3. Another option would be reviewing. I think the reviewing can decrease the occurrence of level of unknown unknowns as the individual expert is no longer the only persons who ask questions, also others. With reviews you might build up a system of numbers which can help you to define on which areas you have lesser knowledge by the number of questions raised or major issues found.

In the first 2 options, I can imagine that it will be hard to take this into your time schedule. If you will use the last option you might take some numbers from the Boehm curve. and play with it From the book (Dutch) "Reviews in de praktijk by J.J. Cannegieter, E. van Veenendaal, E. Van de Vliet and M. van der Zwan" a table they defined a simplified model of the Boehm-law:

Phase Simplified Boehm-law
Requirements 1
Functional Specifications 2
Technical Specification 4
Realization 8
Unit/Module test 16
System/Acceptance test 32
Production 64

Now you can start playing with these figures, keep in mind that this will not lead to reliable figures. At least some figures can be useful instead of no figures.

First: flip them around
Phase Simplified Boehm-law
Requirements 64
Functional Specifications 32
Technical Specification 16
Realization 8
Unit/Module test 4
System/Acceptance test 2
Production 1

Second: if 1 = 1 hour then you might calculate 20% for available unknown unknowns. Here you will have in the requirements phase and addition to the time schedule of 64 * 20% = 12,8 hrs

I think you can use such kind of calculation as in the beginning of a project the chance for unknown unknowns is much higher then at the end.

In this situation you accept that every unknown unknown will have the same impact.
You can add a third step to it:
Third: Add classification based on MoSCoW principle
Must haves = 100%,
Should Haves = 80%
Could Haves = 40%
Would Haves: = 20%

When having 10 M, 4 S, 25 C and 25 W this will lead in requirements phase to:
10*(64*20%))*100%= 128 hrs*100% + 4*(64*20%))*80%= 40.96 hrs + 25*(64*20%)*40%= 128 hrs + 25*(64*20%)*20%= total: 360.96 hrs

Fourth: Here are still a lot of assumptions made. Based on History and complexity the percentages should be adapted. The initial 20% should be defined based on historical figures. The percentages used in the MoSCoW classification can be defined on technical complexity combined with available information. I assume if a lot of questions are raised the chance something is missing will be higher if reviews are executed in a proven state.

Fifth: The numbers used from the simplified Boehm-law can be adapted based on level of clear communication between people. Perhaps the level of interactions between people can be used to minimize the chance that information is not shared in an early stage.

Schedule the Unknown Unknowns

Catherine Powell posted a very good and clear article about dealing with possible problems from systems in the test schedule: Problem Types.

She classifies them into:
- Fully understood problems;
- Partially understood problems that can be worked in parallel;
- Partially understood problems that are linear.

What I missed here is considering the risk of problems which we are not yet aware of: the so called unknown unkowns.

Donald Rumsfeld said: "There are known knowns. There are things we know that we know. There are known unknowns. That is to say, there are things that we now know we don’t know. But there are also unknown unknowns. There are things we do not know we don’t know."

I think Catherine covered with her classification almost the whole saying of D. Rumpsfeld:
* Fully understood problems -> things we know that we know
* Partially understood problems -> the known unknowns and the now unknowns

As she mentioned that this can be used to refine the time schedule, the known knowns are already in place, and the fully and partially understood problems have to be considered to estimate some additional time for it.

The unknown unknowns I still miss in the classification. If I understood her correctly she gives the advice to keep investigating and then add them to one of the classifications.

I can imagine that in the time schedule you already reserve time for this. For time reservation you need some solid ground. As we are talking about the unknown unknowns this is certainly missing. Perhaps you can justify that solid ground by investigating how complex the system is and how many questions you still might ask to the system. Based on this you can visualize the field of uncertainty.

Explicitly reserving time for this can help project management to identify the risk that the schedule will exceed. The big challenge is not only to move the unknown unkowns towards the unknown knows section; it will also be a challenge to make this activity accepted by project management.