Wednesday, April 1, 2009

Magazine: Quality Matters, now with contribution of myself

Recently I send in an article with a though I had in my mind for some while about improving test processes. Sometimes it is better to neglect improvements how obvious they are.

In the 2nd edition of the magazine: Quality Matters my story is presented under the title: "Be innovative! Stop improving your test process."

Sunday, March 29, 2009

The Art of War in Software Testing

Introduction
One of the things I like to do is checking for similarities between history or other methods and software testing. Recently I was heading towards my assignment and got stuck in a traffic jam. You have some of those days. A radio station brought again some information about some war somewhere. Like software testing, no war is the same. No test process is the same. I think there should be some similarity between war and test processes due to the fact they are build upon a mission, vision, strategy, objectives, leadership and a context.
Types of war

On WikiPedia I searched if there was some information about different types of War . On this site I found a table containing types of war explained. (see below)

Environments of warfare

In warfare the environment has a huge impact on the type of battle which take place. You have conventional and unconventional warfare. In conventional warfare an attempt is made to reduce an opponent’s capability through open battle. In unconventional warfare is trying to achieve victory through acquiescence, capitulation, or clandestine support for one side of an existing conflict.



Environments of testing

I think in Software Testing you also might consider conventional and unconventional environments. Below a shot how this could be presented.


Each type of environment has its own type of leadership and needs its own type of strategy and approach.

War strategy

In an old computer game (Civilization II) I learned about Sun Tzu. I understood that this book is a must read for any strategist. Sun Tzu: The Art of War In this book in thirteen chapters an explanation is given how the context of warfare an be defined.


The combination of leadership, environment and goal makes each war different. I think therefore there is a similarity between warfare and software testing. In software testing you also have different objectives, circumstances and resources.

Test approach
When looking at the thirteen chapters of The art of War there is some similarity which can be translated to testing. I tried to visualize it in the image below.
Similarity between warfare and software testing
Is there a similarity between warfare and software testing?
Perhaps there is, in both situations:
- The environment and goal has impact on the type of execution and leadership
- The chance for creating chaos is high
- Dealing with chaos should be part of the strategy
- The desired outcome is victory, only at the end you know about failure
- Best practices is not a guarantee for success, it depends, and most of the time on the context.

Tuesday, March 17, 2009

TestData Generator

A colleague tester pointed me recently about the importance of test data management and the recently offered site towards test data management: Testdata Management.

Behind one of his links there was a free test data generator: GenerateData.com
GenerateData.com offering a free test data generation tool with several functions to generate "standard" test data in several formats: HTML, MS Excel, XML, CSV and SQL.

Next to generate data online you have to ability to download the script which is under GNU open Source License.

I have to admit that this tool is easy to use although I think you have still to manipulate the data for your own use.

This triggers me to search for more opensource generation tools related to testing. A new world opened in front of my eyes, :)

Here some other open source tools:
dbmonster : can be used for DB tuning
benerator: performing load and performance tests
Database populator : Examines the table and populate it according to it (note: downloadable version is from 2004)

These seems somehow promising, perhaps in the nearby future I can try them sometimes.

Monday, March 16, 2009

All those fine test methods and approaches

There are people who think software testing is easy. There are testers who claiming to be experts in the testing area. There are companies who believe their tool is the solution for everything. There are program managers who know that mentioning a method in a plan and using templates solve all problems. There are developers who are able to use development methods as they were intended to do.

I cannot not believe we live in a world this beautiful. I'm a tester and somehow I'm an expert. In certain areas in certain conditions I know more about software testing then others. And still I'm eager to learn. This is one of my main goals. I like software testing as an occupation and as a way of thinking.

Only I believe this world is not that beautiful as it is. And we all know it. Most of the people are claiming that their methods, approaches and tools are just the way we have to work and then all problems will be solved. There are testers who work according those methods and are forcing organizations to change otherwise they are not able to do their job. They forgot that a method or approach is just a tool to help and can and should never be used as it is written on paper.

How often did you hear or did you mention that we should do this because it is: ITIL, CMMi, TMMi, Prince2, TMap, ISTQB, Agile, RUP, or whatever method/approach/ best practice? How often did you ask yourselves if it values the method or the organization?

As methods are often models how to handle things. A model is just a representation of the actual world. Models are fallible and therefore methods too. Using those methods as written makes you fallible.

Fortunately, because of human nature we are trying to avoid being fallible and we are striving for success. Preventing fallibility we adapt. We adapt our vision about methods and adjust our approaches. By doing this we have to be aware that we change the chosen methods/approach which supports the organization.

This is what makes software testing interesting for me; finding solutions under given circumstances reflecting against organizational defined methods and models using the boundaries of those methods/approaches/tools/best practices to provide information about risks and benefits for the organization when rules are broken.

Therefore software testing is not easy and we are not experts and there is not one solution. Testing is fun and should be done by people having expertise and gaining knowledge, able to adapt and having the courage to think out of the box, within the context. Testers should use their knowledge about methods etc. to value organizations instead of preaching their knowledge of a method or tool is the best solution.

Sunday, March 15, 2009

Pretending to be green and not explicitly measuring it

Last year I already introduced on my blog "Ecoliability" as a potential new quality attribute. I still believe this quality attribute can be of use to explicitly measure how "Green" a company or software is.

On CeBIT green IT a entire hall was dedicated to innovations related to energy saving solutions in the IT-world. Looking at this site it seems those solutions are for a wide scale of potions, from efficient usage of processor time through cloud-computing or SaaS till hard ware solutions to save energy.

If organizations pretending or intending to be green; why not explicitly measure this? Most of the software testers like to talk about quality attributes when they need to test something. It is common knowledge that if certain quality attributes getting more attention in projects it have a positive influence on some attributes and a negative influence on others. For example increasing functionality could have positive effects on usability and efficiency only the sustainability and portability might be lesser.

Management make decisions on those effects. If this is so important for management to decide what to choose based on the impact on certain quality attributes this new quality attribute "Ecoliability" can also be of use and must be introduced.

If we start using this quality attribute it can help management to decide if they accept the costs on other attributes for being "Green" or forcing projects to look for other solutions. If being "Green" is a management decision based on a company’s vision we should help them making it measurable and therefore visible.

If you are "Green" you might think about this and start discussions about this new attribute.

Monday, February 23, 2009

Do you change your testing strategies?

Just out of curiosity I'm wondering if you change your testing strategy. Is there room for change? Are you willing to change? Are your brave enough to change?

Deliberately I'm not asking for the need for change. Because of the current recession I think there is an urgent need to change. The question is what should be changed?

In many projects, people are working with proven technologies and methods. Some of the methods are very useful to cover all risks and measure the quality proved by those methods. Some methods are useful as they rely on best practices. Some methods are necessary as organizations are forcing to use those kinds of project management methods.

In all cases those methods are used to solve problems for organizations. I'm just wondering if there is time enough to keep using those solid methods. Is there budget enough to measure all quality? Are resources and skills available to make it a success using those methods?

Perhaps it is time to change. With change I'm more thinking of a context driven test approach as James Bach, Cem Kaner, Michael Bolton and others are suggesting. If I understood that approach good enough the basis of that approach is defining the context first instead of defining the framework for the project first and the "proven" test method.

When thinking about the context and considering the time we live in now we might understand that resources are not available to make it a success. Because of a clear context decisions can be made to accept risks.

Under current conditions risks have to be taken. Accepting risks might lead to new opportunities since the focus is just there were the organization sees it challenges, opportunities and problems which have to be solved.

I'm just curious:
- Do you dare?
- Do you care?
- Do you have enough resources, budget and skills to continue as you did in the past?
- Or do you change?

Sunday, February 22, 2009

How to question errors?

Recently I was in a discussion with a tester/user who wanted more information about the cause of an error and the provided solution. It took quite some time to understand each other. I have to say that I made a huge mistake here. I listen to the tester why he needed more detailed information only draw the wrong conclusion.

The conclusion was merely: Why do you need all that information if you can check if the symptom is solved, it is a minor change and has minimal impact on your process. Demanding more information although no agreements with development are made to provide this will extend the cost of the solution.

Basically the problem was not the level of detail of the documentation and the provided solution. The tester just wanted to know if the developer understood his problem enough and wanted to check until which level the analysis was done.

It took more time to understand each other: the tester wanted more information so he could decide the impact and define the test depth. I tried to convince him that if there is less chance of damage, it might not me wortth to go into this detail.

This made me think how we could talk the same language. for this I created a picture, perhaps this will help in the nearby future to discuss.

The first picture is to identify the type of error and how it is solved. The columns at the end can be used together with the developer to get a mutual understanding about the error, cause and the solution.



The second picture can be used to visualize the questions we might ask and to monitor if the effort for gaining information is worth in relation with the solution and impact. I also left here the possible questions as they have to be defined in their context.


Although I know it is not covering everything, I think it still can be used to visualize our thoughts.



Understanding the problem: the skilled helper

How often are we offering solutions to help people based on our experience? Somehow we are trained to identify possible problems and judging them against our best-practices. Because we are always in some kind of time pressure we tending to help claiming our approach is proven and has some successes.

For example: we are asked to structure the test process and our best practice is the TMap approach. Instead of identifying the actual problems we are defining the test process based on their phases and to see what context we have to work in we perform a so called Product Risk Analysis (PRA).

For some time I'm trying to understand the Context-Driven-Testing (CDT) approach. I think the main idea here is to understand the context first before you come with solutions.

To understand the context you first have to listen. This reminded me about a book I read some years ago: The Skilled Helper: A Systematic Approach to Effective Helping by G. Egan.

I think this approach fits well in our profession also. Aren't we also "helpers"?

Egan's approach consists of three stages which might prevent us jumping into conclusions/solutions. Every stage consists of three steps with continuous evaluation (E).

Stage 1 - The present scenario: Help clients identify, explore, and clarify their problem situations and unused opportunities.




Stage 2 - The Preferred Scenario: Help clients develop goals, objectives or agendas based on an action oriented understanding of the problem situation



Stage 3 - Formulating Strategies and Plans: Help clients develop action strategies for accomplishing goals, that is, for getting from the current to the preferred scenario


I think these stages and steps can be combined with Heuristic Test Strategy Model can help define the correct context and choosing the best strategies and plans for that certain context.

For more information about Context Driven Testing see:
The Seven Basic Principles of the Context-Driven School
Weblogs:
James Bach
Michael Bolton
Cem Kaner

For more information about Egan's approach you also might check: Introduction to ‘The Skilled Helper’ A Systematic Approach to Helping

Saturday, February 14, 2009

Problems and responsibilities

Headache
During a master class SCRUM Jens Østergaard and Bas Vodde gave us the phrase "If a problem gives you a headache then probable it is not your problem."

The idea behind this was that the headache is initiated because you are not able to solve it immediately which made you think without reaching the solution.

Getting headaches is sometimes inevitable in software testing. It is easy to handover the problem to some one else. Will this be the correct way? I think you have to check if it lies in your field of responsibilities and your field of influence.

Responsibility
I always try to think in the number three-approach. Based on this I came to the following classification of problems and responsibilities:
- problems in your area of responsibility, only you don't know how to deal with it
- problems just beyond your area of responsibilities, only it has to do with your field of profession
- problems beyond you area of responsibilities, only you were eager to notice them

In your area
If problems can be assigned directly to activities you are responsible for, then you should not hand over the problem. As you got already the headache, try to find a person who is able to help you. This help can be given in support or advice.

Just beyond your area
If the problem is not related directly to your activities you should check if it is related to your profession. It would help the person you handed the problem needs your support, your expertise and your commitment to help him.

Outside your area
This is an area you have to be careful because you noticed things which should not be your case. Here you have to decide if you are approaching it in a formal way or informal way. Formally could be by escalation or documentation (e.a. e-mail). Informal would be by identifying the person in your field of influence that might be able to guide the information.

Problem solving the headache
To deal with your headache, you might:
- Ignore it, and hope it will go away over a while;
- Take an aspirin;
- Identify your responsibility and take that.

Thursday, February 12, 2009

How many bugs do you need to make a metric?

Last year I attended a presentation by Markus Schumacher about metrics. His presentation can be summarized by his saying: "Metrics should change behavior".

I cannot forget about this saying while I'm busy collecting and presenting data. Often I see presentations of figures how many bugs we already have found during testing. Discussions are held how many bugs we didn't find. A lot of time is spent to make nice charts from this data.

Michael Bolton posted a very good blog posting about metrics: Meaningful Metrics

In his blog he also suggests to create metrics which rising questions instead of creating metrics to be used as a control mechanism.

I don't think you can expect from management that you skip presenting control-metrics from the start. It is also not appreciated you don't present any metrics. What I often do is start simple by presenting the number of issues found in a certain period and what their severity is. Although this won't change behavior or raising questions about quality as it should do, it creates the room to discuss what should be measured. Collecting data and creating metrics is also a dynamic process.

I would summarize this process in the following steps:
1. Ask what metrics are needed immediately, let management prioritize because it cost time and effort;
2. Ask what their goals are for those metrics;
3. Define the goal of the selected metrics together with the customer and define criteria how to use them and to monitor the benefits of those;
4. Add some 1 or 2 other metrics you think which are needed to meet their goal;
5. If they got used to those metrics you might even visualize it in charts;
6. After an iteration or a weekly meeting check if it met their goals;
7. Adapt your metrics based on requests, this also can mean you add or remove metrics;
8. Sometimes you have to dive into your collected to see if questions can be defined based on historical data;
9. If people got punished because of the metrics, remove that information as interpreting metrics is a profession on its own and often the metrics we are dealing with can barely be used as indications;
10. Do this every time again, don't rely on the believe if things were good enough the last tie then it should be good enough for now;

Coming back to the title: "How many bugs do you need to create a metric?" I would say: None. You will need at least one question where you have to find an answer to. This answer should lead to change in behavior, make people move or raise more new questions.

Numbers of bugs can be used to create fancy charts. As long as those charts doesn't move people I would say that charts presenting >1000 bugs, 10 categories over a period of 10 weeks are very useful if you want to exercise with colors and chart types.