Benchmarking test effort time useful?

Recently I was asked by another professional about the information which is created during a test process. Which products are delivered? Which information is reported and therefore collected?

I provided him some information with the remark: It depends.

It used the some of phases from TMap® to give him some food for thoughts about figures/product delivered per phase. This is just short list which cam in my mind at that moment, there are items which can and should be added and or removed.
Preparation
- Review results – provided to test team
- overview progress preparation functional designs/ use cases – provided to test - team/management/ project team
- Bi-weekly progress reports – provided to management and project team
- E-mail/ memo instead of escalation to stakeholder
- Test plan, provided to project team and test team
Specification
- Progress specification of test cases including the number of test cases who owns etc. provided to test team and project team
- Bi-weekly progress reports – provided to management and project team
- E-mail/ memo instead of escalation to stakeholder
Execution
- Issue report provided to project team
- Progress report provided to management and project team
- Checklists test scripts provided to test team
- Daily or ad hoc progress report provided to test team or stakeholders
- E-mail/ memo instead of escalation to stakeholder
Completion
- End report
- Issue data base
- Delivered value

When I met the professional I asked him about the project he is on to. It turns out he was collecting information based on experiences how much effort it cost and how much result is delivered during a period of testing. Based on this information from several different sources he is working on some general figures to support providing information in front how much testing will cost.

This is a very noble idea. Every one wants information how much testing will cost in terms of people, money, resources, skills. Every one wants to know what the results are like number of test cases, number of checks, number of issues found, coverage of testing and more.

Support people who challenging themselves to think further were others have been, continued or stopped.
I hope the professional I’m talking about will succeed and share his information with us.

To support him I will share some thoughts of mine which might help. Somehow I doubt it is easily done to collect information from several projects and compare it with each other and use those results as some kind of benchmark.

I believe this is hard because of differences in:
- Organizations: not only the business differs, they also change in strategy etc.
- Projects: you have to make a distinction between development approaches
- People: teams are not supported by the same people and same commitment
- Technology: It looks like you can compare counting in function points. In my opinion there are differences as technology is used by humans and they have different skills and approaches for example in error handling.
- Test approach and used techniques: these are often adapted based on influences from the items above. This results in numbers with similar names only the value cannot be compared. It will be comparing green apples with red apples. You know about the amount only the taste might differ or not.

Basically it is hard to compare the figures which are collected. What is the value of the number of test cases which are written in a certain period for a certain project? Can these figures be used to calculate the effort and money it will cost? Perhaps adding a certain percentage to be sure?
Can the figures how many cases in combination with found issues be used to tell how many test cases must be executed to provide trust in a system without the system being built yet?
Can the figures how many testers found the number of issues by executing a certain number of test cases to predict the needed resources?

Is benchmarking good? Perhaps some people need this kind of trust and use it as experience because they are missing experience? Is the prediction valuable to use?

I wonder when you are finished with benchmarking/collecting figures/information to provide a fancy chart on which people should trust.

Why should we spend time to collect? I value more the skills and experience I have and the ability to compare them to any new project. Sometimes the experience in numbers can be used, sometimes I can rely on, at least it gives me direction. Should I share it with others? Yes of course, in combination with the context and the lessons learned from it. Can others use it? If they are able to judge it against their particular situation and not using as common proven figures.

Testing on the other side

How often do you see that others believe it is not their failure, it are the others? How often are you sure it is not because of you, it is because of them? How often do you see that you did everything to prevent failure and others are to blame? How often do we all blame the other side?

Perhaps not that often. Still if it is not happening at our place, then it must be triggered at the other side. If on the other side failures are triggered, created and on our side solved/prevented. Why are we spending time at our side to proof it works here?
Currently we seem to focus to measure that everything is OK on our side. We are not to blame when the failure hits the fan. So it must be elsewhere to be found. The other side!

If we are sure that it is OK on our side, and failure is on the other side, why aren't we adapting our strategy of testing and look on the other side? Sot the approach would be: create tests what we should do. This will proof we are right. Now reverse the test scripts and test other things as we reversed our thoughts to blame others, then they must be blamed by things we didn't found. That leaves us the parts we are not testing, the parts from the other side.

The thought is acting opposite. If right is OK, and wrong should be found, we believe everything is right on our side; then the other side is wrong. We should find the failures on the other side. Finding failures is NOT blaming. Finding failures is in this case helping out the others. This means working with them on the other side. This means joining for example development, become a team.

To be able to help we should force us to think differently. This also means that we should force us to change our minds to create space for other thoughts and ideas. Perhaps adding a new role to testing: “the devil’s advocate-tester”. He/she will try to force the team think in a different way; although controlled. This role might change every week just to force us think beyond boundaries.

This might result in an approach were space is created to think differently and to extend borders within a testing project.

So prepare to test also on the other side, just for fun!?

This sounds like testing

Previously compared
This week I already made a posting with a relation between music and testing Patterns in music and software testing. In that post I tried to link it to the way we try to value music towards testing based on identification of certain patterns.

Today I drove in my car to work, normally I listen to the programs on the radio, this time I shared that time. I listened to the radio and again to the band Conorach.

Previously I tried to compare the music with other artist I know. As already spoken I noticed there were some patterns I recognized in the music and based on those I compared it with the others. This time I made also a comparison within the music.

Comparing with numbers
It starts with identification of patterns. If you are aware of patterns you are willing to look at it in different ways. Instead of trusting the known approaches you can search for other relationships.
I think there are certain steps you can make. I'm sure there are books which explain this better than I will do know. Only, books are not available yet/now when the mind is throwing ideas :)

You can listen to music in different ways. Your perception will be defined based on who, how, when, what, why you will listen the music. Are you listening a song you already heard about, listening the whole CD or is it live? Are you in a good mood which fit the music or artist? Do you want to listen since you need to recover from an exhausting testing session? Or, do you need to energy to start with it?
You can listen in different ways, focus might shift/change.

When I was driving in my car I shifted to focus from comparing the music and certain pieces with other artist towards comparing parts between the songs on de CD.

Seems in songs you can use other artists as Oracles, you can also use the songs of that artist as an Oracle. This sounds familiar, don't you think.

Comparing with testing
Imagine you replace artist with system and you replace songs with functionality. This gives the following sentence: "Seems in functionality you can use other systems as Oracles, you can also use the functionality of that system as an Oracle."

To me it seems that music can be used well defining approaches for testing. Perhaps even better then the traditional methods. Ouch, now I might be walking on thin ice. It is not that I am against the traditional project or test methods/approaches. I only experienced too often that maintaining the arguments for using methods is the goal instead of the goal you start using those methods. For me it always helped to change the glasses I was looking to. I tried to look at projects/systems with other point of views. This is necessary as systems are built for humans and organisations. They are also different which result in differences in perception.

Differences in perception is a strength

Music is also different, we all experience songs, music, tunes different, people are different. This leads that perception of music is different. Still we are able to value them within a short period. Some will like the music and some will not. If people have to listen together to music then often is avoided to play music only a small group likes. Imagine you are at work and you play your song and only you like it. Your colleagues might get annoyed by that tune (they wont call if music). This will lead in irritation, lesser performance by them, focus on wrong things. A solution for this is a mutual understanding and knowing that the optimal situation per individual is not obtainable. You have to define a common goal.

Define a common goal

To define a common goal you can use heuristics for it or perhaps use the quality attributes mentioned in ISO-9126. Perhaps this works out. The danger I see is that again best practices become leading instead of first thinking which best practice is the most valuable here. Also, is there a need to learn more best practices first. Learning from music might be a good way.

If a human being is able to value it using their mind and willing to learn from it. Why not use music as a metaphor to help testing software. If people on a work floor can come together about music then it must be valuable to use it also coming together about using systems.

Some instruments are easy to play, especially by experienced people. Some words are better to sing and understand. There might be differences in keeping the tune right. Some can some won't. Compare the guitar sound between songs, artist and time. Even an artist is growing in a direction.

Music is the way
What I'm trying to say is that it might be good if you are following traditional steps to make other moves and learning from music might be such a move. I challenge you to keep learning from situations and don't be distracted by ideas from others like methods etc. Try to listen to the music of the artist I started with: Conorach. I'm curious what you can learn from it. And how you learn from it.

I noticed some riffs from Joe Sattrianni, tunes from the Dubbliners, voices from The Nits, Iron Maiden is also involved and somehow the quietness of Pink Floyd and the piece of Bach. I also noticed other artists I forgot the name. If I try, I can find other songs which seem a bit similar.
I noticed that in a system functionality can be pointed to instruments, some are solid and some are fragile. Some are nice and some are fast. Some are played from paper and some jamming.

If we can come together valuing music would it be able to come together on testing? Accepting the differences in methods/approaches/systems/people?

Patterns in music and software testing

New music
Yesterday I bought a CD from a colleague who plays in a band called Conorach. In the past he already mentioned about this activity and after an announcement of a new CD release I spend time to pay more attention to it and listen to the demo on their site. Somehow I got convinced to learn more about this band. I decided to support them by buying their CD. That CD I obtained yesterday and was able to listen in the car driving back home.

The ride back home was quite a journey; next to learn from their experience I also learnt about how I look/listen at things.

Travelling home
Imagine: you are sitting in a car, driving all alone, it is already dark outside and there are just a few people on the road. You listen to music you were curious about and never heard much about it. It is not that type of cover-band who brings you music of songs you heard somewhere. It is a band with their own sound.

I like to drive in my car when it is quite and dark. It enables me to think more about things I usually don't care/mind/think about. The situation here is that I like different types of music. If I have to define a range it is hard to do as the music cannot be compared. It differs from The Dubbliners to U2 from Bach to Marillion, from Metallica to de Dijk, from the Baseballs to Pink Floyd, from Fats Domino to Rage against the machine. There is no direct similarity between the choices of sound. Though, the knowledge and my experience with these bands shaped my vision and experience with music.

Try to like it
I started to listen to the music which was playing on my radio. Let me call it the "new" music. Somehow I heard sounds which were familiar to me, they were even sounds I like. First thing I noticed is that I started comparing the "new" music with the perception and knowledge I have about my known "old" music. I wondered if it is human nature when you are doing things with an open vision you try to compare it with other similar things you like.
I did. I tried to compare it with music I know and like. I tried to find the best feeling of my knowledge in the "new" music. I did this for every song and noticed that a song as whole item could not be compared with another artist. Parts of it were comparable.

Are there parts I liked?
After a few songs I changed my approach from comparing whole songs with known artists to parts of songs and even usage of instruments to artists and other music. I choose to compare it on a positive way; compare with things I like. I did this on purpose as the trip was not yet to end. It is better to listen in a good mood then in a negative mood. (I assume) Somehow, this approach made me to learn more about the music. I was able to extend my vision. Not only to recognizable sounds of instruments, patterns of music; also combination of instruments compared with voices. I managed to listen how the volume was used and value that experience.

Finally
As trips ends a CD has also an ending. Within the hour I learned some about a “new” sound, how things can be compared, how I compare things, which I had a good feeling about the music, much more to look at and to check. I manage to value this music and willing to spend more time to listen to it. Listening becomes a journey of its own. Therefore alone it must be some good music, at least parts of it.

Relation with testing?
I know there are items written which might support certain ideas, or part of it. Recently I read a blog which is referring to a book from G. Weinberg with respect to holistic thinking and quality assurance. Unfortunately I’m not in the luck yet to read that book/books. To me it doesn’t matter that much. I believe that it values more to a person who is able to come up with own ideas/thoughts then reproducing others; although it is good to support the thoughts with other ideas :)

Looking back to this experience I believe that in software testing we can experience similar situations.
- compare projects as whole vs compare projects as parts
- compare it with one method vs use more models as comparison
- try to find the best of a situation vs wasting energy to worse part
- try to challenge your thoughts vs sticking to traditional ideas
- see it as a tour of experience vs focus on safe processes
- start with open mind vs use best practices to define approach

Personal lesson
From just listening to new music I learned something about myself. I was triggered to be careful to when using best practices. Knowing methods is good, having experience might count. It only count when it is used justified. My thoughts might be fallible, even my first impressions. I believe in the strength of teaching myself and sharing these thoughts with you will be valuable.
It is good for people to be aware of the pitfall to step in old behaviour and come up with “old conclusions”.

Another thing I liked about this was the awareness you can learn from everything if you open your eyes, ears and mind and try to find relationships. What seems to be different is not definitely wrong. Changes focus from negative perception towards positive attitude might result in new energy which leads to new areas to explore.

As patterns can be recognized based on the view you have, you might be aware of patterns. If patterns are in music in this example, patterns are in testing. If you don’t see a pattern, then spend time to focus and defocus.

Awareness can grow if you are open for it and support skills and craft.

Value of certificate or understanding

Today there was a great happening at our family. After weeks of training, my son and daughter were invited to take their exam for karate.
Almost more then half year ago they started to attend this course on weekly basis. Now and then they are showing the moves they learned in the lessons. They not only paying attention to the moves, also to the words, terms and how they should be used.

They are aware that they are preparing for the next grade. Last time it was the white turtle, this time it was the white cranes. Somehow I believe that they understand exactly the meaning of these grades. They know exactly in which order the grades can be obtained. They also are aware of the skills they need to know and able to show.

They took the test and they succeeded. I am proud on my kids as they persisted, listened and behaved in the dojo. They had respect for the teachers and other participants. At the end every one got a badge, a certificate, a sign on their karate passport and an invitation to attend a meeting with a great master. They also got respect from the teacher and the judges.

It seems so obvious that all those items were important like certificate, badge etc. Perhaps it is. Somehow the certificate of my son got lost. It made me proud to see that he did not complain, or else. He felt a bit sorry and still went back to his place. He accepted it as the value of respect was more worth then the certificate, knowing that the teacher told him being a white crane was more valuable. The certificate was just a paper.

This made me think, he might be right. I’m sure he is right. A certificate is just a paper; the respect he earned made him proud and made him to continue. The respect of the teacher made him know that he is allowed to show more skills to others which he was not yet allowed doing.

To me a great lesson I see here is that we are tending to look at our old heroes what they achieved and accept their sounds that we needs certificates. Perhaps we also have to watch out for our young, new heroes and bring everything back in perspective.

After al, a good day that as I learned from my children.

EWT12: Mind mapping and testing

Great minds came together on EWT12
This session of European Weekend Testing: EWT12: Mapping the maps involved using mind mapping to inform the test manager about how to reach a certain coverage for the online map functionality: Bing Maps

This seems to be a mission with too much open endings. I keep referring that one of the major goals for me to participate are knowing more passionate testers and learn during the session and also afterwards. I'm not intending to test the software and find all issues I can found. for me it is also not a game which can be won on the number of issues there are found. Or who reached the highest defect ratio within that hour.

Passionate testers this time were:
Anna Baik
Tony Bruce
Markus Gärtner
Jeroen Rosink


For this session we were working with either the tool Freemind (offline tool) or Mindmeister (online tool with the ability of using 3 maps for free)

On Wikipedia a list of mind mapping software is offered which might be worth to check

Lessons Learned
#1 Prepare practice session how to use a mind mapping tool
Even when you have heard about mind mapping and or practice it sort of, when a tool is used not to share your thoughts but also compare thoughts within a limited timeframe, it is good to spend first some time to guide the participants through the tool.

#2 Mind maps differ in usage, colours and notations
As people think differently, the chance mind maps differ is huge. I believe here we have a thin line how to use an approach like mind mapping. We can demand the participants to use certain ways of notations, this will also limit their thoughts. I suggest providing the participant several ways how mind mapping can be used and how icons and colours can be used. Let the participant decide as long as the decision can and is explained.

#3 Mind maps are not a single mean of communication
In my opinion a mind map cannot be used as a single mean of communication. You cannot use it as test basis when the creator is not providing some explanation. It is a model of the mind of the author of that moment under certain conditions. Often is written and spoken that an image tells more then words. In this case I believe the strength lies in the combination, without words the map is almost meaningless

#4 If mind maps are used in testing keep it dynamic
I think mind maps can be used in testing; perhaps as a certain touring scheme. When it is used; it also should be maintained. When it is used in a test project and a certain level of information is presented, it has to become part of the whole team and the whole team can make changes on it. It can be introduced as an extra output of a stand-up meeting?

#5 Heuristics can be used within a map
You might introduce some structure by using heuristics. I used SDFPOT to play with and continue with that route.

#6 Use multiple mind maps within the team
After I compared the different mind maps I noticed we all have different approaches and different ways of details in the map. I like ton use single words, other like to use notations like test cases and also some times routing within the mind is visible. the strength here is that different levels of detailed information is presented. I'm convinced that not restricting people using mapping in notation will strengthen others by offering ideas.

#7 Mind maps provide more information
When I compared the maps of the mind I see more information then only what is written. If you look at how it is written, in which areas more details are provided and were not, you might come up with questions about importance. or why certain decisions are made. For this mind mapping can be a great approach, not only to tell what is done, also to come up with questions why certain decisions are made.

#8 Mind is changing also the maps
I think using maps like this should not be just once. you should generate this type of maps frequently. To gain more information from the maps how you mind/thoughts are evolving versioning should be used on the maps. On frequent basis you have to schedule also meetings to explain and investigate how thoughts were changing.

My thoughts, my map
When you look at the picture, you see a result of my trials using the tool and adapting some structure in the map. You also see I tried to use some icons, what I will do and what I won't do. Also what I have done and have to do. I played with icons were I think issues are as I noticed strange behaviour. I even tried to see of adding a priority to the items is an option.



Using a tool is great as you can adapt your thoughts by re-shaping, re-placing or even removing without generating a mess. On paper you have to be very careful as mistakes cannot easily undone. You are forced to continue with mistakes. For me, there is more to explore how to use this tool.


Process of mind mapping
Here a suggestion how to use mind mapping. I'm sure there are other ways. I'm sure there are better ways. This is how I think it can also be of some usage.
1. Prepare mind mapping introduction session: exchange knowledge about mind mapping, the tool and experience
2. Assign roles, like tester, analyst, test coordinator/manager, developer, user, etc
3. Agree that no map is wrong or is right, accept there will be differences in level of details
4. When necessary agree on usage of certain icons and colours (not all)
5. Define the mission
6. Execute mission within a defined period (1 hour?)
7. Present and explain the mind maps
8. Make adaptations on you mind map based on mind changes due to new input
9. Make agreement how mind map will be used: every day during a period, or was this one time moment?
10. Maintain mind map
11. Gain information about how the "mind" was working related to mood, level of detail (perhaps the mind was up to something), time behaviour, structure,...

I think it can be used for explaining/presenting and obtain information about:
- the route for testing
- the areas for testing
- the test cases
- the coverage of execution
- which decisions are made, what to test and what not to (lesser details is less important?)
- matching each other thoughts about the test goal of that day
- the possible risks
- the black spots in testing
- ....


WeekendTesting
For those who also want to be challenged by challenging yourselves, you might take part on one of the weekend testing sessions and teach yourselves! Don't hesitate to participate!

For more information see:
Website: http://weekendtesting.com/
Or follow them on twitter
Weekend Testing: http://twitter.com/weekendtesting
Europe Weekend Testing: http://twitter.com/europetesters

First make value then add more value

After reading another article how to measure more and earlier and by doing that saving money at the end. It seems so obvious that people are right when we establish a review process before we start testing. People are claiming that issue found early in a development phase it is also cheaper to solve instead when it is on a productive system.

I have heard and read that this is a written truth. I won’t object to that statement. What I’m worrying about is that people are using at any time the argument: “it is cheaper to establish a review process, or test earlier so issues are found early and therefore it will be cheaper.”

In my opinion, you have to be careful claiming this statement. You first have to earn money before you can calculate the benefits of finding issues earlier. To do this a system must be in use. You cannot spend what you didn’t earn; you cannot count in terms of cheaper when it will delay productivity and therefore cashing the earnings is also starting later on.

My advice is:
1. When you start a project: also define budget for reviewing/ testing earlier
2. When you start a project and budget is less: make wise decision based perhaps on expected value and risks where focus should lie on: sometimes reviewing/inspections/ testing earlier might be cheaper and beneficial above extensive executing lost of test cases.
3. When project is already started and budget is unlimited: you might consider reviewing/inspections of code, test cases, testing earlier etc.
4. When project is already started and budget is fixed: be careful demanding and initiating activities which cost extra money and time, they delay is not covered in the project plan and earnings might become delayed
5. Consider: When project is ended: check if it was necessary to review and if review would allow you delay of deliverance to production
6. Consider: When system is delivered, check if value is delivered and expected money is earned. Answer also the question: would adding a review process make the earnings higher?
7. Consider: If earnings would be higher due to minimizing the costs, would minimizing also result in lesser time to production?

There are enough arguments to start reviewing, inspections or testing earlier, keep in mind that you first have to add value to the business by the application, and then try to add more value if it cannot be done immediately. Sometimes this means that you have to go to production with known bugs to make money to find and solve more bugs.