Article: The Economic Impacts of Inadequate Infrastructure for Software Testing

Some now and then I find on the internet an interesting article. This is one of them:
Planning Report 02-3: The Economic Impacts of Inadequate Infrastructure for Software Testing

Prepared by: RTI for National Institute of Standards & Technology Program Office Strategic Planning and Economic Analysis Group, May 2002

This document gives some nice examples and depth on software testing.

Article: Acceptance Test Driven Development (ATDD)

Elisabeth Hendrickson’s posted on her weblog Test Obsessed An article related to Acceptance Test Driven Development (ATDD): an Overview

Here she links to an article she wrote about this topic which she presented at STANZ 2008 and STARWest 2008. Driving Development with Tests: ATDD and TDD In my opinion a good overview and example how this could work.

Does the testing school fits the organization?

Can Approaches support organizations?
Here is another posting related to testing schools. In my previous posts I started to explore my thoughts if I belong to a school. Schools of testing, can you decide?

And continued to think if there is some dependency related to organizations: Testing Schools and Organizational Schools

As Paul Gerrard there is a lively discussion going on towards this topic Clients, Contexts and Schools

Today, on this rainy and windy day, I continued thinking about it. Though, I still have the feeling that the quest for answers is not over yet. (Probably there are no good answers, instead solutions that might work.)

Over the years I saw different people using methods/approach what they knew the best and sometimes forcing organizations to adapt their processes so their method might work. I also saw people adapting their method/approach towards the processes of the organization.

Both approaches might lead to the solution the organization was waiting for. And both approaches create new risks for the organizations. Some people say that risks can be accepted based on the change of failure in relations with the cost of that failure. Less people are asking why to accept those risks. To give an answer to that question I think it is important why organizations initial choose for a certain approach.

I could try to search for answers why to choose for a certain school. And try to start discussions about it. Instead, I try to think out side the box and find out how both ways can be good, how schools can help, how solutions can be provided to organizations.

How would my "box" look like?
To create a picture of the "box" you might ask the following questions:

1. Does the organization have other goals to improve the testing process and increase quality of the software?
2. Does the organization have the means to get to those sources which support these goals?
3. What is the life cycle of these goals?
4. Can the tester based on his/her skills and experience commit to these goals?
5. Is the tester able to adapt their approach/method to these goals on continues basis?
6. Are tester and organization able to communicate on equal basis?
7. What is the life cycle of the tester in the project?

Imagine how that box would look like if the answers were like:
Example:

1. The organization is not able to change other processes like development on short time. They also using a test method and there entry criteria to dictate that process.
2. The organization has the means like time, money, though the market is not providing the right people within time.
3. Improving test process is long term goal, extend the project life cycle. Improving development process for documentation is short term goal, only should extend over other projects. Deliverance of high quality software is explicitly needed for this project.
4. The tester they found is able to meet these goals under restriction. This means that the tester doesn't have experience with defining strategies based on the several schools. Though he/she is well skilled using on test method like TMap.
5. The tester is not able to change his approach as he is well trained on one method.
6. Based on organizational hierarchy tester didn't get full commitment.
7. As the market for experience testers is small the tester will attend the project till the end.

Now, close your eyes and investigate the feeling these answers are giving you. Now open your eyes and try to answer the same questions no under best circumstances. Close your eyes again and you might notice a different feeling.

Creating borders
Feelings can lead you into a good direction. Though, it is hard to translate them into risks. To identify risks you need borders. I always try to find the borders of a method. This helps me when a method can be used successfully or when a potential risk might rise. Therefore it is good to have those testing schools be defined. This help to define the borders of your picture. And give you information when you are crossing that border. If you cross the border then this is a sign that a school related approach doesn't fit the organization. And reaching the goals might come into danger.

As organizations are dynamic, testers also should be. I think there is an organization for every school. Only a school might not fit the organization. I don't think that it makes you a less good tester if you belong to one school or a better tester if you know about all the schools. It might help to give the organization an usable solution they need if you have knowledge about them.

Conclusion
A tester should be aware of their skills, be honest towards the organization about their skills. It is not wrong to belong to one school. It might help to know about the "Schools of Testing" and their approach and vision. Be aware that good testing is not a purpose, a good solution should be. And you should help the organization to decide that it is and was a good solution.

Testing Schools and Organizational Schools

In the last few days an interesting story was started by Paul Gerrard and James Bach as I mentioned in my previous post: Schools of testing, can you decide? The key in their story is the term Axioms. Can Axioms be defined, used and maintained to help us testers? Do we need Axioms? Can we live without Axioms? Or as Paul Gerrard mentioned in one of the comments: Is Axiom the right word?

Paul Gerrard's replied on both postings with a next posting: Schools Out!

With those articles in mind and their comments on user input I thought about it for this weekend. Somehow, I couldn't get it out of my mind. As was I thinking I'm tending to the Context-Driven school as explained in my previous post. Somehow Paul Gerrard's post makes a lot of sense. It makes sense to let us think about. Or even better, force us to think about.

To understand it all better I tried to paint a picture about it. To understand the object to paint I did a quick search on the internet and found these besides other information these sources about the Schools of testing.

Presentation of Bret Pettichord: Schools of Software Testing these schools are mentioned:

• Analytic School: sees testing as rigorous and technical with many proponents in academia;
• Standard School: sees testing as a way to measure progress with emphasis on cost and repeatable standards;
• Quality School: emphasizes process, policing developers and acting as the gatekeeper;
• Context-Driven School: emphasizes people, seeking bugs that stakeholders care about;
• Agile School: uses testing to prove that development is complete; emphasizes automated testing;

On the Blog of Cem Kaner: Schools of software testing these schools are mentioned:

• Factory school: emphasis on reduction of testing tasks to routines that can be automated or delegated to cheap labor;
• Control school: emphasis on standards and processes that enforce or rely heavily on standards;
• Test-driven school: emphasis on code-focused testing by programmers;
• Analytical school: emphasis on analytical methods for assessing the quality of the software, including improvement of testability by improved precision of specifications and many types of modeling;
• Context-drive school: emphasis on adapting to the circumstances under which the product is developed and used;

They already presented the differences between the schools. Therefore I shall not continue in my own view of schools. What could be interesting is the similarity of these schools. They reflect some timeframe of human thinking and approaching the world.

Something similar also occurred in organizational thinking. In Organization theory, a strategic approach, chapter 2 by Narayanan & Nath they mentioning 3 types of schools:

1. Classical theorists
2. Human relations school
3. Early modernists

Short description of the 3 classical theorist schools: Short description of the 2 Human relations schools:
Short description of the 3 Early modernists schools:

To me it seems that history is repeating in small and much quicker in the Information Technology industry. These schools were defined over the last century while our schools evolve in the last few decades. The challenge might be now how to make our schools fit in the Organizational schools. Here I make the assumption that Organization still tending to one of these schools.

To make testing fit and also work in those difference organizations depends strongly on the skills and experience of the tester. As Testing is getting Global and our occupation has new testers and experience testers it would help that we speak about the same thing. And also be able to explain it to the organization we work for.

In this I see at least 2 levels of communication. One is based how we communicate within our community. This can be done by explaining the principles of our schools. And try to work according to it.

Another level of communicating is making the organization understand what we are doing. This should be translated to the "Organizational School" the organization is based on. To translate we have to explain how testing is helping the organization and what we meant with it. If we are able to this we might be able to get the best out of our schools. I think using Axioms can help to explain the organization what we are doing for them and what we mean with them. Picking the Axiom can be depending on the school we tending to.

Perhaps there is some relation to Law's, we have different kind of laws, they all are captured in the book of laws. Only they apply only to a certain area. Those laws are made to communicate to each other and to make sure we are talking about the same.

To write the book of law's related to software testing would be wrong to do. At least now as we are working in a dynamic environment this needs some kind of flexibility. Though, a book or list of Axioms might help to explain to others what we mean with certain terms and how they can help the organization.

Coming back to my initial questions:

Can Axioms be defined, used and maintained to help us testers? We can define them and use them. Only maintenance would be hard. As who should maintain them.

Do we need Axioms? I think it depends on the skill and experience level of the testers and the understanding of the organization of our occupation.

Can we live without Axioms? I don't think so as there are still projects failing based on commitment of organizations towards obtaining their goal of quality understanding and related to that their undrstanding of the need for us tester's.

Is Axiom the right word? I think it is a better word then Law.

Schools of testing, can you decide?

Today I noticed a posting of James Bach on his blog called The Gerrard School of Testing

In that article he discusses the axioms Paul Gerrard posted on The 12 Axioms of Testing.
James also mentioning the context-driven school and her The Seven Basic Principles of the Context-Driven School

As Dutch is my native language I first had to find out what axiom means. According to Wikipedia, Axiom it means the following: "In traditional logic, an axiom or postulate is a proposition that is not proved or demonstrated but considered to be self-evident. Therefore, its truth is taken for granted, and serves as a starting point for deducing and inferring other (theory dependent) truths."

And this made me think about what school I think I belong to. When starting in the software testing business I learned about one test method. And what that method is saying I took for granted. I assumed it as the initial truth. During practicing I noticed that a method on its own cannot take for granted as it depends on the situation. And sometimes situations seem to look-a-like. Practices from previous projects turned out not completely covering the situation I needed them. They could be best practices, though they need to be altered into new practices.

In all cases the Human Factor was playing a huge role in success of implementing those practices. And beside this Human Factor, also the choice of the organization for development method made a difference. As this lead to the way they intended to communicate in the project as by iterations, increments, documents, delivery moments etc. And what was important to be delivered. In some cases deliverance of a product was a goal in it selves. As some functionality was removed and the initial problem which lead to this development was just partially solved.

This made me think that deliverance of a solution for the problem should be the goal and not deliverance of the product it selves. Though this depends on the situation I was into.

Writing this all down here, I tending to think I could belong to the Context-Driven school, as the principles I believe in fit more in the 7 principles of them. This doesn't make the axioms of Paul Gerrard obsolete. Only would it be short minded to say that those axioms are more the beginning of a vocabulary where instead of explaining the meaning of them the way to get there is explained? That those axioms are communication rules where, what we are talking about?

In the last case for some organizations it would help to talk this language. Though, I'm more tending to say that it depends of the context. Perhaps there are situations where other axioms also work within that organization or project.

Though I'm tending to Context-Driven, as it also supports my thoughts about other articles related to Open System Thinking on my blog, I need to learn more to say that those Axioms are not useful for me.

Cem Kaner made a presentation available related to Context-Driven School which also might worth to take a look at.

[Jeroen: Added on 2-23-2008] Paul Gerrard made another posting Schools Out!. This is worth looking at. It might help us understand how to look at Axioms in respect to Context-Driven School.

I'm not a developer, thought this might help us testers

In the past I ran into an article about visualization of software evolution. It took me some time to dig it up again on the internet.

On this site CVSscan: Visualization of Code Evolution the author ir. Lucian Voinea PDEng gives 3 example tools how to visualize the evolution of code were he claims it can be used as "tool aimed at developers involved in maintenance projects".

Another paper of the same writer on this topic can be found here: Visual assessment of software evolution by L. Voinea, J.J. Lukkien, A. Telea

On his site he explains and offers the following tools:

• CVSscan: Visualization of Code Evolution (file level)
• CVSgrab: Visualization of Code Evolution (project level)
• CSV: Code Structure Visualization
• DreamCode: C++ Abstract Syntax Tree Visualization

You perhaps also want to read the paper he made available from that site called: CVSscan: Visualization of Code Evolution

I think there is also another opportunity here. I think it can be used in Agile/SCRUM projects also. Were writing code is based on usage for that iteration.

Assume that code/functionality changes or better evolves during the project. And as tester you don't immediately know were to look at. Based on turbulence in evolution of code you might decide to test it with more or lesser focus.

Or assume that you first paint a picture with those tools and then measure if the evolution matched the effort you spend on defining Unit tests or other automated tests.

Though I'm quite new on this topic of testing, perhaps others see some advantages of such tools for us testers. Or have already experience with these tools.

If you can not beat the paradox then join them

I recently read an article written by Zeger Van Hese, Jun 9, 2007 called: Software Testing: A Profession of Paradoxes? This paper won the Eurostar award for best paper 2007 Although I don't know about the competetitors, I can imagine why he won it. The article explains in easy and general terms about the paradoxes we testers have to deal with.

Some paradoxes are supported with good examples giving me the feeling: "I have been there, I have done that?" I think if you also come in the position to explain to your manager, project leader or wife/husband why you have such an exiting job, you just hand them this paper.

I hoped there was more information available on the Net which teach me how to deal with paradoxes. Or even better: How to beat them.
Though I didn't find my answer, during this search I found this other article The Paradoxes of Change

Some of the expressions mentioned here are worth to think about:

• "Stick to your knitting"
• "If it turns out I really have to change, I'll change."
• "You find your craft by doing the things that are the most difficult."
• "We must live in the paradox"

Perhaps:

1. Stick to your knitting can be translated in testing terms like: Let me just do my work, you give the code and I'll test. Or, let me just use these techniques as I know these very well.
2. I'll change might be translated to terms like: Give me my marching order and I will march, any direction you like. If you telling me that we tested enough, finally I will agree.
3. Doing the most difficult things perhaps represents the idea of helping the organization by honor as there is currently none available and the job has to be done. I will inform you about the risks
4. We must live is perhaps supporting the idea that we as team might find out difficulties and we as team will challenge them.

What have I learned after reading those articles and writing this post? At least that writing something to combine things explaining those paradoxes without repeating them is hard to do. And also some believe that we testers have to be aware of the existence of those paradoxes. Understand them and most of all, recognize them. And decide for our selves if we keep knitting or join that live with paradoxes.