Saturday, October 25, 2008

If tools are the answer for everything

Often managers are claiming they want to automate the testing process or the test execution.
Lately on seminars or conferences tool suppliers are showing their solution to all kind of answers.
Sometimes the testers are able to convince that using tools can only be fruitful when the process is in place.
Occasionally I wonder does the cost pays it benefits.

Are tools indeed the answer for everything?
When I started working in the software testing business, there were only a handful of test tools available. In certain situations the actually worked. In most cases they became shelf-ware.

If you start searching for tools to support software testing now, then you need more hands to count them. This raises a question to me: Is there no tool which can help us out? Normally I would say: it depends. In this case I'm more tending to the statement: If there are so many tools available, then there is possible a market for it. And if there is a market for it, it has to be based on the principle that "My tool is better then yours". Only other tool supplier would tell the same about you own tool. Based on the number, the chance you find a tool fits your needs is minimized. As by the number the chance on distraction will increase. This distraction might result in a wrong tool selection; which results in creating shelf-ware. Or results in uncontrolled tool implementation.

There is no way back
Perhaps I already mentioned this statement before, only I keep asking this question to myself every time the issue for automation is asked. A former teacher told me that "once you start automating there is no way back."

If you select a tool and start using it; you should have defined a plan for it about the ROI's. If you managed to embed the tool successful in you organization for that moment. You get limited in your flexibility to improve your processes as they have to fit in your tool choises.
Because of this, the used tools might withholding you to improve further.

An option for the future?
What I would like to see is that tools become more flexible. perhaps more SAAS-likely.
Give the organization the option to choose for:
- which UI to use
- which functions to use
- make the application able to use every function you need.

Possible benefits
As organization don't have the need for all functions in current test tools, the tools can be cheaper. The customer pays only for those functions he actual need. Also the need for functions can be incorporated in a improvement plan. this helps also to plan the ROI over the years. more important, the customer can decide what the life-time is for those certain functions and if the process of testing is changing, they are able to use the same tools with different functions.

Perhaps tool vendors should answer the question how they are able to support benefits in the future instead the current need. How they are working on supporting an organization not only to use their tools; also other solutions. As Tools should give an answer to a need and support an organization with obtaining their goals, not only the current processes.

Now: Tool Vendors: Support us with tools able to give us the ability to choose for functionality whether it is your functionality or others to make us succesful using our solution.

Saturday, October 4, 2008

Ignore priorities: Transport Driven Testing

I think we are used to work with priorities related to what to test or what not to test. Based on priorities we also demand deliverance of functionalities by development.
Business priorities are also used to schedule which functionalities should be delivered by development first to make sure that those items are delivered first which gives the business optimal gains.

Sometimes also functionality is delivered because it was easy to fix.

In SAP you use the term "Transports" instead of terms like: "deployment", "delivery", "shipment", etc

To control and schedule transports in SAP you can use SAP Solution Manager. A benefit here is that based on a status you are able import your transports from the test environment to the production environment.

To build a solution in a transport you have to attach them to Solution Manager tickets. A ticket can contain one or more transport numbers. With these transports you made modifications to SAP objects. Each newer ticket might contain modifications on the same objects. a newer version of those changed objects exists now in the library.

Here comes the difficulty:
Imagine that ticket "A" contains a small modification which had lower business priority only it was quick to fix. And ticket "E" contains a high priority solution.

What if there is no tester available to test a ticket called "A" and you are already testing a ticket called "E" which came in later in the transport list and contains modifications on the same objects? If you confirmed ticket E is correct and it will be shipped using an emergency transport to production and later on you import the ticket "A", then the newer solution of objects are overwritten by the older version. This might result in a situation that the solution in ticket "E" doesn't apply anymore.

To prevent this to happen you have to force the tester also to test ticket "A" before an import of objects is made. You are now in a situation where imports are no longer based on business need; it is based on transport need. In this case a ticket is defining the priority for testing. With other words: the location of a transport in the transport list defines which ticket has the highest priority for testing and no longer the business need.

In SAP Solution Manager you have the ability to use the Import All List to monitor which transports should be tested first. Based on the status of the ticket like "Tested, OK" you can define which transports can be imported without major risks. If there are transports which don't have this status, you can place a STOPMARK before that transport. All transports after that STOPMARK have to be imported on manual request. This introduce the risk that when during the next test phase those transports are desired by business and have to be imported using an emergency transport you might have to import other objects as well.

In this situation, the release content is based on business priority, you have to make sure you don't get the easy things also in which are easy to fix and hard to get a tester for. When the release content is fixed, you have to test not only on complexity and business gain; you also have to test to make sure that the STOPMARK is set as low as it can be. This means that you have to work according FIFO (First in-First out), first import the objects which are entered first on the list as the next one might contain a newer version of the object.
This means you have to force the test execution based on the transport list, so you might call it transport driven testing.

Sunday, September 14, 2008

Magazine: Software Test & Performance

During searching on the internet for information I ran into this magazine: Software Test & Performance. Though I somewere I already heard about it but forgot it over the time. Although the magazine contains quite some advertisements it also contains some good articles. And that is what it is all about.

This current issue can be downloaded from:
http://www.stpmag.com/retrieve/stp-0809.htm


This magazine can be ordered and a printed copy is send to your adress or you can download the magazine in PDF-format for FREE.


One of their good services is that they also made previous copies available for download.

Monday, September 8, 2008

SmarTEST a newer version



It is already some weeks ago I read a new book related to software testing called: SmarTEST. This book is written by Egbert Bouman from Valori. Though it is written in Dutch I think it is worth to translate it in English as well.

One of the strengths of this book that it is based on a business approach were 3 models are leading in the IPS-model (Information, Processes, Systems):
- For Information the model: IDQ (Information and Data Quality) is used
- For Processes: POQ (Processes and Organization)is used
- For Systems: ISO9126 is used.



Based on these 3 models their approach is explained. Some times as and addition to already known methods in the Netherlands Like: TMap, TestFrame, TestGoal and ISTQB. In other situations it continues as a new view to our business of software testing.

Another strength is the overview like stakeholders and possible acceptance criteria. Or like the difference between terminology between TMap, ISTQB and SmarTEST.

Though the business-view approach is the basis of the book in my opinion. They might have spent more attention towards the full integration of the quality models and their impact on software testing. Still a experienced software tester might be able to play with them their selves.
Does this make the book bad? I strongly say NO! It will be an addition to your collection as it gives you enough food to think about. Like: How many W-models are there and which W-model is good enough. What is your perception of using Agile as development method? Can other methods be a child of this method or are they all part of the Iterative (Evolutionary) approach?

Do we need more books related to software testing? I think so as our world is changing the approaches also changing. Do we need this book? If you are able to read Dutch books I recommend this book.

for more information see: their site: http://www.smartest.nl/
details:
Price: € 39,95
Book, paperback / 280 Pages/ Dutch
Academic Service / 2e edition/ Published 2008
ISBN-13: 9789012125970

Wednesday, August 27, 2008

Reverse Business Intelligence testing

Currently Business Intelligence is quite hot. Lately there are some shifts in BI-specialized companies as shown in the following article
IBM, Cognos, and the End of Best-of-Breed

Perhaps this will make the live of a tester easier as components will be more integrated. On the other side it might challenge us more in the nearby future as those products might be integrated and placed on the market too swift which might lead in only proven technology when organizations are using the total solutions as they are offered.

What I learned is that organizations always need their own customizations. Sometimes they are forced by habits how the business works; sometimes they are forced by law.

Of course there are several ways of testing these systems. Here an approach which might help to identify if the reporting tool is able to create necessary reports based on existing data. (Keep in mind that this is just a simple description of a possible approach.)

Start testing based on the information needed. And work through the system backwards to identify your starting point.

This might be reached by the following steps:
1. Identify the KPI's
2. Identify the reports these KPI's are involved
3. Prioritize these KPI's
4. Start with the most important KPI
5. Identify the data these KPI consist of
6. Identify the source of those data
7. Identify the quality (appearance) of already existing data related to these KPI's
8. Create a test data set based on the appearance of this data
9. Run the reports

The challenge here is when errors occur, it is hard to say were the error lies, is it in the reporting tool, in the data warehouse or in the source data. At least we know that there are errors which give wrong information about the KPI's. If the location of the error is found and it is related to the (historical) data, then it might be better to correct that data instead of continuing with the next KPI. As those error also might influence the other KPI's.

If there are no unacceptable errors you might continue with the next KPI.

Though this approach will not cover any errors based on data created by the system. For this we might have to extend the test approach by the following steps:
1. Identify the functions of the main system which creates, alters this data
2. Identify the data model and architecture related to these data and functions
3. Define you test cases based on these functions, with this you make sure that data creation is of a certain quality

And continue with testing the next KPI.

Monday, August 25, 2008

How much don't we test?

Every time I get involved in a test project there are questions like: How much test cases do you have? Do we test enough? Why do you need that much time for testing? You have to test everything!

To me, those questions are always asked to soon. At the end of a project I can tell them for sure how many test case I have. It is just counting those cases. If we did enough could be answered if the organization is still happy after going life. Why I need time for testing can be identified based on the decisions made about the quantity of available Money, the needed Quality and the given Time.

And as always it should be:
- Money: Cheaper
- Quality: Better
- Time: Faster


What kind of information would it give in front if we answer the questions about number of test cases, and if we tested enough, and why we need that much time?

I think a more interesting question to answer is what we did not or will not test. If we provide that in terms the business knows they are able to pick one of those three options.

Sunday, August 24, 2008

What planet are you from?

Did you ever come into a situation when someone starts talking about testing you asked yourselves: "What planet are you from?" Or did someone else had this question about you?

Perhaps you have the same feeling when you are reading my postings on this blog.

Still this is quite an interesting question. As starting wondering about planets you accept the idea about universe. As in a universe you have more planets. And perhaps I'm just sitting on another planet you are living on. Still this doesn't mean that we don't have any interaction or need for each other. For example the moon has some role making the earth rotate in a steady way.

Wikipedia says the following about Universe: "The Universe is most commonly defined as everything that physically exists: the entirety of space and time, all forms of matter, energy and momentum, and the physical laws and constants that govern them. However, the term "universe" may be used in slightly different contextual senses, denoting such concepts as the cosmos, the world or Nature"

In projects there is sometimes need for testers, test coordinators, test management. And often they apply for those people with general terms like: The person should be skilled in a test method like TMap. The person should be ISTQB certified. The person should be a team player. And so on.

As test professionals we pretend to be experts based on our knowledge and experience in a certain area. Sometimes, perhaps sometimes too often, other project members don't understand what we are talking about. Though they know we exist and we have to make also our contribution towards the project. It is our task and duty to make them understand how we can influence the project success.

This can be a hard task as our position in projects might differ. Sometimes we are on another location, far away from project management and developers. Sometimes we are working next to each other. In all situations we influence each others activities. There for it is important we know what relation we have towards each other. We have to describe the project universe and explain our roles and positions. Perhaps more important, we don't need to understand each other in full detail, we have to accept the existence of each other and the need for success.

Let us be the moon and the project be the earth. Without us the earth would not rotate in a stable position. It is not necessary for the earth how we do it, as long as we can convince them we are in control.

Perhaps the answer to the question: "What planet are you from?" is no longer important. We don't need to understand each other in full detail as long as our position is clear. If they consist that you answer this question you can always say: I'm not from another planet. I'm just part of your universe. I'm the moon who shines bright white light and makes sure you planet is able to rotate steady.

Saturday, August 23, 2008

RFK and knowledge processing

After writing these blogs: The RFK process explained and Request For Knowledge (RFK) somehow I got the feeling that I never could be the first of this concept. This triggered me to search for the word I was thinking I invented it: Request For Knowledge.

Fortunately for me: I had many hits and I came up to several interesting documents. I think I can say fortunately as these articles and presentations confirmed to me that here is another challenge for us testers or even business analysts to continue working on my initial idea. The challenges to collect information in a structured way were we might define the process to get grip on risks based on initial missing knowledge.

Although there is lots of information here some of those I found.

On Indentifying Knowledge Processing Requirements by Alain Léger, Lyndon J.B. Nixon and Pavel Shvaiko, May 2005
In short, presents a first step towards a successful transfer of knowledge-based technology from academia to industry.
I think the relation towards us testers here can be the usages of use cases to identify knowledge requirements.

Some Applications of Conceptual Knowledge Processing by Prof. Dr. Gerd Stumme
In this presentation an short overview is given how ontology’s are related to tasks and examples.

What I liked here was the introduction of the term: Ontology.

Ontology: "An Ontology is a formal specification of a shared conceptualisation of a domain of interest." T. Gruber, 1993.

Ontologies support a.o. the following tasks of knowledge management
• Acquiring Knowledge
• Organizing Knowledge
• Retrieving Knowledge

BOVIS LEND LEASE ikonnect Facilitated Knowledge Sharing by http://www.knowledgestreet.com/ September 2005

Some of the interesting points here is they specified several roles as Seekers and Facilitators.

Another interesting posting I found was the iWorkshop on [a-i-a.com] iWorkshop™ Knowledge Management and Collaborative Work

If we think a defined RFK process is needed to get more insight in the risks towards the software then knowledge processing might be a way to go. To me this seems to be a very complex process. Though this should not withhold us from collecting and translating the information in possible risks.

As in one of my articles I showed an example how RFK's on one area might impact the testing effort on another area. To improve this process, it perhaps might help to work on a certain ontology. As the moments of tasks mentioned can influence the way we have to define our strategy. We have to learn to adapt instead of forcing our process towards the organization.

Related to roles we can take, I think we will be as well seekers as facilitators.

I know I'm not unique with the term RFK. I believe there are some gains to get, focusing on this process also. It might help us define improvement suggestions towards the organizations. As well in front of a project as well during evaluation of the project.

Sunday, August 17, 2008

The RFK process explained

In an earlier post I submitted the idea of RFK's (Request For Knowledge). The basic idea of monitoring the process of RFK's is measuring were formal requests are made and how and when they are answered.

In software development processes there are some number of methods and techniques to deal with a part of this like inspections, reviews and walkthrough. Based on their character, formal/informal, sometimes notes are made if it is a failure in documentation or a when it is a question is something is unclear. Basically this covers the monitoring of knowledge request for a part when documentation or requirements are already available.

In my opinion you can start the RFK process much earlier. You can start when the project is initiated. I mention "can start" intentionally. As I strongly believe you should not change you development process to this RFK-process. It should be otherwise around. You should define you RFK-process based on your development process and also your organization.

Basically the steps for the RFK process can be defined as following:
1. Define a Knowledge-need-landscape;
2. Name the development method;
3. Identify participants and stakeholders;
4. Define the areas were information is needed related to the system
5. Define the risks and impact of missing knowledge within the method;
6. Define the phases of monitoring the RFK's;
7. Select your toolset how to monitor RFK's;
8. Report on Impact and status of RFK's;
9. Get agreement how to deal with this impact;
10. Adapt your processes based on made decisions related to RFK's;

1. Define a Knowledge-need-landscape
For adapting your RFK-process to the organization you have to identify the locations where a possible need for information exists. To get a clear picture of this it is important you know how your development process will be. Often you will see that a specific development method is chosen.

2. Name the development method
If project management did not named the development method you have raised you first item for the RFK-list. In my opinion it is important to make a statement of the chosen development method. As every method has its strengths and weaknesses. These strengths and weaknesses have to be translated by the impact on knowledge gathering. Often you will see that the weaknesses of a method are translated in other steps which make the method a customization. This will consist in newer possibilities for information needs.

3. Identify participants and stakeholders
If the method is know you will be able to identify the participants of the project. Participants are potential persons to ask for information. There are also stakeholders. If you have identified these you are for a huge part able to tell were information should come from. Only keep in mind, also stakeholders can ask for information.
It is important that mutual agreements are made to provide information, how it should be delivered and when.

4. Define the areas were information is needed related to the system
Based on the RFK-landscape you are able to tell were information is needed. You might start mapping this to the requirements of the new system. The importance of requirements can also be used to classify the importance of RFK. A possible method to do this is using the MoSCoW-method. Next to linking them to requirements you have to link them to the processes and sub processes of the system as I mentioned in my previous post, for some processes they are not that important, for others it is most important.

5. Define the risks and impact of missing knowledge within the method
It is essential to define the risks and impact of missing knowledge. This will help you define the acceptance criteria. If some risks are unacceptable, strong agreements have to be made here and also monitoring should be more in place.

6. Define the phases of monitoring the RFK's
Not all projects need full monitoring. You have to adapt it on the organization as well. If an organization decides not to spend a lot of time on the requirements phase to get them defined in detail, you might decide to monitor not during this phase on requirements, only in the next phase when requirements become detailed enough. What you might think about is that besides developing the system, often a parallel project is started to create manuals and courses. You might also take this into consideration as the information request should match the RFK's in the development cycle. Perhaps new RFK's are initiated. These can be asked to the system under construction as well.

7. Select your toolset how to monitor RFK's
The toolset is used to register the RFK's. This can be done in a home made tool, as I don't think there are already tools to be used for RFK's. You also might create a simple list using a spreadsheet tool. It is important that the tool is flexible as the need for information based on RFK's might grow during the project.
Another tool might be a flip-over on the wall were you place sticky notes for every request.

8. Report on Impact and status of RFK's
Besides of registering RFK's you also have to report on these. And more important, people have to act on the information you provided.

9. Get agreement how to deal with this impact
To make people act on provided information it is important that you have defined the roles and responsibilities for acting on RFK's impact analysis. If this is not done the impact on not acting might have negative influence on the project, you should be able to point people on their responsibilities.

10. Adapt your processes based on made decisions related to RFK's
When RFK's are raised they can be neglected or granted. Either way this has impact on all processes in a development cycle. For instance, if RFK is granted, this leads to newer information. Related to the testing process you have to identify what impact it has on your test cases. If a certain amount of RFK's are rose for a specific area you have to check what your test effort is on that area. If it was low, perhaps you have to increase it as that area is a potential risk of errors as information was not complete or changed over the time.
This also means that you might have to change your acceptance criteria during the testing phases.

When I start writing this article I thought about painting some kind of picture how this process would look like. Only this would reduce the adaptability to all kinds of development methods. Therefore I kept it with defining some steps. I hope these give you some help and convinced you (perhaps just for some small bits) that also RFK's are important.

Sunday, August 10, 2008

A project model from Pettigrew in testing

One of my former teachers though me a simple model about the success of a project. I think this can also be used for the testing process.

1 = Experience the need (as well the internal as external needs)
2 = Clear vision
3 = Capability for change
4 = Good start, (Executable, plan of approach)

I shall not explain much now on this model. Just take your time and think your recent project over. See how they fit in this model. If you saw one of these 4 situations occur, you might now be able to find the actual cause of this result.

For the future, it might help you identifying much faster the direction of your testing process. And perhaps you have still some time to behave such to fill in the question marks prevent your process is getting out of control.