Weekend testing EWT08: Kept sticking in testing

The start of EWT08
I managed to attend this weekend another session facilitated by Anna and Markus on behalf of the European chapter of Weekend testing The application to test was this time Wiki on a Stick. Part of the mission was using SFDPOT
For more information see:
http://www.satisfice.com/articles/sfdpo.shtml
http://www.developsense.com/articles/2005-10-ElementalModels.pdf

Also check out the site EWT08:Wiki in your pocket for a transcript of the session

leaves me just another bit of advertisement to point you towards the weblogs of the testers who attended:
Anna Baik
Ajay Balamurugadas
Michael Bolton
Tony Bruce
Anne-Marie Charrett
Markus Gärtner

The Mission
For me this session was another great one. In front you will never know what you can learn. This time I tried to be more prepared. have the bug depository running, the screen capture tool in place, the connection with my virtual machine using Debian running, information about heuristics ready, I even read a bit again about them. I tried to prepare what questions I could ask with respect to SFDPOT.

My preparation
Like the other session I attended online it started with an introduction and testing in the first our and a wrap-up in the second hour. This time preparation saved me some time. Unfortunately, and that is how life goes, the start is different then you expected. This time I expected a tool I had to install. Instead it was sort of a webpage which should be stored on a place and could be used immediately. I was a bit distracted by this behaviour and stopped working with the idea to test on 2 platforms.

It took me some time to figure out what type and kind of application it was. While "touring" around the application to see in a quick view what the edges of the application was and what would be interesting to investigate further.

Retrospective
After wards I noticed it was hard to stick to an approach with heuristics based on SFDPOT. I remembered that the next time I should start defining the questions I want information about it. For me this is another way of thinking then I am used to like an approach conform TMap or ISTQB. Most of the time you start with a well defined process. Keep in mind that that will not work in EWT as the time is limited to an hour.

I believe that based on defining questions you can explain a bit about coverage. In this case is explanation more like picturing/telling what you have covered and what not in terms of the questions in stead of how many cases you executed and what that coverage would be.

I explained to the group that it was hard to stick to the initial idea and one of the pitfalls is start testing immediately. Michael explained that it is better to build a model first about the system. I like that idea. You can build a model based on documentation. I think reading the documentation with respect to the heuristic model can help you defining you questions.

Value of documentation and model
Building an initial model based on documentation is in my opinion a good way to start. This seems to look similar to other methods: reviewing intensively, defining all objects, prepare test cases. And then start testing.
Off course everything is sort of based on the Deming-cycle: Plan-Do-Check-Act. Only a lot of approaches are made too robust and formal that you loose time in documentation. And try to strive only for perfection instead of a good mix of result within in conditions.

The goal of documentation is mainly provide information. Based on this information developers and testers are lead to define their approach for building and testing. If the information is weak then the chance for mistakes are bigger. An option is to formalize a process of reviewing and adjusting the documentation to details to narrow the perception and vision of the people who has to build and test it.

To start with reviewing is an approach which might work when time is unlimited. A mistake can be to collect all information, judge the information, derive test cases based on that information and sell an advice based on the test results.

How about first building a draft model of the system and the environment? You can document this, you also can make a draft picture of it or make it up in your mind. Based on this you define your boundaries where to look at. If information is missing, you ask the owners/stakeholders. Keep in mind the time you have. For example: during EWT08, we had the program manager also online, we had only one hour available and instead of claiming and demanding information we tried to find our own way. We could have asked a lot of questions which would extend the hour testing.
This would be another way of receiving information about the system.

Information about the system
Seems that information about the system to test is important, the time window is also a leading factor this in combination of the quality of information. It turned out that the available information for this application was harder to get. Some testers who paid thoroughly attention to that and made a remark about the documentation.

Looking back it turns out that information was available in several ways:
- online manual
- in the tool itself (written information)
- the tool itself (functionality)
- people online (the project manager)
- people online (fellow testers)
- bug repository
- information regarding heuristics like SFDPOT

The needed information was different. My approach was focussed more on the functionality and there were I found something interesting Itried to find the information within the tool. this approach distracted me as already mentioned.

A lesson learned
Lesson learned here: Create a model of the system and see what kind of borders you will find. Check this with respect to the environment. The next time I would explicitly ask me questions like "how it will work when using a USB stick." I would build a framework of initial questions using the SFDPOT and during the tour I tried to refine the questions. afterwards I would try to validate if the answers to the questions are enough to make a statement of it. This can also be a statement of gaining too less information.

I believe we testers should be careful to provide a judgement about systems. When will you know you spend enough time and gained enough information? The challenge here is that we are asked to provide an objective advice although the information mostly is translated based on subjective experience.