Testframing: a bit of my perspective

Last week I attended a tutorial on testnet spring event 2011 given by Michael Bolton on Testframing. I already read something about (see: http://www.developsense.com/blog/2010/09/test-framing/ )and now had the chance to see it work. it was a great day as it made me think more about this concept. To preserve my thoughts and ideas I wrote this article. Perhaps you can learn also a bit from it.

Initially I thought it would be an approach were you frame all the tests in constructive sentences, and write them down. So initially it seems to be the same as I was told to do in the past: write test cases and then execute them.

I was wrong then: Why would you write down or use valuable time to write down test cases till the lowest detail when your mind is able to work with the speed of thinking. So there must be more to understand.

During the tutorial I noticed that one of the strength is the way you use a mission to guide the story you want to tell and the observations and actions you do how the story can be told. While explaining there was interaction between the listeners, when something was not clear you change the vocabulary of the proposition or perhaps the vocabulary itself. You even extend the frame using formal or informal connectives

I think that the difference between writing test cases and test framing it takes you much further, it combines the strength of a test strategy with thinking about the justification of testing together with(when necessary real time) adaptation of the path in the system you tending to follow ending up with the explanation of the results you found combined with the justification why you did it. This can be done very fast.

In other approaches like TMap we tend to call it in the Netherlands, there are all kind of delays involved, first the plan, then the written cases, then the execution and after a while a verdict about quality which was stated in the plan.
In some way it is a structured way of working, is it testing? This is another discussion I will not start here. There are other sources about this topic.

In short I learned more about test framing that it is a way of telling a story why and how you tested something combined with the results you found and what you think it tells about the system.
Testframing to me can be done by writing it down, in some cases this makes sense perhaps because of some complexity is involved or some kind of uncertainty about the system you have and you use it to validate your story.
Sometimes you actions are less important and easy to explain afterwards, then you use it without writing it down, just explain it.

Somehow I see a lot of potential in this way of testing. one of the reasons is that it keeps me thinking if I'm doing the right things.

On the same day Michael Bolton also had a keynote were he explained about the dark and the bright future. This was a great view. In the birghter view he explained about first order measurements (some more information can be read at stickyminds.com:
Three Kinds of Measurement and Two Ways to Use Them By Michael Bolton

While explaining first-order-measurement he used a connective "or" this triggered me to think that this is one of the strengths of testframing. You are able to adapt you story based on the first-order-measurements. My first reaction was: This is great!

If I'm able to tell a story about why I'm doing something with what mission (direction) then I'm able to tell the stakeholder/person who cares why Im worth paying for and spending time and resources.
If I'm able to make visible that I'm able to react on circumstances, the information from first-order-measurements, which are happening now instead which happens in the past (the delay when test scripts are written down and the explanation is proved later on) then I provide my self a tool to see if Im on the right way of acting and providing information which are accurate and supports transparency.

Looking back to testframing, I see some different ways to use it with respect to logging the actions:
- use it by thinking in testframing: nothing to write down or explained.
- use it only using your mind, thinking about it and explaining what you have done/doing
- use it in front and write down some notes how you think you want to approach the system
- write down while testing how you framed the mission
- pair testframing: you tell and share your thoughts/mind and another person make notes without interrupting you thinking process (this is another role than the person who challenge you while testing)

Another dimension here is the measurement-order. I think first-order-measurement is an valuable addition to be aware of while framing. Here you can do also some logging as the measurements changed or confirmed the path you were following in your mind.
- before testing and framing first: you might consider which propositions might provide valuable information
- during framing and not logging the testframe itself: log the moment of measurement and result, here you might wonder if you also write down the impact on your story.
- during testframing and also explaining the result: tell the story not only what you have found and why you did it, also explain what impact the measurement had on your frame and ask if you valued your measurements properly. If not there might be some valuable information left behind the framed story.

Somehow I believe testframing is a great way to test a system fast with the right transparency and justification. it is not just a trick you can learn, you have to practice to be able to do it and learn to become better at it. It is a way of thinking and communicating with others which is not only challenge the system as you do it, it also challenge you by explaining how you do it. Perhaps it also tells the story about the tester.

I hope this article challenge you to think a bit on Testframing, for me this is just the beginning in learning about testframing looking for opportunities and time to start with it and learning about it.

Testnet Voorjaarsevent 2010

It happened on may 12th 2010
Last week I attended TestNet voorjaarsevent. Testnet is the Dutch Association for Software Testing. Besides several workshops and small events there are two major events, in the spring (voorjaarsevent) and in the fall (najaarsevent).

Last week the so called "Voorjaarsevent 2010" happened. As usual I got a bit prepared, made the selection of presentations I would like to attend and also which not. What I see often is that the presentations are dealing with old concepts in new clothes. This time there were a few which got my attention.

General impression
* Location: very good, as well as the main entrance, the route to get there as the overall presentation
* Dinner: Good, it was tasty and enough.
* Drinks: like to see the ability also to have drinks like coffee or tea while not attending a presentation
* Exhibitions: Well arranged of the main entrance, good to see those familiar names again and again.
* Presentations: There were a few I would like to see, as what I saw besides the first one, good. Still a lot of old work in new clothes
* Key-notes: Expected more fro the first key-note related to the main topic: "Secure testing"
* People: open minded and pleasant

Program (copied from TestNet.org)



Intentions to see:
- Key note from Stuart Reid: "Improving testing - with or without standards"
- a "debate related to ethics and testing" lead by Nathalie Roosenboom de Vries- van Delft & Budimir Hrnjak
- Rudi Niemeijer: "Safety helmet prohibited" (unfortunately this was parallel with the debate and I had to miss this one)
- Jurian van de Laar: "Testers helping developers or vice versa?"
- Menno Loggere & Nora Visser: "Privacy kills quality"
- Anna & linda Hoff: "The Supertesters"

Besides presentations this event is always a joy to meet "old" and "new" test friends. The setup of this event was a bit different compared with the past. This year they introduced some kind of knowledge tables were exhibiting companies would be able to talk with participants about a topic they had chosen. Also new in my opinion was the set up of also smaller tables which able us to sit down with our friends.

The presentations
I arrived just an half hour before the opening started followed by the key-note given by Stuart Reid. Arriving in the large hall I noticed it was already quite crowded. During the day I heard that over 450 attendees were joining this conference together with me. Who dare to tell that testing is death? We have already a quite strong testing community in the Netherlands. :)

Together with a fellow tester I know from participation/moderation on testforum.nl I went holding a cup of well deserved hot coffee to the main hall, found a spot and sit down. The show started when the chairman opened the conference.

Improving testing - with or without standards
He introduced Stuart Reid. I never saw him a presentation of before and he started with introducing himself. Based on his 27 year experience, tutoring on a university and participating on ISTQB testing techniques if I remembered well he started explaining about how our profession as testers lacking behind in knowledge compared to users and developers. To support this he provided a nice chart with figures from a survey held in 2000. Based on these figures our skills are looking very bad. Didn’t we learn more in the last 10 years? Are these figures still valid? Why are we testers so eager to see figures even if they are from old surveys and rely on them? Or should I accept those and make my own conclusion that certifications did not add any value to the marked as during the years certification exist, we did not added any value to the market. Of course I'm am wrong again.(?)


Using figures and charts from "old" surveys can be useful. Really, there seems to be people who think using "old" figures can reveal the truth.

I believe when using certain information you have to check how dynamic the environment is.

Somehow the IT environment looks to me very dynamic and fast growing when it comes to technology, perhaps also a bit related to the learning skills. If you look at other areas they are less dynamic. Stuart Reid provided also a nice picture from a model created by Hackman & Oldham related to motivation factors. I think that image had more value for his presentation than all the other figures. Here he made his point that we should not only look at the information sources and test techniques, we should be aware of our soft skills also.

Fortunately he was a bit out of time and skipped the slides related to ISTQB etc. In my opinion that kept the message which is more worth to me: we are behind in our knowledge or getting behind and we should adapt now. We have to become professionals with skills. Some testers might believe ISTQB- or other certifications is the only true path. For them, those skipped slides might have some addition.

I hoped he provide some information and guidance so we actually learned something instead of scaring us and directing us that having certifications are a must. Perhaps I am wrong.(?)

The debate between the ethics of testing
After the previous session I attended a "new" idea using a debate to bring testers together. Perhaps people think this is a contradiction. I believe when people talk and hear each other mutual understanding is growing. You don't have to agree with the vision of the other, you have to be aware.


During this debate there were several statements mentioned and introduced by the host. We had to choose party. A pitfall could be that every one would agree or disagree. The hosts mentioned that they will divide the audience in two sides.


There were some good statements although you might agree with the statement; it can be more fun to come up with valid arguments to challenge that statement.

examples:

- "A tester should always speak the truth"
- "A tester can be hold responsible for acceptance"
- and more


I believe it is a good habit when every one agrees upon something immediately, you question if this is true. A tester should be capable to come up with arguments to question any statements.



I heard some very good pro as contra arguments. After each discussion the hosts came up with a wrap up and their own thoughts again.



I see some future for such an approach of discussing about topics. You can play this game in different ways. And learn a lot from it.

Mistery guest
After the voting if testing and ethics can go through one door together a mystery guest was introduced. They found a "known" person, Bart Broekman willing to speak about his vision related to the statements. I think a debate before a "keynote" is a good way to set the mind set. It helps avoiding discussion about irrelevant topics. It supports discussion about what a person really had to say.


Winning the TestNet 2010 debate award
Finally at the end there was a prize to win. I was the lucky one who gained that award. Unfortunately I don't have a picture of accepting the award, though below you see the evidence. Did I won because I am a good debater? I don't think I'm the best. At least I was standing there with dedication and believe. Perhaps that made me win the award. I'm proud of it. Sometimes an award tells more then other valuable meanings. I know there are some testers who fancy the bottle of wine I won together with this cup. I knew that a cup valued more when presenting it to my kids.

Super testers or Super Sisters?
The sisters Anna and Linda Hoff from Know It from Sweden gave a tremendous show presenting their vision/act about testing. (Also to be seen on EuroStar 2010: Advice: See them!) The Supertesters - A Slightly True Story"


During the show I wondered if they were actors/comedians or testers. Based on how they used the terms and images they have to be great testers.

In my opinion they were GREAT, it was actual a very good show were they acted as program manager, a tester and a super tester. They performed using several techniques in presentation, discussion, drilling, singing, rehearsing, using pictures.

If you looked and listen carefully you might have heard how they were using the several testing schools from ISTQB addictives to Bach-followers. Their approach about changing the mind set of testers by drilling and forcing that their answer is the only true answer is in my opinion a good example how we testers are currently forced to think alike.

Some funny moments were added like hiding bugs, finding it, appreciating it and comparing it with the moment from "lord of the rings" and valuing it as "my precious".
Also their understanding about testing like smoke-testing, V-Model (in their vision it is actually a very nice model), load testing and performance testing (I still wondering if the sisters used pictures out their albums)

At the end the showed some mix of lyrics on known-melodies presenting their message.

I wish more people are able to present their message in a show like this with. I believe the time is well spending watching them.

At the end

I had some time reserved to see some other presentation after the dinner, only as usual I didn't make it. I had some interesting discussions afterwards with fellow testers. This is one part I also like about this event: meeting other people. What I understood of them was that some presentations could be better. Personally I think this was a great event, learned some personally, had some laughs, got annoyed about the first speaker and went home with a good feeling.