Testframing: a bit of my perspective

Last week I attended a tutorial on testnet spring event 2011 given by Michael Bolton on Testframing. I already read something about (see: http://www.developsense.com/blog/2010/09/test-framing/ )and now had the chance to see it work. it was a great day as it made me think more about this concept. To preserve my thoughts and ideas I wrote this article. Perhaps you can learn also a bit from it.

Initially I thought it would be an approach were you frame all the tests in constructive sentences, and write them down. So initially it seems to be the same as I was told to do in the past: write test cases and then execute them.

I was wrong then: Why would you write down or use valuable time to write down test cases till the lowest detail when your mind is able to work with the speed of thinking. So there must be more to understand.

During the tutorial I noticed that one of the strength is the way you use a mission to guide the story you want to tell and the observations and actions you do how the story can be told. While explaining there was interaction between the listeners, when something was not clear you change the vocabulary of the proposition or perhaps the vocabulary itself. You even extend the frame using formal or informal connectives

I think that the difference between writing test cases and test framing it takes you much further, it combines the strength of a test strategy with thinking about the justification of testing together with(when necessary real time) adaptation of the path in the system you tending to follow ending up with the explanation of the results you found combined with the justification why you did it. This can be done very fast.

In other approaches like TMap we tend to call it in the Netherlands, there are all kind of delays involved, first the plan, then the written cases, then the execution and after a while a verdict about quality which was stated in the plan.
In some way it is a structured way of working, is it testing? This is another discussion I will not start here. There are other sources about this topic.

In short I learned more about test framing that it is a way of telling a story why and how you tested something combined with the results you found and what you think it tells about the system.
Testframing to me can be done by writing it down, in some cases this makes sense perhaps because of some complexity is involved or some kind of uncertainty about the system you have and you use it to validate your story.
Sometimes you actions are less important and easy to explain afterwards, then you use it without writing it down, just explain it.

Somehow I see a lot of potential in this way of testing. one of the reasons is that it keeps me thinking if I'm doing the right things.

On the same day Michael Bolton also had a keynote were he explained about the dark and the bright future. This was a great view. In the birghter view he explained about first order measurements (some more information can be read at stickyminds.com:
Three Kinds of Measurement and Two Ways to Use Them By Michael Bolton

While explaining first-order-measurement he used a connective "or" this triggered me to think that this is one of the strengths of testframing. You are able to adapt you story based on the first-order-measurements. My first reaction was: This is great!

If I'm able to tell a story about why I'm doing something with what mission (direction) then I'm able to tell the stakeholder/person who cares why Im worth paying for and spending time and resources.
If I'm able to make visible that I'm able to react on circumstances, the information from first-order-measurements, which are happening now instead which happens in the past (the delay when test scripts are written down and the explanation is proved later on) then I provide my self a tool to see if Im on the right way of acting and providing information which are accurate and supports transparency.

Looking back to testframing, I see some different ways to use it with respect to logging the actions:
- use it by thinking in testframing: nothing to write down or explained.
- use it only using your mind, thinking about it and explaining what you have done/doing
- use it in front and write down some notes how you think you want to approach the system
- write down while testing how you framed the mission
- pair testframing: you tell and share your thoughts/mind and another person make notes without interrupting you thinking process (this is another role than the person who challenge you while testing)

Another dimension here is the measurement-order. I think first-order-measurement is an valuable addition to be aware of while framing. Here you can do also some logging as the measurements changed or confirmed the path you were following in your mind.
- before testing and framing first: you might consider which propositions might provide valuable information
- during framing and not logging the testframe itself: log the moment of measurement and result, here you might wonder if you also write down the impact on your story.
- during testframing and also explaining the result: tell the story not only what you have found and why you did it, also explain what impact the measurement had on your frame and ask if you valued your measurements properly. If not there might be some valuable information left behind the framed story.

Somehow I believe testframing is a great way to test a system fast with the right transparency and justification. it is not just a trick you can learn, you have to practice to be able to do it and learn to become better at it. It is a way of thinking and communicating with others which is not only challenge the system as you do it, it also challenge you by explaining how you do it. Perhaps it also tells the story about the tester.

I hope this article challenge you to think a bit on Testframing, for me this is just the beginning in learning about testframing looking for opportunities and time to start with it and learning about it.

Response on Go/No-Go to ship

Questions?
Michael Bolton posted and excellent article called Blog: When Testers Are Asked For A Ship/No-Ship Opinion which made me think and respond about it. I started with commenting on his blog and during that I came up with some thoughts I wanted to share.

Reading this story raised some questions for me I normally are aware of only never asked directly. Perhaps because when dealing myself with it; it is too close to me; the project is in stress. I agree with you that we should not make the decision shipping. Here some questions I have as response to the project manager whether to ship/or not:
- Where did we miss providing enough information? If we provided the proper information she would be more conformable.
- Why did she ask that question at the end of the project and not during the project?
- Why did not we guide her to ask “valid” questions?
- What could we do better to avoid discussions and questions like this at the end?

Do you notice that these are questions to myself instead directly to the project manager. if you have to change, first think what you can do. What value you can deliver. And also when. Looking at these questions, there is more then just providing test results. You have to communicate on other items also. In this case, which message will you have to bring and do you have mutual understanding on this.

I’m sure there are other questions to ask, even more answers to be provided. In my opinion you posted here a basic rule. When thinking further on this, based on this question to ask or not to ask, testers have to deliver all kinds of documents/ metrics and so on, just to “help” the project manager making decisions.

The bright and dark side
The bright side is not having all information and asking the team. Only the moment is “too” late when you get this question at the end. It is a bright situation since you are not exaggerating the documents you deliver and you have time to adapt to the situation. You must check continually if you provide value.

I believe there is another dark side. The dark side is asking the team “all kinds of information not knowing yet it will be valuable or usable and still I need it just in case I come up with questions afterwards forgetting that providing information cost time and resources not delivering other valuable products”.

What I have seen in the past was to gain control by collecting all possible information. Sometimes collecting information is not that bad. It becomes bad when you communicate about it and no one is waiting for it and you have to explain they should.

Awareness
If you have to explain the value of information afterwards, then you are too late. You have to guide them and help them to understand the information you create/ provide. You also are responsible only to deliver that information which is needed/values to the product directly or indirectly. This means you have to communicate and interpret the behaviour of the stakeholder.

To me, testing is more then only finding issues, or proving functionality works. It is also a process to make the results you find be accepted. You must be aware that you have to deliver that information which is requested and make sure that vision about responsibility is agreed upon. Perhaps keep checking if the information you provided is valuable and also understood as you meant it to be understood.

Go/No-go?
Are you the messenger for the GO/No-Go advice? I believe you are not the decision maker on this. You should provide information the decision maker can make that decision. Sure, you should help him/her. Only by providing information within the proper context. You also have to explain and guide how that information can and should be used.

If a question like this is coming from the project manager, you might see that as a sign you did not provided the right information and or guided her/him through the information you provided. Instead of asking question to the project manager, first start asking them to yourself.