EWT17: Rocket science in software testing

Introduction
This weekend I attended another session of European Weekend Testers. This session was facilitated by Thomas Ponnet and had another approach then in the past. This time we could prepare ourselves a bit. The tool under test was offered before the session started.

What has this to do with rocket science? It was the plug-in which had to be tested.

The Participants were:
Shruti Gudi,
Jeroen Rosink,
Tony Bruce,
Zeger van Hese,
Catalin Anastasoaie,
Katya Kemeneva,
Dominique Comte,
Pradeep Soundararajan,
Jaswinder Kaur Nagi,
Thomas Ponnet,
Anna Baik,
Markus Gärtner

I have to admit it was a great crowd, a great session and an useful round-up.

What to learn
Last week Andreas Prins posted on his blog the question what can be learned: Attitude or methods? with weekendtesting. In his posting I wonders when reading articles related to weekendtesting he never see a reference like: "As ISTQB Chapter X page xxx we must do this or that" it is a good remark from him. Mainly in projects I don't refer to pages of ISTQB either. Not in this context. When testing in the weekend, or on a project, you refer towards your experience or even sources were people are telling about their experience in a certain context. That experience can be based on theory combined with common since and the situation.

What can you learn in Weekend Testing. I'm not able to tell you what you will learn. You might learn how to look at yourselves. You might learn to think beyond the borders of the regular testing projects you are into. You might learn from the approaches from others. You might learn how to learn.

The mission
The missing this week was different then others, this time we had a manager of a band who had a gig that evening and wanted to make sure that the plug-in he found was suitable and stable. If not, would we be able to propose alternatives.

The approach
Basically I looked at the application to the following points
- try to play wav while plug-in is not available
- try to play wav after plug-in is selected
- tried to alter the sound of the wav using several preset schemes.
- asked the manager about when it is stable, plug-in /laptop
- asked the manager about under which condition plug-in was used
- played with multiple files,
- used other wavs
- used wav and midi together,, no option to mix tunes
- used the key board and options while music was playing, it interferes with the output.
- the midi/wav player, played a bit with that.
- looked at the minihost and used several schemes/presets
- used the buttons on the minihost
- tried to work together with multiple minihosts
- try to record music
- tried using strange actions like using short-keys how app. reacted

Some issues
Below you find the highlight of issues I found during the session. These were findings from my side.
Error01: message shown when opening the minihost
Sometimes when using the minihost this error is shown. Not always reproducible
Error02: error shown when opening recorded wav
when opening the “test.wav file just recorded this message is shown, although the wav file created/recorded using mic is 1kb.
Error3: opening another own wav file error is shown, wav not played
When opening another .wav file format the following message is shown and wav is not played.
Err0r04: Recording not working
When using the recorder it shows that a number of kb is created. Even the file and location is shown correct only the actual recording is not made
Error05: when new file is played, not selected
When a song is ended and a new is played in the list, the selection is not made, the original keeps “blue”
Error 06: in global settings window: “tempo” is not working
When using the Tempo slider, no effect on wav-output
Error07 multiple files able to select, last file is played
Error08 buttons not fine approachable, usability is less
When trying to turn on the buttons it is no following the direction of the mouse
Error09when playing a song, and hitting button on midi/wav recorder interferes with output
When playing a tune and you press other buttons of this app, then music/tune is stopped/ hanging for a few moments.
Error10 pressing The F3 button while playing a wav makes the music hang,
When pressing the F3 button on minihost.exe while playing application is hanging. No other interaction with system possible

Some Lessons Learned
Of course there are a lot of things you can learn when testing. There are even more things you already have learned. Some of the lessons I learned this weekend are just refreshing or confirmations of other valuable lessons.
1. Although information about a single application is required, when testing it together it is a combined answer. You might consider it as single object, when it is tested together with other tools you have to consider their stability also
2. To understand or able to test a part of a object, you need to know the context, in this case about what stability means for the user and not according to the tester.
3. You are not in the position to provide advice, look at the article from Michael Bolton: http://www.developsense.com/blog/2010/05/when-testers-are-asked-for-a-shipno-ship-opinion/ You can provide information
4. When you are asked as a team, you have to work as a team. Even after short introduction it is hard to get everyone’s attention.
5. Domain knowledge is a prerequisite when it is directly asked by “ the manager”
6. if the manager is not there, find someone in the team with domain knowledge
7. Don’t get distracted by crashes, when they are reproducible, then you can avoid them, if you are still able to use the functionality then, you might earn something with your gig
8. It is easy to forget the lessons learned from previous sessions, The assumption is easily made that every one knows you and how you think. Information which seems to be obvious is often forgotten when acting longer in a project. Perhaps recap some questions? Magic words: FOCUS/DEFOCUS
9. Reminded about the posting of Markus about being blunt or not towards manager: http://blog.shino.de/2010/04/11/testing-and-management-mistakes-causes/


The discussion part
During the session several questions and suggestions were raised. Information was missing or needed. Some were about the domain knowledge like "what software compressor for music is", "How to communicate with the manager", "acting like a team or not" and so on (you might check the transcript for details)

Also at the end some valuable remarks were made related to "old experience", "skype is not a good tool to use ", "the manager already checked for a tool, why should we check for more functionality", "if the plug-in was the objective, should it be tested alone?", "Are we able to answer the question to provide and advice?"

Looking as a process to the discussion you can also notice some familiar behaviour. We all had a common goal, still we acted like individuals. We try to get information which would be valuable for us at that moment. In my opinion we did not asked what would be valuable for the team. We also tried to do our job good due to the minimum of time and focussed therefore more on ourselves. When you look carefully, there were some persons who tried to become a group and act like a group. Perhaps due to time, differences in experience, differences in testing approach, differences in objectives we did not succeed to act like a team. If you look at the end, we are more explaining what we have done and what the traps are. The focus lied more on "did we succeed the mission". I believe we missed in some part a good lesson: "what did we learn and was it fun?" and also "Which personal lessons can you take to a next session."

What would be more valuable, to meet the mission as an individual or to act as a team, learn from each other and perhaps meet the missions objectives or perhaps change it during and afterwards?
Conclusion
This weekend session was a great one. A mission with an attitude of the manager. A great crowd of testers, a discussion you can learn from. I had a lot of fun and learned old and new lessons.

Testing on the other side

How often do you see that others believe it is not their failure, it are the others? How often are you sure it is not because of you, it is because of them? How often do you see that you did everything to prevent failure and others are to blame? How often do we all blame the other side?

Perhaps not that often. Still if it is not happening at our place, then it must be triggered at the other side. If on the other side failures are triggered, created and on our side solved/prevented. Why are we spending time at our side to proof it works here?
Currently we seem to focus to measure that everything is OK on our side. We are not to blame when the failure hits the fan. So it must be elsewhere to be found. The other side!

If we are sure that it is OK on our side, and failure is on the other side, why aren't we adapting our strategy of testing and look on the other side? Sot the approach would be: create tests what we should do. This will proof we are right. Now reverse the test scripts and test other things as we reversed our thoughts to blame others, then they must be blamed by things we didn't found. That leaves us the parts we are not testing, the parts from the other side.

The thought is acting opposite. If right is OK, and wrong should be found, we believe everything is right on our side; then the other side is wrong. We should find the failures on the other side. Finding failures is NOT blaming. Finding failures is in this case helping out the others. This means working with them on the other side. This means joining for example development, become a team.

To be able to help we should force us to think differently. This also means that we should force us to change our minds to create space for other thoughts and ideas. Perhaps adding a new role to testing: “the devil’s advocate-tester”. He/she will try to force the team think in a different way; although controlled. This role might change every week just to force us think beyond boundaries.

This might result in an approach were space is created to think differently and to extend borders within a testing project.

So prepare to test also on the other side, just for fun!?

Free pie for everyone or just an issue?

My son has his birthday very soon so we tried to order some cakes and pies online. We loved the cakes and pies from that certain store and never had problems. now it seems to become a very cheap birthday. While placing the order it seems that nothing was charged.
It was tempting to add more items to the chart, though we didn't do it. The first screen was already an issue. Items were selected and nothing charged. (This is a Dutch site so here some translations so you all can learn some Dutch from this site also
aantal = number
personen = persons
per stuk = per piece
prijs = price
totaal = total
)
The first issue see below: I assume that since it is counting the quantity of persons and persons are not charged, the total amount to pay is also zero.

The image below represents the confirmation screen and were you can check for your details. The title of this window is: "Controle van de gegevens" which is similar to "Check of the information" Here you see that it counting the number of items and presenting the price of the previous screen. Still nothing is charged.


The last screen in the confirmation screen. O how much did we hope we would see the zero here again. Unfortunately we got surprised we had something to pay. Somehow it seems that here something is working well, only we were not able to check if the price was right.

Remark: At the first image you see a certification called "thuiswinkel waarborg" this seems a certification to provide the consumer trust and liability for there online orders.

Lesson learned: when something seems to be free, don't order too much as you have to pay after all.

Another lessons could be: certification doesn't say anything about the functionality. (Imagine how this statement can be transformed to other types of certification )

Test Automation by the book

Previous title was: "How TMap Next is used for automated testing" changed it into a proper title :)

Automate testing is hot. It has been hot since I work in the software testing business. It will be hot for a long time now. Often I hear arguments about automated test tools are shelf ware.

Sometimes I hear automated testing by the book is not possible. I might have formed an opinion that the Test Approach: TMap Next by Sogeti fails when it concerns automated testing.

When I think and talk about automated testing I think how to use tools and not how beautiful tools are. Sometimes you find another usage of items and methods. Recently I found someone using the Test method as written to automate his testing.

As you see, the book contains the method and it is used to automate his testing. Therefore the method is used for automated testing. In this example it was used to support the LSMW (Legacy System Migration Workbench) in SAP. An LSMW is a tool that supports the one-time or periodic transfer of data from a variety of sources without any programming. Only in this example it had to ran multiple times and therefore user interaction was needed to press the Enter button. As you see. The book/method is very useful to support in automated testing.

As you see the book is lying optimal on the "Enter-key". You might consider this as a steady firm tool. As some testers which are familiar with the "old"Tmap approach, we have been told to use it as a tool not as a goal. I think this is a good example to use a method as a tool.

To order this book as a tool you might take a look at: TMap next

Fun: Black Box Test Machines

Recently I found this site were some example programs are made available to
Black Box Test Machines by Workroom productions from James Lyndsay.

Quoted from his site: "The following freeware machines are testing puzzles - crosswords for testers. Open one up and play with it. Try to answer these three questions: What's it doing? What's it doing wrong? How can I be so sure?"

When you have some spare time you might take a look at those applications and try to answer the questions. It is also worthwhile to read his award winning paper: Adventures in Session-based Testing"

Quotes to think about

Recently, driving in my car to my assignement I heard a commercial on the radio where they mentioned a quote from John Maynard Keynes (English economist, journalist, and financier, 1883-1946): "Ideas shape the course of history!"

I loved that quote as I'm full of ideas. Perhaps I can shape the course some day :)

This triggered me to search for a few other quotes:

G. Moore: "Everybody sets out to do something, and everybody does something, but no one does what he sets out to do."

B. Shaw (Caesar & Cleopatra (1898): "When a stupid man is doing something he is ashamed of, he always declares that it is his duty."

J. Dryden (All for love (1976): "Errors, like straws, upon the surface flow; he would search for pearls must dive below."

A. Tennyson: "Sweet is it have done the thing one ought."

John Updike (The Carp (1978): "An expert takes nothing personally. Nothing is even precisely his fault. If a bridge collapses or a war miscarries, he has already walked away. He still has his expertise."

Testing Games

This morning started good. I posted an article about a possible relation between Rooms and Pillar Mining and Software testing. I found some time to play a game on the Wii and now having a short break. With my thoughts partly on testing and partly on playing games, this though brought me to Google on games related to software testing. Here some I found:

Bug Fix Bingo by K. J. Ross & Associates
Planning poker by Mountain Goat Software
Non-approved test methods by Charles K. Kincaid

Perhaps you can extend that list of test methods.
While writing this blog I hoped I would find more examples. Unfortunately, I didn't. So here some thoughts of mine to use when you want to do something with your time only there is nothing left: Held a competition with your colleagues to check:
1. who is able to find a bug in already tested software;
2. who is able to perform as fastest a number of test cases from beginning till end
3. who found the most issues and present it in a fancy chart
4. who is able to construct a data set in such a situation that your colleague must use functionality to correct the data and be able to finish the test case.
5. Make funny photo's of your colleagues impersonate test methods

Link: Testing challenges by Matt Heusser

On a previous post I already mentioned about some good questions asked by Michael Bolton on Matt Heussers weblog: Creative Chaos.

Matthew create a testing challenge which is worth to pay attention to.
For more info see:
A new testing challenge
New testing challenge - II
New testing challenge - III
New testing challenge - IV
New testing challenge - V
Perhaps you can put some though in the basket?

A strategy game: Civilization 2

One of my favorite games to play is Civilization II. And I think it has all to do with software testing. Here a very short explanation to relate this game to software testing.

This game is a strategy game, were you have to build a civilization where you can choose for 3 goals to win the game:
1. Be the first civilization who builds and lounge a spaceship as first;
2. Eliminate all other civilizations;
3. Keep playing until the year 2020.

The reasons why I think it is like software testing are:
1. before starting you have to determine how the world have to look like. Should there be a large landmass (longer time to play) or a smaller area (shorter time to play);
2. how many opponents you have to beat. If you choose for the maximum there are other circumstances you have to take care of in combination of you strategy to win;
3. were do you focus initially on: building lesser cities with all improvements of larger armies of lots of cities with lesser improvements. This decision will influence the rest of the game;
4. do you "cheat" or just continue playing when something did not went as you hoped for?;
5. when "cheating" which outcome do you accept at that moment.

Ad1: translated to testing: do you choose for time boxing or do you test until you met all acceptance criteria.
Ad2: If a testing team is larger you have to communicate more with the team and make decisions every one satisfies. The lesser opponents, the easier it might be to satisfy them.
Ad3: An optimal designed city might be translated to an optimal defined test process. Only this might prevent you to become the first civilization to build and lounge a spaceship. It is very good to make the year 2020. Focusing on a lesser defined process by building as much as armies (perhaps translated to test scripts) might help you conquer the world and become the strongest.
Ad4: cheating can be done by saving the game. And when you get into battle restore the game until you get the desired result. Or when entering a "hidden" village, accept the newly gained knowledge, or perhaps you found another city or a certain amount of money. This can be compared with regression testing: keep retesting until the expected or acceptable result is there. And not continuing with a loss of armies or accepting the horde of barbarians you released.
Ad5: Based on the situation and moment of the game you restore your saved game and accept in the beginning the newly gained knowledge which enables you to build other improvements. Or perhaps accept the found treasure which enables you to speed up the building of an army so you are able to beat the opponent the next round.

Beside these examples this game triggers you to evaluate every situation and create an open and creative mind. When playing this game more often you might be able to select different levels of playing this game. Start on the easiest level when playing first and see if you are able to define you strategy when playing on the hardest level. Like in software testing, initially you might be able to coordinate a simple project. Keep learning from those projects, encrease your skills and see if you are capable to do more difficult projects.

Tetris as Test Management tool

I just had a crazy idea. What if we use the famous game Tetris as Test Management tool instead of other plannings tools like Microsoft Project?

Wikipedia has some information about this game: Tetris

On this site you have some key-words which are similar to testing:
Gameplay: "The object of the game is to manipulate these tetrominoes, by moving each one sideways and rotating it by 90 degree units, with the aim of creating a horizontal line of blocks without gaps."

In testing we also trying to plan every activity without leaving gaps, so testers are not sitting still.

Variations: "It is difficult to place a standard on the game, as newer releases frequently progress it either to make the game better or to keep players interested."

As in testing: there are several standards, schools, methods and approaches.

Tetris variants: "A number of Tetris variants exist. Some feature alternate rules and pieces, and others have completely different gameplay."

Like said to variations: Some methods have similar approaches and some are approaching the test process completely different.

Is it possible to play forever?: "The conclusion reached was that a player is inevitably doomed to lose"

We are not allowed to test forever. There is always some kind of project management who is speeding up the time and makes us stop.

How can we use Tetris as management tool?

1. Pick the proper version of Tetris which suites your needs;
2. Identify the types of blocks of your Tetris game;
3. Name those types in testing terms like: Plan & Control, Preparation, Specification, Execution, Regression testing, and so on;
4. Define your rules: perhaps: every complete line is a successful planned iteration; or another variant could be: every level is one iteration.
5. Make screenshots after a predefined number of blocks, this can be your test process planning.

I know there are a lot of reasons why not to use it. Only what could be the reasons to use it instead? And what are the risks if we use it?