A day at Test Automation Day 2011

On June 23rd 2011 I was in the position to attend the Test Automation Day 2011 as an advantage since Squerist, the company I work for, was the founding partner of this event. In my profession I started with automated testing although I like to think more how test processes can contribute value to an organization I still have a weak spot for automated testing. This made me curious about what to expect.

I attended in the afternoon due to other obligations. This leaves me to the experiences of other people. I entered the exhibition area just before lunch and noticed the crowd entering the room. Normally I recognize some people from past conferences, now I saw new faces with one thing similar: passion for their profession.
I talked with a few of them and they heard new interesting stuff how certain tools might fit into their business. They got already some value back for attending. Others heard more or less the same story. Isn't it always this way.

I attended 3 keynotes: From Martin Gijssen (De kracht van open source testtools), to Mark Fewster (Experience Driven Test Automation) and Scott Barber
(Automating Performance Tests: Tips to Maximize Value and Minimize Effort)

Somehow I missed another presentation in the middle. instead I had a great talk with others.

What I remembered about the presentation was that Martin was able to speak about open source tools without making a judgment what we should do. In generally it depends, sometimes open source tools are good to use, sometimes not. Personally I expected more information how, when and the pitfalls instead focusing on a framework. Though some introduction was needed and if time is valuable you cannot tell the whole story. What left was the importance of a framework, a test architecture and key-word driven testing is still kept alive.

About Mark Fewster: I like this guy, If you sit far behind in the room and you don't have the full picture he has some similarity with David Letterman only in his best days. (hope he is not offended by this. I enjoyed listening and watching to him) With passion and wit he stood there convincing about the experiences he collected about test automation. Based on this he shared us his future. One of the things in his future was the maturity of tools supporting test techniques. I think he might be right, there will be coming more tools than ever.
I think another road in the future is the need for focus of tools on the human aspects.

So I think we have here 2 paths: 1 path of tools which will support more techniques and move towards the developer to build the scripts for testing and other tools which will support the "keyusers" to tell in what they want to do or did with testing.
What the future will bring, we will never know. At least he triggered me to look forward to his book in which he collected the experiences of others.

The last keynote was provided by Scott Barber, I never saw him before. Wow, he is an artist, how he sells his story. He got me all the way. Although it was mainly focused on performance testing, he sold me the idea that it is quite different then functional testing. More complex. And perhaps equally fun :-)
During the time he gave us 10 tips for free to focus on while setting up your performance test. I think they also are generally usable for all the test we perform

Top 10 performance test tips by Scott Barber on Twitter
10 data design
9 variance
8 object orientation
7 Iteration/agile
6 Error detection
5 Human validation
4 Model production
3 Reverse validation
2 Tool driven design
1 value first

For details about his tips, just contact him. I'm still looking forward for his presentation as the information was to much to write down. And he has some story to tell. (Afterwards I spoke with him and he is willing to tell you about his experiences)

Some of the eye-openers/lessons I remembered:
1. Performance testing is for developers, let them help you,
2. provide information about performance on continue basis, not only as a big bang
3. Think about the release management, test management and infrastructure before you start measuring performance
4. Also in performance testing, numbers doesn't say much. Perhaps start presenting you measurements with the happy faces and sad faces.
5. Although I was not directly involved with performance testing, in the past indirectly I learned some items and were involved with some performance issues too.


I think this conference was a good one, for a first time I think they can be proud of it. A lot of information was shared, good presentations (and some less) were given. They missed the short break after the 2 keynotes and between the "business cases". Now there was less room to share experiences of the presentations. As all conferences, you hear some old wine in new bags or what is it called. I strongly believe in the skill to learn from the information you get and read between the lines.
I had some good fun that day and was surprised. It triggered my thoughts and extended some perceptions/views and ideas.

Seminar: Are you innovating

Although this is not a correct translation it might make things more clear then call it "Innoveer jij mee!"

Today I attended a free seminar sponsored by Ordina with to great speakers: Gojko Adzic @gojkoadzic en Anko Tijman @Agiletesternl

The evening started with a long row to obtain "bad" coffee. With this start everything can only improve. And it did. Sure, there are always some things which are worse or could be better. Let me start with some small notes about the impression I gained. It is not the full story, just some snap shots and ideas.

Gojko's "Sleeping with the enemy".
As the chairmen mentioned, "Oh, no, not that again" It was not about testers and developers. Us and them. Hearing Gojko mentioned these words I immediately thought about the song from Pink Floyd Us and Them

Looking to the lyrics it can be so true. It also start with:
"Us and Them
And after all we're only ordinary men"

Gojko did not emphasize the idea of Us and Them. What I remembered was that he want to skip the functions. In teams we don't need functions, we perhaps need roles. We should get rid of our who's to blame culture.

He started with a great statement "Why are we[programmers] going to stop telling testers how to do their job? They know testing much better than we do!"
There is still some discussion possible if developers should test or testers should test. I think we should do both. I will come back to this later on.

In my opinion he used a very strong example. (Keep in mind that these are some pieces I remember from this evening. Therefore these are my interpretation and words).
A driving instructor is not telling you: you hit that car, and that car and that pedestrian. A driving instructor is teaching you how to drive.

A similar thing we testers should also do. Teach the developers how to test, participate. If they are able to do the testing for you. You gained time to do more important things.

Meanwhile @michelkraaij tweeted a statement @gojkoadzic, you keep making the distinction between developers and testers. Testers ARE developers!

We discussed this in the short break and Gojko marked that within a role there is also persons are involved. I translated that into a thought that personality is also important how a person can interact within his role.

So I polluted the twitter sphere with this idea as response:
ppl matters,personal value creates personality help define skills 2 form attitude able to act in a role

My idea with this statement is that it is easy to think that every one should be able to test or develop or do the same. Skills should be equal. Who does is doesn't matter. If people are involved we should respect them and human beings with a certain value. People differ and therefore have different personalities. I think the personality able a person to use their values under certain circumstances which will express their skills under those conditions. When it is noticed, people will respond on it and this will influence their attitude. And now based on the attitude, the presentation of the role which is needed will be express also. (don't shoot me for these thought, it are just ideas which seems reasonable to me, I did not read a book to support these ideas)

As promised, based on this concept it should not either be developers or testers who do the testing. Testing should be done by those who care about testing. If a developer cares about testing then let him test.

This was in other words also Gojko thought; he explained that when testers were involved quality reduced as they rely on the tester and no longer on their own skills/value (testing).

After all Gojko held a great presentation with hand drawn slides :) With perhaps a credo: "let developers help us to test so the testers can do other important things."

I think this can also be expressed by the words from Pink Floyd from the song mentioned earlier "And who'll deny that's what the fightings all about
Get out of the way, it's a busy day"

Agility in the waterfall?

After dinner Anko started the presentation he also held at Eurostar 2011 called "Building a Quality Driven Team". After some introduction he came up with some slides I initially thought he wants to bring us back into the waterfall. I can come up with argument I reject his story. I think he showed me some other ideas concepts. He made me think. Although his slides seemed to look like waterfall explanations. If I turn that concept around: He made me able to bring put arguments to the traditional project manager who relies on requirement, design build etc till it is good and supported by waterfalls ect.

The model Anko made for me was a model I could use to explain how certain phases of Agile testing fits in a V-model approach. If a PM fancies a V-model, why not use it to explain your thought. It can help to express that your ideas about Agile attitude and quality firs in his project also, only there are some other conditions. You might be able to build a bridge between different thoughts and gain some understanding. This might be a solid base for obtaining commitment.

Anko posed another statement about the emotional acceptance factor. I would translate that into an example I learned a few years ago.

I was on a project with the ideas of Agile and with a lot of commitment from business, team and project management. We tested together and did a great job. As well as developers and testers were able to tell on daily basis what they did and how the project was evolving and value was introduced. We were almost finished and one of the stakeholders complimented us with the result and made one final addition. I see you did a great job, I see that the quality is somehow made visible. The only thing I would ask you to do: Coach and support one of my key-users, make him test in his uncoventional way. if you are able to help him gain trust it would help me to sell it to the business.

This was not about not trusting our results. This was about some feeling based on emotions. This was emotional acceptance.

Hearing those words, I start remembering more of those examples. It is not explicitly written in books. It is there. And we should care.

There are more things I have remembered and should remember. After all, it was a valuable spend evening. Of such value I want to share it with you.

At the end I had some great conversations with other people and thanked Anko and Gojko.

While driving home I forgot to ask him if he had a copy of his book available and ready to sign, perhaps another time (@gojkoadzic you have my card :-) )

Testnet Voorjaarsevent 2010

It happened on may 12th 2010
Last week I attended TestNet voorjaarsevent. Testnet is the Dutch Association for Software Testing. Besides several workshops and small events there are two major events, in the spring (voorjaarsevent) and in the fall (najaarsevent).

Last week the so called "Voorjaarsevent 2010" happened. As usual I got a bit prepared, made the selection of presentations I would like to attend and also which not. What I see often is that the presentations are dealing with old concepts in new clothes. This time there were a few which got my attention.

General impression
* Location: very good, as well as the main entrance, the route to get there as the overall presentation
* Dinner: Good, it was tasty and enough.
* Drinks: like to see the ability also to have drinks like coffee or tea while not attending a presentation
* Exhibitions: Well arranged of the main entrance, good to see those familiar names again and again.
* Presentations: There were a few I would like to see, as what I saw besides the first one, good. Still a lot of old work in new clothes
* Key-notes: Expected more fro the first key-note related to the main topic: "Secure testing"
* People: open minded and pleasant

Program (copied from TestNet.org)



Intentions to see:
- Key note from Stuart Reid: "Improving testing - with or without standards"
- a "debate related to ethics and testing" lead by Nathalie Roosenboom de Vries- van Delft & Budimir Hrnjak
- Rudi Niemeijer: "Safety helmet prohibited" (unfortunately this was parallel with the debate and I had to miss this one)
- Jurian van de Laar: "Testers helping developers or vice versa?"
- Menno Loggere & Nora Visser: "Privacy kills quality"
- Anna & linda Hoff: "The Supertesters"

Besides presentations this event is always a joy to meet "old" and "new" test friends. The setup of this event was a bit different compared with the past. This year they introduced some kind of knowledge tables were exhibiting companies would be able to talk with participants about a topic they had chosen. Also new in my opinion was the set up of also smaller tables which able us to sit down with our friends.

The presentations
I arrived just an half hour before the opening started followed by the key-note given by Stuart Reid. Arriving in the large hall I noticed it was already quite crowded. During the day I heard that over 450 attendees were joining this conference together with me. Who dare to tell that testing is death? We have already a quite strong testing community in the Netherlands. :)

Together with a fellow tester I know from participation/moderation on testforum.nl I went holding a cup of well deserved hot coffee to the main hall, found a spot and sit down. The show started when the chairman opened the conference.

Improving testing - with or without standards
He introduced Stuart Reid. I never saw him a presentation of before and he started with introducing himself. Based on his 27 year experience, tutoring on a university and participating on ISTQB testing techniques if I remembered well he started explaining about how our profession as testers lacking behind in knowledge compared to users and developers. To support this he provided a nice chart with figures from a survey held in 2000. Based on these figures our skills are looking very bad. Didn’t we learn more in the last 10 years? Are these figures still valid? Why are we testers so eager to see figures even if they are from old surveys and rely on them? Or should I accept those and make my own conclusion that certifications did not add any value to the marked as during the years certification exist, we did not added any value to the market. Of course I'm am wrong again.(?)


Using figures and charts from "old" surveys can be useful. Really, there seems to be people who think using "old" figures can reveal the truth.

I believe when using certain information you have to check how dynamic the environment is.

Somehow the IT environment looks to me very dynamic and fast growing when it comes to technology, perhaps also a bit related to the learning skills. If you look at other areas they are less dynamic. Stuart Reid provided also a nice picture from a model created by Hackman & Oldham related to motivation factors. I think that image had more value for his presentation than all the other figures. Here he made his point that we should not only look at the information sources and test techniques, we should be aware of our soft skills also.

Fortunately he was a bit out of time and skipped the slides related to ISTQB etc. In my opinion that kept the message which is more worth to me: we are behind in our knowledge or getting behind and we should adapt now. We have to become professionals with skills. Some testers might believe ISTQB- or other certifications is the only true path. For them, those skipped slides might have some addition.

I hoped he provide some information and guidance so we actually learned something instead of scaring us and directing us that having certifications are a must. Perhaps I am wrong.(?)

The debate between the ethics of testing
After the previous session I attended a "new" idea using a debate to bring testers together. Perhaps people think this is a contradiction. I believe when people talk and hear each other mutual understanding is growing. You don't have to agree with the vision of the other, you have to be aware.


During this debate there were several statements mentioned and introduced by the host. We had to choose party. A pitfall could be that every one would agree or disagree. The hosts mentioned that they will divide the audience in two sides.


There were some good statements although you might agree with the statement; it can be more fun to come up with valid arguments to challenge that statement.

examples:

- "A tester should always speak the truth"
- "A tester can be hold responsible for acceptance"
- and more


I believe it is a good habit when every one agrees upon something immediately, you question if this is true. A tester should be capable to come up with arguments to question any statements.



I heard some very good pro as contra arguments. After each discussion the hosts came up with a wrap up and their own thoughts again.



I see some future for such an approach of discussing about topics. You can play this game in different ways. And learn a lot from it.

Mistery guest
After the voting if testing and ethics can go through one door together a mystery guest was introduced. They found a "known" person, Bart Broekman willing to speak about his vision related to the statements. I think a debate before a "keynote" is a good way to set the mind set. It helps avoiding discussion about irrelevant topics. It supports discussion about what a person really had to say.


Winning the TestNet 2010 debate award
Finally at the end there was a prize to win. I was the lucky one who gained that award. Unfortunately I don't have a picture of accepting the award, though below you see the evidence. Did I won because I am a good debater? I don't think I'm the best. At least I was standing there with dedication and believe. Perhaps that made me win the award. I'm proud of it. Sometimes an award tells more then other valuable meanings. I know there are some testers who fancy the bottle of wine I won together with this cup. I knew that a cup valued more when presenting it to my kids.

Super testers or Super Sisters?
The sisters Anna and Linda Hoff from Know It from Sweden gave a tremendous show presenting their vision/act about testing. (Also to be seen on EuroStar 2010: Advice: See them!) The Supertesters - A Slightly True Story"


During the show I wondered if they were actors/comedians or testers. Based on how they used the terms and images they have to be great testers.

In my opinion they were GREAT, it was actual a very good show were they acted as program manager, a tester and a super tester. They performed using several techniques in presentation, discussion, drilling, singing, rehearsing, using pictures.

If you looked and listen carefully you might have heard how they were using the several testing schools from ISTQB addictives to Bach-followers. Their approach about changing the mind set of testers by drilling and forcing that their answer is the only true answer is in my opinion a good example how we testers are currently forced to think alike.

Some funny moments were added like hiding bugs, finding it, appreciating it and comparing it with the moment from "lord of the rings" and valuing it as "my precious".
Also their understanding about testing like smoke-testing, V-Model (in their vision it is actually a very nice model), load testing and performance testing (I still wondering if the sisters used pictures out their albums)

At the end the showed some mix of lyrics on known-melodies presenting their message.

I wish more people are able to present their message in a show like this with. I believe the time is well spending watching them.

At the end

I had some time reserved to see some other presentation after the dinner, only as usual I didn't make it. I had some interesting discussions afterwards with fellow testers. This is one part I also like about this event: meeting other people. What I understood of them was that some presentations could be better. Personally I think this was a great event, learned some personally, had some laughs, got annoyed about the first speaker and went home with a good feeling.

Yesterday, I met some of my heroes

On November 12th 2008 I attended the EuroStar 2008 conference.

Of course I went to visit some presentations, gain some information by those stands. Also important was to see other or former colleagues.

As some or perhaps most know the Netherlands is a small country. When it is rainy, which is currently in this time of year, traffic jams are things you can count on. I did the same. Only I should have counted further then just an half an hour.

I calculated 1/2 hour additional time through traffic jams, 5 minutes registration, 15 minutes for a coffee break including showing my face to the stand of my employer Squerist and then straight ahead to the first keynote.

Only I needed all this calculated time to arrive at the location. And on the quest for some coffee I came in the room where the keynote was already started.

The keynote given by Randall Rice went about Trends That May Shape Software Testing which gave me some ideas to think about how the future can be related to SOA, Energy etc.

Almost at the end (I skipped the outsourcing part) my need for coffee beat the interest for outsourcing. Only the coffee bar was closed so I had to get some from the restaurant. (I expected that coffee is for free the whole day, what was I wrong)

Drinking my coffee within 3 minutes I walked to the Stand of Squerist. On the way to it I saw one familiar face. Egbert Bouman, author of a book related to software testing: SmarTest. I this book because it presents another approach on software testing were Business is taken a central place and compares their approach with TMap, ISTQB, TestFrame and TestGoal by not telling they are right or wrong. He explains how they do it. He was the first hero I met that day. Why is he one of my heroes: He has something to say and is willing to share it with others, listening to you and also keep recognizing you after you have talked to him.

Arriving atthe stand I saw there some of my colleagues, when they read this, they will also know that I collected them as my personal heroes as they are giving me the chance to become what I like to be :) (I'm thinking this is getting a bit slobbery) So I ran to the presentation about model based testing presented by Elise Greveraars: Tester Needed? No Thanks, We Use MBT!

It was quite a good presentation, only I have some doubts/questions now on Model Based Testing. Why do we need another new role for a tester? Why is Model Based Testing an answer if we can't use models which are there to create code from because if they containing errors we are creating those errors also in our test cases? Why do we always need tools? Why not start thinking first and keep it simple so every one can understand it.

After this presentation there was a short break. I went again to the dungeons of the building. Yes, literally to the dungeons as there was the exhibitor section. And there can be fun in the dungeons, as there were some former colleagues I worked with and some more new people to talk to. Even a long lost former colleague who went to Finland to find his luck. (Rolf, it was good to see you again!)

I had to hurry to be on time for a workshop given by Michael Bolton: Heuristics: Solving Problems Rapidly
He is good!!!! As he writes on his blog about the "Heuristics Art Show, EuroSTAR 2008" It was a show. And I'm glad I didn't miss that show. Michael has the gift to make the crowd feel happy by given examples and by his enthusiasm. He has to gift to exploit the happy feeling to interaction and understanding. At least he provided me with some guiding directions to think, read, and discuss more about heuristics. Michael: thank you for this show. At that point you already were ahead for becoming one of my heroes.

I intended to have a meeting during lunch with him. Somehow I wasn't the only one in the building. So I didn't skip lunch and eat my sandwich alone :)
During lunch I spoke shortly with him only he wanted to attend another presentation with the promise we would meet eachother lateron. I forgot that I planned to go to the presentation from Graham Freeburn: Make Your Testing Smarter - Know Your Context! And also forgot to go.

So I spend my time to meet other people and also a person I met also earlier this year, Derk-Jan De Grood. He is author of a book called TestGoal. This is another great book which is also available in English. It is not offering a new method or technique, it provides guidelines how testers can give contribution to their job and a framework to structure it. (I couldn’t resist to looking at the demo and will mail you about my findings)

After a while he came down the stairs, entering the dungeon together with Graham Freeburn and another person I forgot about his name. Michael and also Graham took the time to explain the meaning of certification. Why testing the tester is more useful. How to ask correct questions. Giving us a context to think in.
Graham, thanks for that hour. (or was it more). To you, Mr. Michael Bolton, thanks for taking the time to gently push me in a direction to think more and further about software testing and their context. I recommend also others to monitor his blog

Almost at the end of the day, no time to get home as there certainly would be traffic jams all over the country, I attended the book presentation from related to Agile testing by Anko Tijman and Eric Jimmink: Test2.0. Besides the presentation there was also food. Plain good "Snert" with bread.

Software Quality Conference 2008, Dusseldorf

On April 17th 2008 I visit the Software Quality Conference in Dusseldorf. Based on past experience I knew that it would be a interesting day. The main topics I intended to visit was Model based testing and security testing.

Presentations I attended:
Security Development Lifecycle by Michael Kranawetter, Microsoft (D): Interesting points here were the obvious awareness you are never on time on testing security. As at the moment you are deploying a fix, the next exploit is already waiting. A strong point of the presentation was defining a security development lifecycle. M. Kranawetter recommended to plan about 20% of the time of a regular testing process for security testing. See picture below for the model of the security lifecycle:

Based on this lifecycle he mentioned that there must be actions taken for process improvement.

He ended his presentation with a short movie see: http://video.google.com/videoplay?docid=5627966010916286426 somehow I forgot the reason he showed this to us, though it was very funny.

Software Security Metrics 101 – Why & How? by Dr. Markus Schumacher, Virtual Forge (D): This presentation gave using examples a basic overview why security is important and will become more important in the nearby future. He came up with a quote that 0.2% of the CPU's are placed in PC's/Servers and the remaining 99.8% in regular products like watches, toasters, cars and perhaps in the nearby future in milk cans. He used this statement to express that our environment will change in the future as we will use more of the information on those cpu's like RFID's etc. At least he opened my eyes that testing security for those situations like embedded software will take a much important place in the nearby future. And since embedded will result in using cheaper production methods, security might become an issue over there which needs our attention.

Model-based Testing Enhances Action-word Based Testing to Boost Test Automation by Emmanuel Verge (Fr): In this presentation the position of model based testing was explained and how their tools can support that approach. It gave some basic view how a process of model based testing looked like based on an example UML model using their tools. One of the strengths I see is that you start defining you test model based on requirements and define models for it which can be used for deriving test cases. The tool they have to support this is called: Leirios Test Designer. Using this tool test cases can be designed based on the model of test cases. You immediately get an overview how your coverage is against the models and therefore the requirements. If the test cases are defined you can use their tool called Leirios Test Publisher to create test script.

In the approach related to model based testing you have the following phases:

1. Requirement Management
2. Model-based testing
3. Test Management
4. Test Automation

A Maturity Model for Model-based Testing by Thomas Rossner, Imbus (D): One of the first statements T. Rossner made was: "Model based testing is not UML" With this statement he tried to trigger us that we have to see model based testing as a process where models are used. During his presentation he explained more in detail how a maturity model fits in Test Process Improvement (TPI) Model. He explained which key-areas are suitable for usage in model based testing and how the maturity levels would be according to him. As he defined that model based testing also knows a certain maturity leveling, therefore it is hard to expect that the usage of model based testing will result immediately in fast and better results. I think the main thought he wanted to give is that you can also improve your model based testing process during time based on a customization of the TPI model.

Some Exhibitors I talked to are:

1. Leirios: After I had the presentation about model based testing I visit their stand. One of the strengths of their tool is not only that if a requirement changes you only have to alter on just a limited number of locations you script. I think that based on their approach and tooling as tester you are able to tell management what the impact for testing will be if a requirement will change based on the effect of the number of test script that will change. I think it is worthwhile to take a look at their approach and their tools.


2. Microsoft: At this stand some detailed information was given about the usage of Visual Studio Team System 2008. And how it incorporate several processes of development in one suite. I think one of the benefits of using tools like this is that you can embed tests better to development. They gave me some trial versions of the Team Foundation Server and Team Suite. I hope I can post some experience of these tools very soon on my blog.


3. Metrixware: They provided information about a tool to monitor not only the infrastructure during test processes, though also monitor the system while it is in production. This area is quite new for me, perhaps it is a tool which can be supporting infrastructure testing.


4. SQS: I obtained a demonstration about SQS Professional. As I know this tool from own experience it was good to get again confirmation the focus more on automating the test process rather then automating test execution. I still think the first gains can be get in this area.


5. Frologic: The presented a tool called Squish. The strength of this "record/playback" tool is that it supports different platforms. And supports different scripting languages like: perl, Python, Java, Tk and some more. for an overview you might take a look at: Squish


6. Wibas: I was triggered by the Map Of Change they presented (see picture below). I think with this map the tending to trigger people that testing is not a process on their own. It should be part of the process of improvements.

Some interesting links related to:

security testing:
The Security Development Lifecycle
Security Developer Center
Michael Howard's Web Log
Security Development Lifecycle (SDL) Banned Function Calls

Model-based testing:
Wikipedia - Model based testing
model-based testing home page
Model-Based Testing in Practice
Model Based Test Generation Tools