Somehow the question about how many test cases is so important for "important" people. As if they are getting paid by numbers instead of valued for delivered value. Somehow a false trust is derived from figures. People rely on numbers and assume that number of test cases represents good quality. This seems so obvious the way of doing.
The posting of James Christie triggered me to answer using my weblog.
I value the blog posting of But how many test cases? written by James Christie by its content in relation with his defined context. This context is lost when you translate it to numbers. Below you will find several attempt to make a good posting valued wrong.
Example 1:
Imagine that your blog posting is rated by the number of letters. In your posting you use about 5534 characters. Telling the same story using twitter you need over 40 tweets. Does this show value? It seems that one post in a blog provide more value then one tweet, although the tweet which pointed me to that blog was also very valuable. So is 1 more then 40?
Example 2:
Or what about the coverage of letters. You used all letters of the alphabet, this means your coverage is 100%, does this provide some information about the quality of your posting?What about assigning numbers to it?
Impressive usage of the letter “e” based on numbers it is far most the extensive used letter. Does this provide information? I don’t think so, Perhaps the letter “e” should be used more often, perhaps in relation with other letters. Even this way of thinking is wrong. It doesn’t tell any thing about the context.
Example 3:
What about visualizing the numbers. Below you see a snap shot I you look only at the numbers mentioned in his blog.
I also left some noise in it. The (con)text is now removed.
What does it value now? What information can be obtained? Perhaps the 100.000 mentioned in the text is impressiveConclusion
Is it correct to drive our testing on numbers? Is it useful to explain coverage in terms of test cases executed? Is the weblog of James in the based on my examples valid and good? The numbers are clear and proven? Are you counting the time?
I compliment James with his blog. A lot of time is spend to "proof" something which is explained incorrect. Wrong questions are asked.
Thanks Jeroen, I love this. It's a nice example of how metrics often measure something that is completely irrelevant. if you can't measure what's important, then measure what you can. Then, whatever it is that you are measuring becomes important, so no need to question it!
ReplyDeleteBy the way, it's a good job I worked in "bizarrely" or it would have screwed up my perfect coverage.
Again brilliant James,
ReplyDeleteHow brilliant you explained the process how metrics are used wrong.
"how metrics often measure something that is completely irrelevant. if you can't measure what's important, then measure what you can. Then, whatever it is that you are measuring becomes important..."
Shifting from value for business intended to change behavior to "presumed" value for person as just an activity.
Lovely post. It reminds me of my article "What Counts" at http://www.developsense.com/articles/2007-11-WhatCounts.pdf
ReplyDelete---Michael B.
Michael, I read you article, another great way to express that counting is useless when counting the wrong things. Funny how Monty Python also made the similar translation about complexity and number of words used.
ReplyDeleteWith the third example above I also ment to show that based on patterns also false conclusions are drawn.
In your article I would compare that with the example about noticing more things in a few seconds while testing based on a schedule presenting numbers information get lost.
Thanks for sharing that article with me.