Benchmarking test effort time useful?

Recently I was asked by another professional about the information which is created during a test process. Which products are delivered? Which information is reported and therefore collected?

I provided him some information with the remark: It depends.

It used the some of phases from TMap® to give him some food for thoughts about figures/product delivered per phase. This is just short list which cam in my mind at that moment, there are items which can and should be added and or removed.
Preparation
- Review results – provided to test team
- overview progress preparation functional designs/ use cases – provided to test - team/management/ project team
- Bi-weekly progress reports – provided to management and project team
- E-mail/ memo instead of escalation to stakeholder
- Test plan, provided to project team and test team
Specification
- Progress specification of test cases including the number of test cases who owns etc. provided to test team and project team
- Bi-weekly progress reports – provided to management and project team
- E-mail/ memo instead of escalation to stakeholder
Execution
- Issue report provided to project team
- Progress report provided to management and project team
- Checklists test scripts provided to test team
- Daily or ad hoc progress report provided to test team or stakeholders
- E-mail/ memo instead of escalation to stakeholder
Completion
- End report
- Issue data base
- Delivered value

When I met the professional I asked him about the project he is on to. It turns out he was collecting information based on experiences how much effort it cost and how much result is delivered during a period of testing. Based on this information from several different sources he is working on some general figures to support providing information in front how much testing will cost.

This is a very noble idea. Every one wants information how much testing will cost in terms of people, money, resources, skills. Every one wants to know what the results are like number of test cases, number of checks, number of issues found, coverage of testing and more.

Support people who challenging themselves to think further were others have been, continued or stopped.
I hope the professional I’m talking about will succeed and share his information with us.

To support him I will share some thoughts of mine which might help. Somehow I doubt it is easily done to collect information from several projects and compare it with each other and use those results as some kind of benchmark.

I believe this is hard because of differences in:
- Organizations: not only the business differs, they also change in strategy etc.
- Projects: you have to make a distinction between development approaches
- People: teams are not supported by the same people and same commitment
- Technology: It looks like you can compare counting in function points. In my opinion there are differences as technology is used by humans and they have different skills and approaches for example in error handling.
- Test approach and used techniques: these are often adapted based on influences from the items above. This results in numbers with similar names only the value cannot be compared. It will be comparing green apples with red apples. You know about the amount only the taste might differ or not.

Basically it is hard to compare the figures which are collected. What is the value of the number of test cases which are written in a certain period for a certain project? Can these figures be used to calculate the effort and money it will cost? Perhaps adding a certain percentage to be sure?
Can the figures how many cases in combination with found issues be used to tell how many test cases must be executed to provide trust in a system without the system being built yet?
Can the figures how many testers found the number of issues by executing a certain number of test cases to predict the needed resources?

Is benchmarking good? Perhaps some people need this kind of trust and use it as experience because they are missing experience? Is the prediction valuable to use?

I wonder when you are finished with benchmarking/collecting figures/information to provide a fancy chart on which people should trust.

Why should we spend time to collect? I value more the skills and experience I have and the ability to compare them to any new project. Sometimes the experience in numbers can be used, sometimes I can rely on, at least it gives me direction. Should I share it with others? Yes of course, in combination with the context and the lessons learned from it. Can others use it? If they are able to judge it against their particular situation and not using as common proven figures.