Reverse Business Intelligence testing

Currently Business Intelligence is quite hot. Lately there are some shifts in BI-specialized companies as shown in the following article
IBM, Cognos, and the End of Best-of-Breed

Perhaps this will make the live of a tester easier as components will be more integrated. On the other side it might challenge us more in the nearby future as those products might be integrated and placed on the market too swift which might lead in only proven technology when organizations are using the total solutions as they are offered.

What I learned is that organizations always need their own customizations. Sometimes they are forced by habits how the business works; sometimes they are forced by law.

Of course there are several ways of testing these systems. Here an approach which might help to identify if the reporting tool is able to create necessary reports based on existing data. (Keep in mind that this is just a simple description of a possible approach.)

Start testing based on the information needed. And work through the system backwards to identify your starting point.

This might be reached by the following steps:
1. Identify the KPI's
2. Identify the reports these KPI's are involved
3. Prioritize these KPI's
4. Start with the most important KPI
5. Identify the data these KPI consist of
6. Identify the source of those data
7. Identify the quality (appearance) of already existing data related to these KPI's
8. Create a test data set based on the appearance of this data
9. Run the reports

The challenge here is when errors occur, it is hard to say were the error lies, is it in the reporting tool, in the data warehouse or in the source data. At least we know that there are errors which give wrong information about the KPI's. If the location of the error is found and it is related to the (historical) data, then it might be better to correct that data instead of continuing with the next KPI. As those error also might influence the other KPI's.

If there are no unacceptable errors you might continue with the next KPI.

Though this approach will not cover any errors based on data created by the system. For this we might have to extend the test approach by the following steps:
1. Identify the functions of the main system which creates, alters this data
2. Identify the data model and architecture related to these data and functions
3. Define you test cases based on these functions, with this you make sure that data creation is of a certain quality

And continue with testing the next KPI.

How much don't we test?

Every time I get involved in a test project there are questions like: How much test cases do you have? Do we test enough? Why do you need that much time for testing? You have to test everything!

To me, those questions are always asked to soon. At the end of a project I can tell them for sure how many test case I have. It is just counting those cases. If we did enough could be answered if the organization is still happy after going life. Why I need time for testing can be identified based on the decisions made about the quantity of available Money, the needed Quality and the given Time.

And as always it should be:
- Money: Cheaper
- Quality: Better
- Time: Faster

What kind of information would it give in front if we answer the questions about number of test cases, and if we tested enough, and why we need that much time?

I think a more interesting question to answer is what we did not or will not test. If we provide that in terms the business knows they are able to pick one of those three options.

What planet are you from?

Did you ever come into a situation when someone starts talking about testing you asked yourselves: "What planet are you from?" Or did someone else had this question about you?

Perhaps you have the same feeling when you are reading my postings on this blog.

Still this is quite an interesting question. As starting wondering about planets you accept the idea about universe. As in a universe you have more planets. And perhaps I'm just sitting on another planet you are living on. Still this doesn't mean that we don't have any interaction or need for each other. For example the moon has some role making the earth rotate in a steady way.

Wikipedia says the following about Universe: "The Universe is most commonly defined as everything that physically exists: the entirety of space and time, all forms of matter, energy and momentum, and the physical laws and constants that govern them. However, the term "universe" may be used in slightly different contextual senses, denoting such concepts as the cosmos, the world or Nature"

In projects there is sometimes need for testers, test coordinators, test management. And often they apply for those people with general terms like: The person should be skilled in a test method like TMap. The person should be ISTQB certified. The person should be a team player. And so on.

As test professionals we pretend to be experts based on our knowledge and experience in a certain area. Sometimes, perhaps sometimes too often, other project members don't understand what we are talking about. Though they know we exist and we have to make also our contribution towards the project. It is our task and duty to make them understand how we can influence the project success.

This can be a hard task as our position in projects might differ. Sometimes we are on another location, far away from project management and developers. Sometimes we are working next to each other. In all situations we influence each others activities. There for it is important we know what relation we have towards each other. We have to describe the project universe and explain our roles and positions. Perhaps more important, we don't need to understand each other in full detail, we have to accept the existence of each other and the need for success.

Let us be the moon and the project be the earth. Without us the earth would not rotate in a stable position. It is not necessary for the earth how we do it, as long as we can convince them we are in control.

Perhaps the answer to the question: "What planet are you from?" is no longer important. We don't need to understand each other in full detail as long as our position is clear. If they consist that you answer this question you can always say: I'm not from another planet. I'm just part of your universe. I'm the moon who shines bright white light and makes sure you planet is able to rotate steady.

RFK and knowledge processing

After writing these blogs: The RFK process explained and Request For Knowledge (RFK) somehow I got the feeling that I never could be the first of this concept. This triggered me to search for the word I was thinking I invented it: Request For Knowledge.

Fortunately for me: I had many hits and I came up to several interesting documents. I think I can say fortunately as these articles and presentations confirmed to me that here is another challenge for us testers or even business analysts to continue working on my initial idea. The challenges to collect information in a structured way were we might define the process to get grip on risks based on initial missing knowledge.

Although there is lots of information here some of those I found.

On Indentifying Knowledge Processing Requirements by Alain Léger, Lyndon J.B. Nixon and Pavel Shvaiko, May 2005
In short, presents a first step towards a successful transfer of knowledge-based technology from academia to industry.
I think the relation towards us testers here can be the usages of use cases to identify knowledge requirements.

Some Applications of Conceptual Knowledge Processing by Prof. Dr. Gerd Stumme
In this presentation an short overview is given how ontology’s are related to tasks and examples.

What I liked here was the introduction of the term: Ontology.

Ontology: "An Ontology is a formal specification of a shared conceptualisation of a domain of interest." T. Gruber, 1993.

Ontologies support a.o. the following tasks of knowledge management
• Acquiring Knowledge
• Organizing Knowledge
• Retrieving Knowledge

BOVIS LEND LEASE ikonnect Facilitated Knowledge Sharing by http://www.knowledgestreet.com/ September 2005

Some of the interesting points here is they specified several roles as Seekers and Facilitators.

Another interesting posting I found was the iWorkshop on [a-i-a.com] iWorkshop™ Knowledge Management and Collaborative Work

If we think a defined RFK process is needed to get more insight in the risks towards the software then knowledge processing might be a way to go. To me this seems to be a very complex process. Though this should not withhold us from collecting and translating the information in possible risks.

As in one of my articles I showed an example how RFK's on one area might impact the testing effort on another area. To improve this process, it perhaps might help to work on a certain ontology. As the moments of tasks mentioned can influence the way we have to define our strategy. We have to learn to adapt instead of forcing our process towards the organization.

Related to roles we can take, I think we will be as well seekers as facilitators.

I know I'm not unique with the term RFK. I believe there are some gains to get, focusing on this process also. It might help us define improvement suggestions towards the organizations. As well in front of a project as well during evaluation of the project.

The RFK process explained

In an earlier post I submitted the idea of RFK's (Request For Knowledge). The basic idea of monitoring the process of RFK's is measuring were formal requests are made and how and when they are answered.

In software development processes there are some number of methods and techniques to deal with a part of this like inspections, reviews and walkthrough. Based on their character, formal/informal, sometimes notes are made if it is a failure in documentation or a when it is a question is something is unclear. Basically this covers the monitoring of knowledge request for a part when documentation or requirements are already available.

In my opinion you can start the RFK process much earlier. You can start when the project is initiated. I mention "can start" intentionally. As I strongly believe you should not change you development process to this RFK-process. It should be otherwise around. You should define you RFK-process based on your development process and also your organization.

Basically the steps for the RFK process can be defined as following:
1. Define a Knowledge-need-landscape;
2. Name the development method;
3. Identify participants and stakeholders;
4. Define the areas were information is needed related to the system
5. Define the risks and impact of missing knowledge within the method;
6. Define the phases of monitoring the RFK's;
7. Select your toolset how to monitor RFK's;
8. Report on Impact and status of RFK's;
9. Get agreement how to deal with this impact;
10. Adapt your processes based on made decisions related to RFK's;

1. Define a Knowledge-need-landscape
For adapting your RFK-process to the organization you have to identify the locations where a possible need for information exists. To get a clear picture of this it is important you know how your development process will be. Often you will see that a specific development method is chosen.

2. Name the development method
If project management did not named the development method you have raised you first item for the RFK-list. In my opinion it is important to make a statement of the chosen development method. As every method has its strengths and weaknesses. These strengths and weaknesses have to be translated by the impact on knowledge gathering. Often you will see that the weaknesses of a method are translated in other steps which make the method a customization. This will consist in newer possibilities for information needs.

3. Identify participants and stakeholders
If the method is know you will be able to identify the participants of the project. Participants are potential persons to ask for information. There are also stakeholders. If you have identified these you are for a huge part able to tell were information should come from. Only keep in mind, also stakeholders can ask for information.
It is important that mutual agreements are made to provide information, how it should be delivered and when.

4. Define the areas were information is needed related to the system
Based on the RFK-landscape you are able to tell were information is needed. You might start mapping this to the requirements of the new system. The importance of requirements can also be used to classify the importance of RFK. A possible method to do this is using the MoSCoW-method. Next to linking them to requirements you have to link them to the processes and sub processes of the system as I mentioned in my previous post, for some processes they are not that important, for others it is most important.

5. Define the risks and impact of missing knowledge within the method
It is essential to define the risks and impact of missing knowledge. This will help you define the acceptance criteria. If some risks are unacceptable, strong agreements have to be made here and also monitoring should be more in place.

6. Define the phases of monitoring the RFK's
Not all projects need full monitoring. You have to adapt it on the organization as well. If an organization decides not to spend a lot of time on the requirements phase to get them defined in detail, you might decide to monitor not during this phase on requirements, only in the next phase when requirements become detailed enough. What you might think about is that besides developing the system, often a parallel project is started to create manuals and courses. You might also take this into consideration as the information request should match the RFK's in the development cycle. Perhaps new RFK's are initiated. These can be asked to the system under construction as well.

7. Select your toolset how to monitor RFK's
The toolset is used to register the RFK's. This can be done in a home made tool, as I don't think there are already tools to be used for RFK's. You also might create a simple list using a spreadsheet tool. It is important that the tool is flexible as the need for information based on RFK's might grow during the project.
Another tool might be a flip-over on the wall were you place sticky notes for every request.

8. Report on Impact and status of RFK's
Besides of registering RFK's you also have to report on these. And more important, people have to act on the information you provided.

9. Get agreement how to deal with this impact
To make people act on provided information it is important that you have defined the roles and responsibilities for acting on RFK's impact analysis. If this is not done the impact on not acting might have negative influence on the project, you should be able to point people on their responsibilities.

10. Adapt your processes based on made decisions related to RFK's
When RFK's are raised they can be neglected or granted. Either way this has impact on all processes in a development cycle. For instance, if RFK is granted, this leads to newer information. Related to the testing process you have to identify what impact it has on your test cases. If a certain amount of RFK's are rose for a specific area you have to check what your test effort is on that area. If it was low, perhaps you have to increase it as that area is a potential risk of errors as information was not complete or changed over the time.
This also means that you might have to change your acceptance criteria during the testing phases.

When I start writing this article I thought about painting some kind of picture how this process would look like. Only this would reduce the adaptability to all kinds of development methods. Therefore I kept it with defining some steps. I hope these give you some help and convinced you (perhaps just for some small bits) that also RFK's are important.

A project model from Pettigrew in testing

One of my former teachers though me a simple model about the success of a project. I think this can also be used for the testing process.

1 = Experience the need (as well the internal as external needs)
2 = Clear vision
3 = Capability for change
4 = Good start, (Executable, plan of approach)

I shall not explain much now on this model. Just take your time and think your recent project over. See how they fit in this model. If you saw one of these 4 situations occur, you might now be able to find the actual cause of this result.

For the future, it might help you identifying much faster the direction of your testing process. And perhaps you have still some time to behave such to fill in the question marks prevent your process is getting out of control.

Success of improving your Test Process depends on culture

Have you ever been in a situation were you know based on facts and methods how you want to improve your test process and still it fails?
Have you been in such a situation were you did everything right and still you did not gained the expected result?
Did you had that feeling afterwards that the organization was not yet mature enough to deal with your improvement suggestions?

Although there are enough challenges you have to concur, I think one of the important things is the culture of en organization. And I think this is a hard one to beat because organizations are based on human activities and humans are thinking and acting different. Still it should not be impossible as organizations consist of humans who think the have a common bounding. This common bounding should be based on something.

Geert Hofstede defined a model sometimes called the Union Model.


Symbols: These can be words, statures, images or other important signs which express what is important and meaningful for an organization. Examples: Language, Humor, clothing, Logo etc. .

Every organization is using them intentionally or unintentionally.

Heroes: In organizations heroes are normally role models. These are persons who people think are important for the organization. This can be based on their behavior, their contribution to the organization or their profession. Normally these are the founders or other important leading persons. Often their are more heroes in an organization, as well as small as big heroes.

Rituals: These are social actions or habits which are found important in an organization. These rituals show often some kind of pattern. Examples: way of greeting, eating and drinking together, Friday-drink, monthly meetings, way using communication like email, phone, etc..

Values: Are those things people approve and trying to comply with like: loyalty, solidarity.

It is hard to measure deeper values like fundamental principles. They are often dealing with visions and presuppositions how it is actually working. If there is consensus within the organization about the fundamental principles no discussions have to be held and they also give direction to possible solutions.

As in an union, you have to remove shell by shell to get to the center.
Sometimes in software testing you are a pioneer. Then it is hard to get examples for these shells. Though you have a world of experience you can take examples out of. Perhaps here we also make a mistake. We take examples from other projects, books and trying to place them in the organization you are working for. When you are aware you are doing this, please then be also careful, because now you are trying unintentionally to change the culture. I think if you are dealing with organizational aspects of culture, then do in intentionally.

If you relate this to testing then the first shell can be easily defined.

As Symbol you can take the name of the test method like: TMap. Or terms like: Test script, test case, test plan, ISTQB-certification. (I tried to find an example of Humor, though our job is a lot of fun, and humor is there enough, it is hard to give examples. Perhaps this one might work: "It is not a bug, it is a self developed feature!")

About heroes: you might take your test manager as hero. Though we have some good names in the world also like those who wrote books and expressing our profession all over the world like J. Bach, P. Gerrard and more.

When coming to rituals and values, you get more to the hart of the organization. Still this will be also the area you have to pay attention to as they will influence for a great deal your success for improving the test process.

I think it will be a waste of time if you force an organization to improve if your improvement suggestions don't fit into the culture. There are lot of books written what should be improved and how to do this. Only did they also mention to pay attention for culture? Or was the goal which fits the means?

I think when improvements have to be done we as testers have to pay attention to the culture of the organization. We might start identifying the shells of the cultural union. We have to try to translate those improvements in terms as they could be part of the content of these shells.

Development Models and the impact on Testing

During my career as tester I learned that development methods have impact on testing. Also when sometimes a tester is working based on a test method it has impact on the success of the development method.

In books related to software testing often an approach is defined which is based no the Waterfall method. In the newer books they adjust that method to make it fit for incremental and iterative methods. Sometime this is done by only mentioning that it also works for other approaches.

In this Blog I'm trying to give some information about 3 development streams. It might not be completely perfect but I think it might be useful for the readers to give a picture how I see things. While I already know I want to write, I also know that this article will be much longer then I normally write. To miss information I decided not to split this topic in several other postings. Therefore I will give you some table of content:
- 3 approaches of development
- How development methods are linked to these models
- Short overview benefits and disadvantages per development method
- Overview how other models are related to these methods
- Impact towards software testing.

3 approaches of development
In general you have 3 development approaches. Waterfall, Incremental and Iterative (Evolutionary).
For more information you might check out the links which refer to description on Wikipedia.

In short these approaches can be described as following:
Waterfall approach: The whole system will be implemented at once;
Incremental approach: The system is divide is several steps, and is implemented in that order. Once a step is implemented you can not go back to a previous step to change functionality or do some "refactoring";
Iterative approach (Evolutionary): This is an approach were you continue development towards a final system. You are able to go back and change the code of the previous iteration. every iteration delivers a better system, even without adding new functionality.



How development methods are linked to these models
Often several types of development are called to be a variant on Agile Development, I rather prefer to keep speaking in terms of approaches. In this case Agile belongs to the more Evolutionary approach. Sometimes an approach belongs based on the actual implementation to either Incremental or Iterative. In that case you should not only look at the name of the development method like RUP or DSDM. You also have to investigate how that method is implemented. Perhaps the following picture gives some overview.


Short overview benefits and disadvantages per development method
In the last few years we tending to neglect Waterfall-approaches and claiming that methods as RUP, XP and Agile are always better. Only in some situations it is better to play save then take the risk of using methods you don't control on a level which is necessary. Mostly systems which has a long life-time like aerospace systems, life care systems or systems where you are a strong market leader and those don't change that much. I could be wise to use for these systems a waterfall approach. In the figures below an overview is given about the benefits and disadvantages of the 3 approaches. I know they are not complete. Still it can be handled as a guideline.







Overview how other models are related to these methods
To understand the impact of these approaches on Software Testing you have to understand how an approach effects the organization. To get a short overview of this I took several other models and views which can represent an organization.
Models:
- Organizational Model: Documentation about the organizational processes, procedures and activities;
- Information Model: Documentation how the organizational processes are supported/executed;
- Data Model: Documentation about the kind and location of information used;
- Process Model: Documentation how data is processed to support the pervious models .

Views:
- Conceptual/Logical View: High level view of the organization;
- Physical/Technical View: View of the architectural and informational representation of the system;
- Implementation View: How the system should be implemented in the organization.

In the tables below I tried to translate the development approaches towards the impact on documentation using several other models and views. To visualize this I used several colors.

When a certain cell is colored Green, then during the project this will not change much after it is finished. When the color is Yellow, there is a small chance it will change. The color Orange represents an obvious chance of change and the color Red: there will be changes.

The situation here is that when a chance occurs/ is made. The "defined" steady objects might also change when it is a result of new awareness. When this happens the test cases will change because they are built based on this documentation. Because changing documentation in previous steps takes time it gives an impression how steady the development process will be and therefore the test process. On the other hand it also give a feeling for ability to adapt your test process on those changes.

Views for a Waterfall-approach


Views for a Incremental-approach

Views for a Iterative-approach



Impact towards software testing.
Below you find a short overview of the characteristics of the development approach, their impact on the project development process and the impact on the test process. Keep in mind that these items are not complete. It is just mentioned to serve as a guideline. There is always a dependency towards the actual implementation of the development method.

I hope this blog gives you more feeling how development processes are impacting the test process. Bare with me when I made some miss interpretations. It came never to my mind to make a statement about which approach is better then the other. It should help you as a guideline to identify impact on your own process and therefore helping to identifying risks and controlling those.

Request For Knowledge (RFK)

Terms like RFC or CR are well known in the development processes. They often are raised during the testing phase and some new features have to be developed. Most of the time this has its impact on the testing process. It is hard to test them within the same time period of the testing phase.

Mainly these are created based on new knowledge of the capabilities of the technology or system or even process. Prevention of this can be done by performing formal reviews or inspections. The outcome is registered in some way. Though these are not directly linked in testing strategies were additional risk might be involved.

For example: If formal reviews are executed, errors are identified and registered in a review log. The information about how many defects in documentation were found in certain areas are not used as they already solved those issues. So the risk for errors in the system is reduced. Only the change for RFC's is not implicitly reduced.

Perhaps we should introduce the term Request For Knowledge (RFK) and also make a formal process of it.

If I have to think about a definition for this it could be something like this:
A RFK is a formal request for additional information about a system, architecture, process, data or requirement.

The process could be something like this:
1. Create and maintain a RFK backlog; every person of the project can add items to it.
2. Assign a RFK backlog owner;
3. Start monitoring during requirement-specification phase
4. Every RFK should have a status, owner and result
5. For every phase transition, also handover the RFK-backlog.
6. Before the handover, identify the impact of outstanding RFK's on the next phase
7. Take counter actions for that impact.

The reason to do this is creating a process to get more and new information which could help reduce RFC's which have to be delivered tight away during testing phase.
I think when we start monitoring every area / subsystem/ process where questions are raised can give us information where functionality might change.

If in a period of time certain areas keeps requesting for knowledge then the chance is huge that the functionality doesn't meet the actual desired requirements. So the chance is also huge RFC's are introduced on a later stage.
On the other hand, if certain areas don’t have RFK's, perhaps this is a signal that the requirements are also not defined thoroughly.

If you prioritize the areas and combine it with number of RFK's you are able to identify risk areas earlier.



If we assume that based on above table "Process A" had a number of RFK's for "Sub System 2" which was unexpected high. And we see that "Process A" is also used in "Sub System 1". The impact of changes in "Sub System 2" when information is retrieved during testing phase brings high potential risk for "Sub System 1". If we have this information in front of starting testing, the test strategy can be adopted to cover this risk.

These figures can also be used identify business involvement. In projects sometimes certain department give full involvement right from the beginning. Certain departments start involving at the end of the project and the chance of raising RFC's is increased.
These figures can also be used if certain area's have a lot of RFK's, the users are instructed comprehensively by more detailed manuals and training as the chance they miss-use the system is increasing.

Perhaps you, reader, can come up with other usages of the RFK's.