Neil Pandit of Sopra Group, winner of the Logica Triple Star Award for Innovation at the EuroSTAR 2009 conference, gives us a taste of his winning presentation where he described the benefits of a visual approach to risk-based integration testing.

With reduced budgets, resources and a need to deliver projects faster, risk-based testing (RBT) is an area that is increasingly being seen as an approach that stakeholders and project managers are keen to adopt. Often, project stakeholders are attracted towards risk-based testing as the benefits are easily promoted: reduced testing costs through targeted testing and reduced risk of failure in production.

RBT is about developing an objective technique to prioritise the testing effort, with a view to reducing the overall testing effort, while minimising any increase in risk exposure to the organisation. It is, therefore, an ideal approach to be applied to projects where there is a combination of limited time, money and people available. Unfortunately there are often misconceptions about the use and objectives of RBT, which can lead to it being perceived as a ‘taboo’ technique within some organisations.

This short article will provide details of the presentation that I delivered at EuroSTAR 2009 entitled “A Visual Approach to Risk-Based Integration Testing.” The visual approach highlights potential risks using colours annotated onto a system architecture diagram resulting in the creation of a ‘heatmap’. The process considers both the business impacts and technical complexities of the systems, with the architecture diagram being mutually agreed by all stakeholders within the project. This article will also highlight some of the misconceptions and problems encountered when using RBT and highlight other testing areas where the concept of heatmaps can be applied.

However, before I discuss the detail of using this visual technique, it is useful to consider a relevant concept which highlights why two key roles in a project (developer and business user) have very different perspectives of RBT. This observation together with the consequences mentioned below is the primary reason for developing this approach to RBT.

A developer that is asked to apply a risk-based approach to testing for Unit Testing or Unit Integration Testing will tend to focus on the technical nature and complexity of the modules or solution. As far as the developer is concerned, the highest risk will be in areas that are considered new, unstable or technically complex. The developer’s focus when applying a risk-based approach to testing is therefore more at a technical level rather than on any knowledge of business criticality of the functionality being developed. If asked which areas the system integration or user acceptance testers should focus their testing efforts on, the developer would generally point to the technically complex areas as these are believed to be of the highest risk.

From a business user’s perspective, however, these technically complex areas are of little importance and his/her focus will tend to be on ensuring that the critical business processes required to keep the business operational are functioning correctly.

This difference in focus can subsequently result in two parties with very different perceptions on risks and how testing might mitigate that risk with both developer and business user having ‘supposedly’ performed a risk-based approach to testing; this can result in a combination of two serious consequences:

1. The business user encounters a critical defect in an area of functionality that has not been tested by the developer due to the functionality being perceived as having a low technical likelihood of failure.

2. The business user does not test a non-critical process and so fails to detect a defect in a technically complex area that the developer would have expected the business user to have tested, resulting in unanticipated production problems.

This is where some RBT approaches may be biased towards only the business impact of failure and ignores the technical likelihood of failure.

From my experience on a client where this specific observation was made, both the business and technical teams were overwhelmed by the number of supposedly ‘non-critical’ technical production defects and instability in the systems to the point where it was significantly impacting the business. This was despite the business users having tested the ‘critical’ processes based on business impact. The inability to fully grasp what RBT can provide and the further amplification of problems during System Integration and User Acceptance Testing leads us into the key misconceptions of RBT.

Clarification of objectives of RBT: The objectives of RBT can often vary in organisations, depending on who you speak to. From a testing perspective the objective is typically to mitigate risk while reducing overall testing effort, from a project manager’s perspective it is often to reduce costs but not compromise on quality, whereas from the business area, RBT can be viewed as a reason by the project team to do less testing, and is sometimes therefore perceived as actually increasing the overall risk.

Independent of software delivery: Another misconception is that the RBT approach can be developed independently of software delivery. A prioritised order of execution can be produced, but if your systems and components are subsequently delivered in a different order by your development partner, this may crucially affect both the scheduling and effectiveness of your test execution.

No defects in later test phases: There is a misconception that RBT will find all the major defects, so there is great surprise when a high severity defect is found in UAT. The two issues here are:

1. It is ‘risk’-based, which implies there will be risks remaining and therefore potential defects.

2. The defect may be a major severity defect but this may be in a non-critical system or process. Again as a result of defects remaining, the perception by the business is that RBT still leaves you with defects.

In addition to the above misconceptions, testing of major system integration projects can result in a number of other problems:

Obtaining ‘objective’ risk assessment of multiple integrated systems: It may be straightforward to apply an RBT approach when testing a single system as it usually involves a single technical / business resource assessing the risks, but with a large number of systems and interfaces, and potentially a number of resource involved, it will be harder to objectively assess the risks of failure of systems.

Focus is on systems and not interfaces: Typically when applying RBT, the focus tends to be on systems and so interfaces and data transfers can get overlooked. Often, the business users may not know that data is being transferred, as they may not be aware of the lower technical architecture of how data is transferred from one system to another. This may result in too much emphasis being placed on testing individual systems and not enough on interface testing.

Reporting does not focus on risks mitigated but on ‘numbers’ of scripts executed: Although execution is risk-based, progress reporting tends to continue to focus on the number of scripts that have been run. Typically, project managers are still looking for how many scripts have been run, rather than the risks that have been mitigated. Progress reporting also lacks visible representation to key stakeholders and clarity over where the problem systems or interfaces are within the architecture / process. By stating in a weekly report that the interface between system X and system Y is not working means very little when out of context with the overall end-to-end business process.

An approach that I have successfully applied uses the concept of heatmaps, which takes into consideration both the misconceptions and problems highlighted above of testing large system integration projects.

A heatmap is a visual representation of data or information through the use of colours. This method of communication is accessible to all levels of users and is commonly used in other sectors outside of software testing.

So where have you seen them before? I am sure you are all familiar with weather maps which use different colours to indicate warmer areas than others or different colours to indicate heavier rainfall across different areas. Other examples are in the stock market where you see that blue typically means that shares are rising and red means that they are falling.

Our approach to applying heatmaps for RBT similarly provides a visual representation of the testing priorities through the use of colours. In terms of what we use as our ‘map’, as the most commonly understood diagram on a project is the system architecture diagram, this is used as the basis for applying the heatmap approach.

We then set up a workshop with the technical experts or owners of the systems to get an understanding of the technical complexities and risks of failure of the systems and interfaces so that we can annotate these onto the diagram using colours. The principle is to apply different colours to indicate different levels of priority to the system architecture diagram to demonstrate the technical likelihood of failure.

This provides a technical heatmap based on four levels of priority such that critical may relate to a new system or interfaces being created, unstable existing production system, new technology / supplier etc. A low likelihood would be where there were minor / no changes to a system or interface or where a system or interface has a low production defect history.

Having created the technical heatmap, we set up another workshop with the business areas to understand firstly the critical business processes and map this to the system architecture diagram. We then perform the same prioritisation exercise using four levels of prioritisation with the remainder of the systems to agree a prioritised business impact of the systems and interfaces.

So a critical priority system may be one where failure incurs a significant regulatory fine, has direct financial / customer impact or reputational impact. A low severity impact may be a system or interface which may have manual workarounds that can be put in place immediately or internal systems and interfaces which are non-customer facing.

So, at this point we have a system architecture diagram with the technical likelihood of failure and an architecture diagram with the business impacts of failure of each system and interface. Using a matrix we take the business and technical priorities for each system and combine these into an overall prioritised approach. So a system that is considered to be high (orange) from a technical risk perspective and a high (orange) from a business impact perspective will have its overall rating increased to a critical (red) rating.  This now gives us a combined system architecture diagram.

As a result of using this technique, we end up with a final combined agreed diagram that not only takes the business impact into consideration but also the technical complexities of the solution. This now satisfies not only the business users in addressing their concerns but also addresses the technical areas that testing is focusing on potentially less stable areas.

This diagram also tells us that the areas of focus for testing need to be on broker sales system, customer database and policy management systems as well as three critical interfaces. You will still be testing other lesser priority systems and interfaces along the way as part of any end-to-end test, but the key point here is to understand the systems and interfaces that form the primary focus of our testing against those that we simply use to help us in our test.

However critical the system might be, we still need to look at the technical risks of failure rather than simply testing a system or interface because the business has identified it as critical.

There are many benefits of using a visual approach as described above. In terms of the key benefits, these are:

Highlights the RBT approach visually: The primary benefit is that it enables you to visually prioritise and focus your System Integration Testing.

Provides ease of understanding for all stakeholders: This is a simple method of communicating the RBT approach to all stakeholders who may not have much knowledge around testing.

Incorporates technical risk into test design: Provides testers with a clear focus on test design of scripts ensuring scripts are created that cover the maximum number of hotspots while incorporating the technical risks. Your test design will still cover lower priority systems but you can adapt the design so that you are not spending significant time identifying data variants in areas of the integrated system that are of lower priority.

Provides risk-based reporting against specific interfaces and systems: Enables you to use the heatmaps to provide visual progress reports on specific interfaces and systems.

In order to apply this technique successfully it is important to be aware of some considerations and challenges that are typically encountered throughout the process. Being aware of these challenges will help you to apply this technique successfully.

Obtaining full buy-in from subject matter experts and technical resources: This is not an activity that testing can do in isolation and so needs full buy-in from the business users and technical architects to attend workshops. The demand for key resources to attend workshops requires them to get away from their day jobs.

Understanding impact of IT-driven processes: There may be some activities that are not business-led processes and so these may get omitted from the assessment. These may be things such as maintenance activities, updating rating tables, postcode tables that are run by separate IT teams, which could have a major impact to system functionality if failed but are not shown in the architecture diagram or identified as a business process.

Agreeing organisational business impact: It can be tempting for some business areas to exaggerate the organisational impact of failure of systems as they do not want to be supporting unstable processes. If this results in additional business resource performing corrective action or manual workarounds then budgets may also need to be protected. My experience has shown that some personalities exaggerated the impact of certain issues in order to have their business areas tested to a greater depth, rather than taking into consideration the wider impact on the organisation.

Ensuring use of up-to-date system architecture diagrams: We need to ensure that the architecture diagrams include all the systems that are being used by the business and not just those supported by IT. They may not include bespoke systems that are potentially created / purchased by the business or even redundant systems that are no longer supported by IT, but still being used by the business!

Lack of availability of fully integrated test environments: Finally we have agreed the RBT approach using the system architecture diagram but in reality it may not be possible or feasible to create a fully integrated test environment. The alternatives such as test harnesses, manual checking of files etc, all need to be assessed when planning an RBT approach as this may impact the risk assessment itself if you are not actually using the correct systems.

So while this article and approach has focused primarily on using heatmaps for system integration, this same technique can be successfully applied to other areas:

Can be applied to prioritise system testing: We have applied this approach at a system integration level but this information can also help prioritise any lower level testing such as system testing ensuring that there is maximum coverage within the specific systems.

Application to third party software: You can also apply this to third party commercial off-the-shelf (COTS) products where there may be a core product and you may want to know which components have changed as a result of the bespoke changes / configuration that is required.

Prioritisation of defect fixing: We can use the testing priority of systems and interfaces to help with determining the priority of both project and production defects such that the approach can be used to prioritise the order in which defects should be fixed first.

Regression testing prioritisation: Through identifying prioritised business processes and end-to-end tests it can also inform you of what to include in your regression pack.

Reassessing RBT for incremental developments: For incremental developments, your RBT approach may change as you may have new systems and interfaces being developed for each increment. If an increment is adding new interfaces and functionality, the heatmap can be revisited to consider the new impact of the increment.

This has been a brief introduction to using heatmaps for providing a visual approach to RBT but the key points to take away are:

Agree the objectives of RBT: RBT will continue to be used within organisations, but if not applied correctly will be disregarded by the same people it is intended to benefit (project managers, stakeholders, the business users).

More effective focus of System Integration Testing effort: We need to have more effective focus on System Integration effort as poorly planned System Integration testing and incorrect focus could result in significant time and effort being wasted.

Heatmaps – a possible solution: This approach can provide an objective risk-based and visual approach to System Integration Testing which combines both the technical risk with the business impact.

Ensures common understanding: It provides clarity to stakeholders over what has and what has not been tested and can assist with reporting progress visually.

Finally, in terms of the key challenges with this approach, we must remember that the heatmaps and any RBT approach cannot be performed by testing in isolation, and so it is essential to get full buy in from both business subject matter experts and technical resource. This process is about getting all relevant parties involved from the outset in achieving the common objective of agreeing the scope, priority and focus of System Integration Testing.

Neil Pandit

Senior test consultant

www.sopragroup.co.uk