Test Strategy: Regression Tests

Regression test is a critical element in any testing strategy. It aims at finding mis-behaviour of a previously correct function, attribute or feature. These bugs basically fall into 3 basic types:

1. change exposes an existing bug;
2. change breaks something in the same part of the system; and
3. local change breaks something in another part of the system.

Among the three types, the third type is the most difficult to identify. However, regression test is not something we could treated lightly as breaking any existing features currently used by customer could easily cause great dis-satisfaction to the quality of the system.

Repeat All Strategy
With a repeat all strategy, a brute force strategy, is to repeat all tests that were previously run. This assumes that a set of regression tests was developed and well aligned with required quality. Usually, this strategy would be taken if risk assessment gave solid evidence to its need and sufficient time and resources are available. If all previous run tests were executed, we should be able to find the important regression bugs. Realistically, test automation is probably the only feasible mean to a repeat all strategy due to the large number of tests usually in the regression test suite. Even test automation is employed, there are always some manual tests that cannot be automated.

Repeat Some Strategy
For one reason or another, it is often not possible to repeat all regression tests. So, a subset of the overall regression tests must be selected to repeat. There are three major techniques to do so.

1. Traceability
Briefly, traceability is where tests are related to behavioural descriptions of the system like requirement specification elements, design specification elements or quality risks. We can look at what requirement, design element or quality risk is affected by the change, trace back to the associated tests and select those tests for re-execution.

2. Change Analysis
In this case, one looks at structural descriptions of the system to figure out how change effects could ripple through the system. Unless there is an in-depth understanding of programming and the system's design, help from designers and developers would be needed.

3. Quality Risk Analysis
With traceability and change analysis, technical risk, where bugs are likely to be found, is used to decide what to re-test. However, the quality risk analysis should be revisit to determine areas of high business risk. Even if bugs are unlikely, areas with high business risk should be re-tested.

Since not all tests were re-run, it is possible to miss some regression bugs if they occur in un-anticipated areas. So, it is advisable to use cross-functional tests to get a lot of accidental regression testing. The analogy is clearing a minefield. The old fashioned approach of clearing minefields by walking through the field looking for each mine one at a time has been superseded by the sue of large earth-moving equipment fitted with heavy rollers and flails. These clear huge swaths of the minefield at once, and are unlikely to miss anything. Likewise, a cross-functional test touches lots of areas of the system at one time. The key assumption behind cross-functional tests is that you won't run into too many bugs. If you do, then you'll not get through any single test, as you'll get blocked partway into your cross-functional voyage through the system.

Sourced from Surveying Test Strategies: A Guide to Smart Selection and Blending by Rex Black, Testing Experience Issue 2-2008. Slight modification has been made to the original text.