According to the online Cambridge Advanced Learner's Dictionary, strategy means a detailed plan for achieving success in situations such as war, politics, business, industry or sport, or the skill of planning for such situations. Therefore, test strategy could be interpreted as a detailed plan for achieving success in testing something. For software development, the something is usually a software or system or any related work products.
Types of Test Strategy
In Rex Black's article "Surveying Test Strategies: A Guide to Smart Selection and Blending" which was published in the Testing Experience Issue 2-2008, he has summarized test strategies into 7 categories:
Although Regression Tests is more of a type of testing, it is so important and different that it deserves its own set of strategies. Please refer to the other postings for more information on these strategies.
Showing posts with label IT: Testing: Strategy. Show all posts
Showing posts with label IT: Testing: Strategy. Show all posts
Tuesday, September 23, 2008
Monday, September 22, 2008
Test Strategy: Regression Tests
Regression test is a critical element in any testing strategy. It aims at finding mis-behaviour of a previously correct function, attribute or feature. These bugs basically fall into 3 basic types:
Repeat All Strategy
With a repeat all strategy, a brute force strategy, is to repeat all tests that were previously run. This assumes that a set of regression tests was developed and well aligned with required quality. Usually, this strategy would be taken if risk assessment gave solid evidence to its need and sufficient time and resources are available. If all previous run tests were executed, we should be able to find the important regression bugs. Realistically, test automation is probably the only feasible mean to a repeat all strategy due to the large number of tests usually in the regression test suite. Even test automation is employed, there are always some manual tests that cannot be automated.
Repeat Some Strategy
For one reason or another, it is often not possible to repeat all regression tests. So, a subset of the overall regression tests must be selected to repeat. There are three major techniques to do so.
1. Traceability
Briefly, traceability is where tests are related to behavioural descriptions of the system like requirement specification elements, design specification elements or quality risks. We can look at what requirement, design element or quality risk is affected by the change, trace back to the associated tests and select those tests for re-execution.
2. Change Analysis
In this case, one looks at structural descriptions of the system to figure out how change effects could ripple through the system. Unless there is an in-depth understanding of programming and the system's design, help from designers and developers would be needed.
3. Quality Risk Analysis
With traceability and change analysis, technical risk, where bugs are likely to be found, is used to decide what to re-test. However, the quality risk analysis should be revisit to determine areas of high business risk. Even if bugs are unlikely, areas with high business risk should be re-tested.
Since not all tests were re-run, it is possible to miss some regression bugs if they occur in un-anticipated areas. So, it is advisable to use cross-functional tests to get a lot of accidental regression testing. The analogy is clearing a minefield. The old fashioned approach of clearing minefields by walking through the field looking for each mine one at a time has been superseded by the sue of large earth-moving equipment fitted with heavy rollers and flails. These clear huge swaths of the minefield at once, and are unlikely to miss anything. Likewise, a cross-functional test touches lots of areas of the system at one time. The key assumption behind cross-functional tests is that you won't run into too many bugs. If you do, then you'll not get through any single test, as you'll get blocked partway into your cross-functional voyage through the system.
Sourced from Surveying Test Strategies: A Guide to Smart Selection and Blending by Rex Black, Testing Experience Issue 2-2008. Slight modification has been made to the original text.
- change exposes an existing bug;
- change breaks something in the same part of the system; and
- local change breaks something in another part of the system.
Repeat All Strategy
With a repeat all strategy, a brute force strategy, is to repeat all tests that were previously run. This assumes that a set of regression tests was developed and well aligned with required quality. Usually, this strategy would be taken if risk assessment gave solid evidence to its need and sufficient time and resources are available. If all previous run tests were executed, we should be able to find the important regression bugs. Realistically, test automation is probably the only feasible mean to a repeat all strategy due to the large number of tests usually in the regression test suite. Even test automation is employed, there are always some manual tests that cannot be automated.
Repeat Some Strategy
For one reason or another, it is often not possible to repeat all regression tests. So, a subset of the overall regression tests must be selected to repeat. There are three major techniques to do so.
1. Traceability
Briefly, traceability is where tests are related to behavioural descriptions of the system like requirement specification elements, design specification elements or quality risks. We can look at what requirement, design element or quality risk is affected by the change, trace back to the associated tests and select those tests for re-execution.
2. Change Analysis
In this case, one looks at structural descriptions of the system to figure out how change effects could ripple through the system. Unless there is an in-depth understanding of programming and the system's design, help from designers and developers would be needed.
3. Quality Risk Analysis
With traceability and change analysis, technical risk, where bugs are likely to be found, is used to decide what to re-test. However, the quality risk analysis should be revisit to determine areas of high business risk. Even if bugs are unlikely, areas with high business risk should be re-tested.
Since not all tests were re-run, it is possible to miss some regression bugs if they occur in un-anticipated areas. So, it is advisable to use cross-functional tests to get a lot of accidental regression testing. The analogy is clearing a minefield. The old fashioned approach of clearing minefields by walking through the field looking for each mine one at a time has been superseded by the sue of large earth-moving equipment fitted with heavy rollers and flails. These clear huge swaths of the minefield at once, and are unlikely to miss anything. Likewise, a cross-functional test touches lots of areas of the system at one time. The key assumption behind cross-functional tests is that you won't run into too many bugs. If you do, then you'll not get through any single test, as you'll get blocked partway into your cross-functional voyage through the system.
Sourced from Surveying Test Strategies: A Guide to Smart Selection and Blending by Rex Black, Testing Experience Issue 2-2008. Slight modification has been made to the original text.
Test Strategy: Philosophical Approaches
Philosophical test strategies start with a philosophy or belief about testing.
Exhaustive Strategy
With an exhaustive test strategy, one assumes that everything and anything can and will have bugs. It is decided that the possibility of missing bugs is unacceptable and that management will support a considerable effort to find all the bugs. Attempt would then be undertaken to test extensively across the functionality, the quality risks, the requirements and whatever else one can find to cover. The essence of this strategy is captured in an old tester's joke, derived from the catchphrase on the back of the US currency: "In God we trust... all others we test."
Shotgun Strategy
With a shotgun strategy, one also assumes that everything and anything can and will be buggy. However, one accepts that it is infeasible to test everything. Since there is a lack of idea on where to find bugs, we test wherever and whatever comes to mind. Test attempts are randomly distribute within the given resource and schedule boundaries, like pellets from a shotgun.
Externally Guided Strategy
With an externally guided test strategy, one accepts that it is infeasible to test everything, nor to know where the bugs are. However, we trust that other people might have a good idea of where the bugs are. Therefore, we ask for their guidance and test according to their direction, including asking them to help in determining if the observed results are correct. Common guidance include developer, user, technical or customer support, help desk staff, business analyst, sales people and marketing staff.
If the underlying philosophies and beliefs behind these strategies are correct, they can be appropriate. For example, testing weapons systems like nuclear missile guidance software clearly requires an exhaustive strategy. However, when applied in inappropriate situations, they could lead to dangerously misaligned test efforts.
Sourced from Surveying Test Strategies: A Guide to Smart Selection and Blending by Rex Black, Testing Experience Issue 2-2008. Slight modification has been made to the original text.
Exhaustive Strategy
With an exhaustive test strategy, one assumes that everything and anything can and will have bugs. It is decided that the possibility of missing bugs is unacceptable and that management will support a considerable effort to find all the bugs. Attempt would then be undertaken to test extensively across the functionality, the quality risks, the requirements and whatever else one can find to cover. The essence of this strategy is captured in an old tester's joke, derived from the catchphrase on the back of the US currency: "In God we trust... all others we test."
Shotgun Strategy
With a shotgun strategy, one also assumes that everything and anything can and will be buggy. However, one accepts that it is infeasible to test everything. Since there is a lack of idea on where to find bugs, we test wherever and whatever comes to mind. Test attempts are randomly distribute within the given resource and schedule boundaries, like pellets from a shotgun.
Externally Guided Strategy
With an externally guided test strategy, one accepts that it is infeasible to test everything, nor to know where the bugs are. However, we trust that other people might have a good idea of where the bugs are. Therefore, we ask for their guidance and test according to their direction, including asking them to help in determining if the observed results are correct. Common guidance include developer, user, technical or customer support, help desk staff, business analyst, sales people and marketing staff.
If the underlying philosophies and beliefs behind these strategies are correct, they can be appropriate. For example, testing weapons systems like nuclear missile guidance software clearly requires an exhaustive strategy. However, when applied in inappropriate situations, they could lead to dangerously misaligned test efforts.
Sourced from Surveying Test Strategies: A Guide to Smart Selection and Blending by Rex Black, Testing Experience Issue 2-2008. Slight modification has been made to the original text.
Test Strategy: Dynamic Approaches
Like the agile test strategies, dynamic test strategies minimize up front planning and test design, focusing on making the test execution period responsive to change and able to find as many bugs as possible.
Intuitive Strategy
With an intuitive test strategy, system is tested according to the collective experience, wisdom and gut instincts of the test team. Discussions about what to test, anecdotal evidence from past projects, and oral tradition are prime drivers.
Exploratory Strategy
With an exploratory test strategy, tester would simultaneously learn about the system's behaviour and design while tests are run and bugs are found. This approach would continuously refine the tests and adjust the focus of further testing based on the test results.
Bug Hunting Strategy
With a bug hunting strategy, tester uses bug profiles, taxonomies (classifications), and hunches (bug assumptions) to focus testing where he/she thinks the bugs are. The hunting metaphor is a good one for all these strategies, which are more alike than different. A critical success factor to both hunting and fishing: Hunt where the birds are and fish where the fish are. Likewise, this test strategy requires that tester be right about where he/she thinks the bugs are, often under conditions of schedule and personal pressure.
Dynamic test strategies value flexibility and finding bugs highly. They do not usually produce good information about coverage, systematically mitigate risks, or offer the opportunity to detect bugs early in the development lifecycle. They are certainly much better than no testing at all and, serve as an blended with analytical strategies, serve as an excellent check and balance that can catch gaps in the analytically designed tests.
Sourced from Surveying Test Strategies: A Guide to Smart Selection and Blending by Rex Black, Testing Experience Issue 2-2008. Slight modification has been made to the original text.
Intuitive Strategy
With an intuitive test strategy, system is tested according to the collective experience, wisdom and gut instincts of the test team. Discussions about what to test, anecdotal evidence from past projects, and oral tradition are prime drivers.
Exploratory Strategy
With an exploratory test strategy, tester would simultaneously learn about the system's behaviour and design while tests are run and bugs are found. This approach would continuously refine the tests and adjust the focus of further testing based on the test results.
Bug Hunting Strategy
With a bug hunting strategy, tester uses bug profiles, taxonomies (classifications), and hunches (bug assumptions) to focus testing where he/she thinks the bugs are. The hunting metaphor is a good one for all these strategies, which are more alike than different. A critical success factor to both hunting and fishing: Hunt where the birds are and fish where the fish are. Likewise, this test strategy requires that tester be right about where he/she thinks the bugs are, often under conditions of schedule and personal pressure.
Dynamic test strategies value flexibility and finding bugs highly. They do not usually produce good information about coverage, systematically mitigate risks, or offer the opportunity to detect bugs early in the development lifecycle. They are certainly much better than no testing at all and, serve as an blended with analytical strategies, serve as an excellent check and balance that can catch gaps in the analytically designed tests.
Sourced from Surveying Test Strategies: A Guide to Smart Selection and Blending by Rex Black, Testing Experience Issue 2-2008. Slight modification has been made to the original text.
Test Strategy: Process Oriented Approaches
Process oriented test strategies take the methodical approach one step further by regulating the test process.
Standardized Strategy
With a standardized test strategy, tester follows official or recognized standards. For example, the IEEE 829 standard for test documentation, created by a volunteer standards committee of the non-profile Institute for Electronics and Electrical Engineers, is used by some organizations to ensure regularity and completeness of all test documents. Such standardization can help to make the test process transparent and comprehensible to developers, managers, business analysts and other non-testers. However, one must take care not to introduce excessive or obstructive levels of bureaucracy or paperwork.
Agile Strategy
One increasingly popular test strategy, the agile test strategy, has arisen from the programming side of software development. Here, tester follows lightweight processes, mostly focused on technical risk (likely bugs). A heavy emphasis is placed on automated unit testing, customer acceptance testing and being able to respond to late changes without excessive costs. These strategies are tailored for small teams on short projects with immediate access to the users. Large, long, geographically distributed, or high risk projects are likely to find that the strategy does not scale.
Automated Strategies
The topic of automated unit testing brings us to a group of test strategies that rely heavily on automation. One such strategy is an automated random test strategy, where a large amount of random input data are sent to the system. Another such strategy is an automated functional test strategy, where tester tests system functionality using repeatable scripts. Either strategy might involve an automated load, performance or reliability testing element. Very obvious, these strategies rely heavily on the ability to effectively automate most of the testing that needs to be done.
Sourced from Surveying Test Strategies: A Guide to Smart Selection and Blending by Rex Black, Testing Experience Issue 2-2008. Slight modification has been made to the original text.
Standardized Strategy
With a standardized test strategy, tester follows official or recognized standards. For example, the IEEE 829 standard for test documentation, created by a volunteer standards committee of the non-profile Institute for Electronics and Electrical Engineers, is used by some organizations to ensure regularity and completeness of all test documents. Such standardization can help to make the test process transparent and comprehensible to developers, managers, business analysts and other non-testers. However, one must take care not to introduce excessive or obstructive levels of bureaucracy or paperwork.
Agile Strategy
One increasingly popular test strategy, the agile test strategy, has arisen from the programming side of software development. Here, tester follows lightweight processes, mostly focused on technical risk (likely bugs). A heavy emphasis is placed on automated unit testing, customer acceptance testing and being able to respond to late changes without excessive costs. These strategies are tailored for small teams on short projects with immediate access to the users. Large, long, geographically distributed, or high risk projects are likely to find that the strategy does not scale.
Automated Strategies
The topic of automated unit testing brings us to a group of test strategies that rely heavily on automation. One such strategy is an automated random test strategy, where a large amount of random input data are sent to the system. Another such strategy is an automated functional test strategy, where tester tests system functionality using repeatable scripts. Either strategy might involve an automated load, performance or reliability testing element. Very obvious, these strategies rely heavily on the ability to effectively automate most of the testing that needs to be done.
Sourced from Surveying Test Strategies: A Guide to Smart Selection and Blending by Rex Black, Testing Experience Issue 2-2008. Slight modification has been made to the original text.
Test Strategy: Methodical Approaches
Methodical test strategies rely on some relatively informal but orderly and predictable approaches to figure out where to test.
Learning Based Strategy
With a learning based strategy, tester uses checklists that is developed over time to guide testing. These checklists are developed based on where bugs were found before, good ideas learnt from others or any other sources.
Function Based Strategy
With a function based strategy, tester identifies and then test each and every function of the system, often one at a time.
State Based Strategy
Similar to the function based strategy, with a state base strategy, tester would identify and test every state and every possible state transition that can occur in the system.
Quality Based Strategy
With a quality based strategy, tester uses a quality hierarchy such as ISO-9126 to identify and test the important characteristics of the system. For example, some groups in Hewlett-Packard use functionality, localization, usability, reliability, performance and scalability. IBM uses capability, usability, performance, reliability, installability, maintainability, documentation and operability.
With a methodical test strategy, tester follows these standard inventories of test objectives. These strategies can be quick and effective for systems that remain relatively stable or systems which are similar to those tested before. Significant changes might render these strategies temporarily ineffective until test objectives are adjusted to the new system or organizational realities.
Sourced from Surveying Test Strategies: A Guide to Smart Selection and Blending by Rex Black, Testing Experience Issue 2-2008. Slight modification has been made to original text.
Learning Based Strategy
With a learning based strategy, tester uses checklists that is developed over time to guide testing. These checklists are developed based on where bugs were found before, good ideas learnt from others or any other sources.
Function Based Strategy
With a function based strategy, tester identifies and then test each and every function of the system, often one at a time.
State Based Strategy
Similar to the function based strategy, with a state base strategy, tester would identify and test every state and every possible state transition that can occur in the system.
Quality Based Strategy
With a quality based strategy, tester uses a quality hierarchy such as ISO-9126 to identify and test the important characteristics of the system. For example, some groups in Hewlett-Packard use functionality, localization, usability, reliability, performance and scalability. IBM uses capability, usability, performance, reliability, installability, maintainability, documentation and operability.
With a methodical test strategy, tester follows these standard inventories of test objectives. These strategies can be quick and effective for systems that remain relatively stable or systems which are similar to those tested before. Significant changes might render these strategies temporarily ineffective until test objectives are adjusted to the new system or organizational realities.
Sourced from Surveying Test Strategies: A Guide to Smart Selection and Blending by Rex Black, Testing Experience Issue 2-2008. Slight modification has been made to original text.
Test Strategy: Model Based Approaches
Model based strategies develop models of how the system should behave or function.
Scenario Based Strategy
With a scenario based strategy, tester tests according to real world scenarios which should span sufficiently the system's functionality.
Use Case Based Strategy
In the object-oriented world, a close relative to the scenario based strategy is the use case based strategy, where tester relies on a type of object-oriented design documents known as use cases. These use cases are models of how users, customers and other stakeholders supposed to use the system and how it should work under those conditions. Besides, these use cases can be translated into test cases.
Domain Based Strategy
With a domain based strategy, tester analyzes different domains of input data accepted by the system, data processing performed by the system and output data produced by the system. Domains are classifications based on similarities identified in inputs, processing or outputs. Based on these domains, the best test cases in each domain could be picked according to the likelihood of bugs, prevalent usage, deployed environments or even all three of these factors.
With a model based strategy, tests are designed, developed and executed to cover models that has been built. These strategies are useful to the extent that essential aspects of the system were captured in the model. Of course, these strategies reply on the ability of tester to develop good models. These strategies would break down when the models cannot or the tester does not capture all of the essential or potentially problematic aspects of the system.
Sourced from Surveying Test Strategies: A Guide to Smart Selection and Blending by Rex Black, Testing Experience Issue 2-2008. Slight modification has been made to the original text.
Scenario Based Strategy
With a scenario based strategy, tester tests according to real world scenarios which should span sufficiently the system's functionality.
Use Case Based Strategy
In the object-oriented world, a close relative to the scenario based strategy is the use case based strategy, where tester relies on a type of object-oriented design documents known as use cases. These use cases are models of how users, customers and other stakeholders supposed to use the system and how it should work under those conditions. Besides, these use cases can be translated into test cases.
Domain Based Strategy
With a domain based strategy, tester analyzes different domains of input data accepted by the system, data processing performed by the system and output data produced by the system. Domains are classifications based on similarities identified in inputs, processing or outputs. Based on these domains, the best test cases in each domain could be picked according to the likelihood of bugs, prevalent usage, deployed environments or even all three of these factors.
With a model based strategy, tests are designed, developed and executed to cover models that has been built. These strategies are useful to the extent that essential aspects of the system were captured in the model. Of course, these strategies reply on the ability of tester to develop good models. These strategies would break down when the models cannot or the tester does not capture all of the essential or potentially problematic aspects of the system.
Sourced from Surveying Test Strategies: A Guide to Smart Selection and Blending by Rex Black, Testing Experience Issue 2-2008. Slight modification has been made to the original text.
Test Strategy: Analytical Approaches
Analytical test strategies start with analysis as its foundation.
Object Guided Strategy
With an object guided strategy, tester looks at requirements, design and/or implementation objects to determine testing focus. These objects can include requirement specifications, design specifications, use cases, UML diagrams, entity-relationship (E-R) diagrams and database schema. This approach relies on extensive documentation, and breaks down when documentation is not available.
Risk Based Strategy
With a risk based strategy, tester uses formal or informal techniques to assess and prioritize quality risks. Various available sources of information can be used. As with the object guided strategy, tester should also draw on the insights of cross functional project team members and other stakeholders. Adjust the tests and the extent of testing according to the risk priority levels. Unlike the object guided variant, this approach can handle situations where there is little or no project documentation.
Fully Informed Strategy
With a fully informed strategy, tester starts with the object guided or risk based strategy, but take the analysis further. Study the system, the usage profiles, the target configurations and as much other data as one can find. Design, develop and execute tests based on the broad, deep knowledge gained in analysis. This approach is great if there is adequate time to research the system.
The items being analyzed are sometimes called the test basis. The results of the analysis guide the entire testing, often through some form of coverage analysis during test design, development, execution and results reporting. These strategies tend to be thorough, good at mitigating quality risks and finding bugs. However, they do require an up-front investment of time.
Sourced from Surveying Test Strategies: A Guide to Smart Selection and Blending by Rex Black, Testing Experience Issue 2-2008. Slight modification has been made to the original text.
Object Guided Strategy
With an object guided strategy, tester looks at requirements, design and/or implementation objects to determine testing focus. These objects can include requirement specifications, design specifications, use cases, UML diagrams, entity-relationship (E-R) diagrams and database schema. This approach relies on extensive documentation, and breaks down when documentation is not available.
Risk Based Strategy
With a risk based strategy, tester uses formal or informal techniques to assess and prioritize quality risks. Various available sources of information can be used. As with the object guided strategy, tester should also draw on the insights of cross functional project team members and other stakeholders. Adjust the tests and the extent of testing according to the risk priority levels. Unlike the object guided variant, this approach can handle situations where there is little or no project documentation.
Fully Informed Strategy
With a fully informed strategy, tester starts with the object guided or risk based strategy, but take the analysis further. Study the system, the usage profiles, the target configurations and as much other data as one can find. Design, develop and execute tests based on the broad, deep knowledge gained in analysis. This approach is great if there is adequate time to research the system.
The items being analyzed are sometimes called the test basis. The results of the analysis guide the entire testing, often through some form of coverage analysis during test design, development, execution and results reporting. These strategies tend to be thorough, good at mitigating quality risks and finding bugs. However, they do require an up-front investment of time.
Sourced from Surveying Test Strategies: A Guide to Smart Selection and Blending by Rex Black, Testing Experience Issue 2-2008. Slight modification has been made to the original text.
Subscribe to:
Posts (Atom)