Combinatorial Coverage Measurement

Abstract

 

Combinatorial testing applies factor covering arrays [6, 7] to test all t-way combinations of input or configuration state space. In some testing situations, it is not practical to use covering arrays, but any set of tests with n parameters covers at least some proportion of t-way combinations up to t ≤ n. This report describes measures of combinatorial coverage that can be used in evaluating the degree of t-way coverage of any test suite, regardless of whether it was initially constructed for combinatorial coverage.

 

Introduction

Because testing all possible combinations of input values is nearly always intractable, combinatorial testing using factor covering arrays is a reasonable alternative [8, 9, 10]. But it is not always practical to re-design an organization’s testing procedures to use tests based on covering arrays. Testing procedures often develop over time, and employees have extensive experience with a particular approach. Units of the organization may be structured around established, documented test procedures. This is particularly true in organizations that must test according to contractual requirements, or relevant standards. And because much software assurance involves testing applications that have been modified to meet new specifications, an extensive library of legacy tests may exist. The organization can save time and money by re-using existing tests, which may not have been developed as factor covering arrays.

Short of creating new test suites from scratch, one approach to obtaining the advantages of combinatorial testing is to measure the combinatorial coverage of existing tests, then supplement as needed. Depending on the budget and criticality of the software, 2-way through 5-way or 6-way testing may be appropriate. Building covering arrays for some specified level of t is one way to provide t-way coverage. However, many large test suites naturally cover a high percentage of t-way combinations. If an existing test suite covers almost all 3-way combinations, for example, then it may be sufficient for the level of assurance that is required. Determining the level of input or configuration state space coverage can also help in understanding the degree of risk that remains after testing. If 90% - 100% of the relevant state space has been covered, then risk is likely to be smaller than would remain after testing that covers a much smaller portion of the state space. This report describes measures of combinatorial coverage that can be helpful in evaluating the degree of t-way coverage of any test suite, regardless of whether it was initially constructed for combinatorial coverage.

Authors:
D. Richard Kuhn/Raghu N. Kacker/Yu LeiYu Lei

Source and Full Report:
http://nvlpubs.nist.gov/nistpubs/ir/2012/NIST.IR.7878.pdf