Abstract
Combinatorial testing applies factor covering arrays [6, 7] to
test all t-way combinations of input or configuration state space. In
some testing situations, it is not practical to use covering arrays, but any
set of tests with n parameters covers at least some proportion of t-way
combinations up to t ≤ n. This report describes measures of
combinatorial coverage that can be used in evaluating the degree of t-way
coverage of any test suite, regardless of whether it was initially constructed
for combinatorial coverage.
Introduction
Because testing all possible
combinations of input values is nearly always intractable, combinatorial
testing using factor covering arrays is a reasonable alternative [8, 9, 10].
But it is not always practical to re-design an organization’s testing procedures
to use tests based on covering arrays. Testing procedures often develop over
time, and employees have extensive experience with a particular approach. Units
of the organization may be structured around established, documented test
procedures. This is particularly true in organizations that must test according
to contractual requirements, or relevant standards. And because much software
assurance involves testing applications that have been modified to meet new
specifications, an extensive library of legacy tests may exist. The
organization can save time and money by re-using existing tests, which may not
have been developed as factor covering arrays.
Short of creating new test suites
from scratch, one approach to obtaining the advantages of combinatorial testing
is to measure the combinatorial coverage of existing tests, then supplement as
needed. Depending on the budget and criticality of the software, 2-way through
5-way or 6-way testing may be appropriate. Building covering arrays for some
specified level of t is one way to provide t-way coverage.
However, many large test suites naturally cover a high percentage of t-way
combinations. If an existing test suite covers almost all 3-way combinations,
for example, then it may be sufficient for the level of assurance that is
required. Determining the level of input or configuration state space coverage
can also help in understanding the degree of risk that remains after testing.
If 90% - 100% of the relevant state space has been covered, then risk is likely
to be smaller than would remain after testing that covers a much smaller
portion of the state space. This report describes measures of combinatorial
coverage that can be helpful in evaluating the degree of t-way coverage
of any test suite, regardless of whether it was initially constructed for
combinatorial coverage.
Authors:
D. Richard Kuhn/Raghu N. Kacker/Yu LeiYu Lei
Source and Full Report:
http://nvlpubs.nist.gov/nistpubs/ir/2012/NIST.IR.7878.pdf
0 yorum:
Yorum Gönder