Preface, |
|
xiii | |
Authors, |
|
xvii | |
Note of Appreciation, |
|
xix | |
Nomenclature, |
|
xxi | |
Chapter 1 Combinatorial Methods in Testing |
|
1 | (20) |
|
1.1 Software Failures And The Interaction Rule |
|
|
1 | (7) |
|
1.2 Two Forms Of Combinatorial Testing |
|
|
8 | (3) |
|
1.2.1 Configuration Testing |
|
|
9 | (1) |
|
|
10 | (1) |
|
|
11 | (4) |
|
1.3.1 Covering Array Definition |
|
|
12 | (1) |
|
1.3.2 Size Of Covering Arrays |
|
|
13 | (2) |
|
1.4 The Test Oracle Problem |
|
|
15 | (1) |
|
1.5 Quick Start: How To Use The Basics Of Combinatorial Methods Right Away |
|
|
16 | (1) |
|
|
17 | (1) |
|
|
18 | (3) |
Chapter 2 Combinatorial Testing Applied |
|
21 | (16) |
|
2.1 Document Object Model |
|
|
21 | (9) |
|
|
|
|
|
|
|
2.1.1 Constructing Tests For DOM Events |
|
|
22 | (3) |
|
2.1.2 Combinatorial Testing Approach |
|
|
25 | (1) |
|
|
25 | (4) |
|
2.1.4 Cost And Practical Considerations |
|
|
29 | (1) |
|
2.2 Rich Web Applications |
|
|
30 | (5) |
|
|
2.2.1 Systematic Variable Detection In Semantic URLs |
|
|
31 | (1) |
|
2.2.2 Javascript Fault Classification And Identification |
|
|
31 | (2) |
|
|
33 | (2) |
|
|
35 | (2) |
Chapter 3 Configuration Testing |
|
37 | (14) |
|
3.1 Runtime Environment Configurations |
|
|
37 | (2) |
|
3.2 Highly Configurable Systems And Software Product Lines |
|
|
39 | (5) |
|
3.3 Invalid Combinations And Constraints |
|
|
44 | (4) |
|
3.3.1 Constraints Among Parameter Values |
|
|
44 | (2) |
|
3.3.2 Constraints Among Parameters |
|
|
46 | (2) |
|
3.4 Cost And Practical Considerations |
|
|
48 | (1) |
|
|
49 | (1) |
|
|
50 | (1) |
Chapter 4 Input Testing |
|
51 | (20) |
|
4.1 Partitioning The Input Space |
|
|
51 | (4) |
|
4.2 Input Variables Versus Test Parameters |
|
|
55 | (2) |
|
4.3 Fault Type And Detectability |
|
|
57 | (4) |
|
4.4 Building Tests To Match An Operational Profile |
|
|
61 | (3) |
|
4.5 Scaling Considerations |
|
|
64 | (2) |
|
4.6 Cost And Practical Considerations |
|
|
66 | (1) |
|
|
67 | (1) |
|
|
68 | (3) |
Chapter 5 Test Parameter Analysis |
|
71 | (42) |
|
|
5.1 What Should Be Included As A Test Parameter |
|
|
72 | (2) |
|
5.2 Combination Anomalies |
|
|
74 | (2) |
|
5.3 Classification Tree Method |
|
|
76 | (5) |
|
|
81 | (22) |
|
5.4.1 Flexible Manufacturing System Example |
|
|
81 | (8) |
|
|
89 | (5) |
|
|
94 | (9) |
|
5.5 Selecting The System Under Test |
|
|
103 | (3) |
|
5.6 Combinatorial Testing And Boundary Value Analysis |
|
|
106 | (5) |
|
|
111 | (1) |
|
|
111 | (2) |
Chapter 6 Managing System State |
|
113 | (30) |
|
|
6.1 Test Factor Partitions With State |
|
|
114 | (5) |
|
6.1.1 Partitions For Expected Results |
|
|
115 | (1) |
|
6.1.2 Partitions With Constraints |
|
|
116 | (1) |
|
6.1.3 Direct Product Block Notation |
|
|
116 | (3) |
|
6.2 Test Factor Simplifications |
|
|
119 | (3) |
|
6.2.1 All The Same Factor Value |
|
|
119 | (1) |
|
6.2.2 All Different Factor Values |
|
|
119 | (1) |
|
6.2.3 Functionally Dependent Factor Values |
|
|
119 | (2) |
|
6.2.4 Hybrid Factor Values |
|
|
121 | (1) |
|
6.3 Sequence Unit Replay Model |
|
|
122 | (4) |
|
6.4 Single Region State Models |
|
|
126 | (7) |
|
6.5 Multiple Region State Models |
|
|
133 | (4) |
|
|
137 | (3) |
|
|
140 | (3) |
Chapter 7 Measuring Combinatorial Coverage |
|
143 | (20) |
|
7.1 Software Test Coverage |
|
|
144 | (1) |
|
7.2 Combinatorial Coverage |
|
|
145 | (7) |
|
7.2.1 Simple t-Way Combination Coverage |
|
|
146 | (1) |
|
|
147 | (1) |
|
|
148 | (1) |
|
7.2.4 Variable-Value Configuration Coverage |
|
|
149 | (3) |
|
7.3 Using Combinatorial Coverage |
|
|
152 | (4) |
|
7.4 Cost And Practical Considerations |
|
|
156 | (4) |
|
7.5 Analysis Of (t+1)-Way Coverage |
|
|
160 | (1) |
|
|
161 | (1) |
|
|
161 | (2) |
Chapter 8 Test Suite Prioritization By Combinatorial, Coverage |
|
163 | (16) |
|
|
|
8.1 Combinatorial Coverage For Test Suite Prioritization |
|
|
163 | (3) |
|
|
166 | (1) |
|
8.3 Prioritization Criteria |
|
|
167 | (2) |
|
8.4 Review Of Empirical Studies |
|
|
169 | (4) |
|
8.4.1 Subject Applications |
|
|
169 | (1) |
|
8.4.2 Prioritization Criteria |
|
|
169 | (4) |
|
|
170 | (1) |
|
|
170 | (1) |
|
8.4.2.3 Evaluation Metric |
|
|
171 | (1) |
|
|
171 | (2) |
|
8.5 Tool: Combinatorial-Based Prioritization For User-Session-based Testing |
|
|
173 | (1) |
|
8.5.1 Apache Logging Module |
|
|
173 | (1) |
|
8.5.2 Creating A User Session-Based Test Suite From Usage Logs Using CPUT |
|
|
173 | (1) |
|
8.5.3 Prioritizing And Reducing Test Cases |
|
|
173 | (1) |
|
8.6 Other Approaches To Test Suite Prioritization Using Combinatorial Interactions |
|
|
174 | (1) |
|
8.7 Cost And Practical Considerations |
|
|
175 | (1) |
|
|
176 | (1) |
|
|
176 | (3) |
Chapter 9 Combinatorial Testing And Random Test Generation |
|
179 | (14) |
|
9.1 Coverage Of Random Tests |
|
|
180 | (4) |
|
9.2 Adaptive Random Testing |
|
|
184 | (2) |
|
9.3 Tradeoffs: Covering Arrays And Random Generation |
|
|
186 | (3) |
|
9.4 Cost And Practical Considerations |
|
|
189 | (1) |
|
|
190 | (1) |
|
|
191 | (2) |
Chapter 10 Sequence-Covering Arrays |
|
193 | (10) |
|
10.1 Sequence-Covering Array Definition |
|
|
193 | (2) |
|
10.2 Size And Construction Of Sequence-Covering Arrays |
|
|
195 | (3) |
|
10.2.1 Generalized t-Way Sequence Covering |
|
|
197 | (1) |
|
10.2.2 Algorithm Analysis |
|
|
197 | (1) |
|
10.3 Using Sequence-Covering Arrays |
|
|
198 | (1) |
|
10.4 Cost And Practical Considerations |
|
|
199 | (1) |
|
|
199 | (3) |
|
|
202 | (1) |
Chapter 11 Assertion-Based Testing |
|
203 | (10) |
|
11.1 Basic Assertions For Testing |
|
|
204 | (4) |
|
11.2 Stronger Assertion-Based Testing |
|
|
208 | (1) |
|
11.3 Cost And Practical Considerations |
|
|
209 | (1) |
|
|
210 | (1) |
|
|
210 | (3) |
Chapter 12 Model-Based Testing |
|
213 | (14) |
|
|
214 | (1) |
|
12.2 Access Control System Example |
|
|
215 | (1) |
|
|
216 | (2) |
|
12.4 Integrating Combinatorial Tests Into The Model |
|
|
218 | (4) |
|
12.5 Generating Tests From Counterexamples |
|
|
222 | (2) |
|
12.6 Cost And Practical Considerations |
|
|
224 | (1) |
|
|
225 | (1) |
|
|
225 | (2) |
Chapter 13 Fault Localization |
|
227 | (10) |
|
13.1 Fault Localization Process |
|
|
228 | (3) |
|
13.1.1 Analyzing Combinations |
|
|
229 | (1) |
|
13.1.2 New Test Generation |
|
|
230 | (1) |
|
|
230 | (1) |
|
|
230 | (1) |
|
13.2 Locating Faults: Example |
|
|
231 | (4) |
|
13.2.1 Generating New Tests |
|
|
234 | (1) |
|
13.3 Cost And Practical Considerations |
|
|
235 | (1) |
|
|
236 | (1) |
|
|
236 | (1) |
Chapter 14 Evolution From Design Of Experiments |
|
237 | (10) |
|
|
237 | (2) |
|
14.2 Pairwise (Two-Way) Testing Of Software Systems |
|
|
239 | (6) |
|
14.3 Combinatorial t-Way Testing Of Software Systems |
|
|
245 | (1) |
|
|
246 | (1) |
Chapter 15 Algorithms For Covering Array Construction |
|
247 | (14) |
|
|
|
|
247 | (2) |
|
15.1.1 Computational Approaches |
|
|
247 | (1) |
|
15.1.2 Algebraic Approaches |
|
|
248 | (1) |
|
|
249 | (3) |
|
|
252 | (3) |
|
15.4 Cost And Practical Considerations |
|
|
255 | (3) |
|
15.4.1 Constraint Handling |
|
|
255 | (1) |
|
15.4.2 Mixed-Strength Covering Arrays |
|
|
256 | (1) |
|
15.4.3 Extension Of An Existing Test Set |
|
|
257 | (1) |
|
|
258 | (1) |
|
|
258 | (3) |
Appendix A Mathematics Review, |
|
261 | (6) |
|
|
261 | (4) |
|
A.1.1 Permutations And Combinations |
|
|
261 | (1) |
|
|
262 | (1) |
|
|
263 | (1) |
|
A.1.4 Number Of Tests Required |
|
|
264 | (1) |
|
|
265 | (12) |
|
A.2.1 Expression Operators |
|
|
265 | (1) |
|
A.2.2 Combining Operators |
|
|
266 | (1) |
Appendix B Empirical Data On Software Failures, |
|
267 | (6) |
Appendix C Resources For Combinatorial Testing, |
|
273 | (4) |
Appendix D Test Tools, |
|
277 | (16) |
|
|
278 | (15) |
|
|
278 | (1) |
|
D.1.1.1 t-Way Test Set Generation |
|
|
278 | (1) |
|
|
278 | (1) |
|
D.1.1.3 Constraint Support |
|
|
279 | (1) |
|
D.1.1.4 Coverage Verification |
|
|
279 | (1) |
|
D.1.2 Command Line Interface |
|
|
279 | (3) |
|
|
282 | (11) |
|
D.1.3.1 Create New System |
|
|
284 | (4) |
|
|
288 | (1) |
|
|
289 | (2) |
|
D.1.3.4 Save/Save As/Open System |
|
|
291 | (1) |
|
D.1.3.5 Import/Export Test Set |
|
|
291 | (1) |
|
D.1.3.6 Verify t-Way Coverage |
|
|
292 | (1) |
References, |
|
293 | (16) |
Index, |
|
309 | |