Preface |
|
xvii | |
|
|
1 | (60) |
|
|
3 | (58) |
|
|
3 | (6) |
|
|
4 | (1) |
|
|
5 | (1) |
|
|
5 | (1) |
|
1.1.4 Book structure and contents |
|
|
6 | (2) |
|
1.1.5 Scope of application |
|
|
8 | (1) |
|
1.1.6 Terminology and notation |
|
|
9 | (1) |
|
1.2 VVT Systems and Process |
|
|
9 | (23) |
|
1.2.1 Introduction---VVT systems and process |
|
|
9 | (1) |
|
|
10 | (2) |
|
1.2.3 VVT concepts and definition |
|
|
12 | (7) |
|
1.2.4 The fundamental VVT dilemma |
|
|
19 | (1) |
|
1.2.5 Modeling systems and VVT lifecycle |
|
|
20 | (4) |
|
1.2.6 Modeling VVT and risks as cost and time drivers |
|
|
24 | (8) |
|
1.3 Canonical Systems VVT Paradigm |
|
|
32 | (7) |
|
1.3.1 Introduction---Canonical systems VVT paradigm |
|
|
32 | (2) |
|
1.3.2 Phases of the system lifecycle |
|
|
34 | (3) |
|
1.3.3 Views of the system |
|
|
37 | (2) |
|
1.3.4 VVT aspects of the system |
|
|
39 | (1) |
|
1.4 Methodology Application |
|
|
39 | (17) |
|
|
39 | (1) |
|
1.4.2 VVT methodology overview |
|
|
40 | (3) |
|
|
43 | (7) |
|
|
50 | (6) |
|
|
56 | (5) |
|
Part II VVT Activities and Methods |
|
|
61 | (434) |
|
2 System VVT Activities: Development |
|
|
63 | (90) |
|
|
63 | (2) |
|
2.1.1 Systems development lifecycle phases and VVT activities |
|
|
63 | (1) |
|
2.1.2 VVT activity aspects |
|
|
64 | (1) |
|
2.1.3 VVT activity format |
|
|
65 | (1) |
|
2.2 VVT Activities during Definition |
|
|
65 | (15) |
|
2.2.1 Generate Requirements Verification Matrix (RVM) |
|
|
65 | (2) |
|
2.2.2 Generate VVT Management Plan (VVT-MP) |
|
|
67 | (2) |
|
2.2.3 Assess the Request For Proposal (RFP) document |
|
|
69 | (2) |
|
2.2.4 Assess System Requirements Specification (SysRS) |
|
|
71 | (1) |
|
2.2.5 Assess project Risk Management Plan (RMP) |
|
|
72 | (2) |
|
2.2.6 Assess System Safety Program Plan (SSPP) |
|
|
74 | (3) |
|
2.2.7 Participate in System Requirements Review (SysRR) |
|
|
77 | (1) |
|
2.2.8 Participate in System Engineering Management Plan (SEMP) review |
|
|
77 | (2) |
|
2.2.9 Conduct engineering peer review of the VVT-MP document |
|
|
79 | (1) |
|
2.3 VVT Activities during Design |
|
|
80 | (11) |
|
2.3.1 Optimize the VVT strategy |
|
|
80 | (3) |
|
2.3.2 Assess System/Subsystem Design Description (SSDD) |
|
|
83 | (2) |
|
2.3.3 Validate system design by means of virtual prototype |
|
|
85 | (1) |
|
2.3.4 Validate system design tools |
|
|
86 | (1) |
|
2.3.5 Assess system design for meeting future lifecycle needs |
|
|
87 | (3) |
|
2.3.6 Participate in the System Design Review (SysDR) |
|
|
90 | (1) |
|
2.4 VVT Activities during Implementation |
|
|
91 | (13) |
|
2.4.1 Preparing the test cycle for subsystems and components |
|
|
91 | (5) |
|
2.4.2 Assess suppliers' subsystems test documents |
|
|
96 | (1) |
|
2.4.3 Perform Acceptance Test Procedure---Subsystems/Enabling products |
|
|
97 | (3) |
|
2.4.4 Assess system performance by way of simulation |
|
|
100 | (2) |
|
2.4.5 Verify design versus implementation consistency |
|
|
102 | (1) |
|
2.4.6 Participate in Acceptance Test Review---Subsystems/Enabling products |
|
|
103 | (1) |
|
2.5 VVT Activities during Integration |
|
|
104 | (16) |
|
2.5.1 Develop System Integration Laboratory (SIL) |
|
|
104 | (2) |
|
2.5.2 Generate System Integration Test Plan (SysITP) |
|
|
106 | (2) |
|
2.5.3 Generate System Integration Test Description (SysITD) |
|
|
108 | (3) |
|
2.5.4 Validate supplied subsystems in a stand-alone configuration |
|
|
111 | (1) |
|
2.5.5 Perform components, subsystem, enabling products integration tests |
|
|
112 | (2) |
|
2.5.6 Generate System Integration Test Report (SysITR) |
|
|
114 | (2) |
|
2.5.7 Assess effectiveness of the system Built in Test (BIT) |
|
|
116 | (4) |
|
2.5.8 Conduct engineering peer review of the SysITR |
|
|
120 | (1) |
|
2.6 VVT Activities during Qualification |
|
|
120 | (29) |
|
2.6.1 Generate a qualification/acceptance System Test Plan (SysTP) |
|
|
121 | (2) |
|
2.6.2 Create qualification/acceptance System Test Description (SysTD) |
|
|
123 | (2) |
|
2.6.3 Perform virtual system testing by means of simulation |
|
|
125 | (1) |
|
2.6.4 Perform qualification testing/Acceptance Test Procedure (ATP)---System |
|
|
126 | (3) |
|
2.6.5 Generate qualification/acceptance System Test Report (SysTR) |
|
|
129 | (2) |
|
2.6.6 Assess system testability, maintainability and availability |
|
|
131 | (6) |
|
2.6.7 Perform environmental system testing |
|
|
137 | (3) |
|
2.6.8 Perform system Certification and Accreditation (C&A) |
|
|
140 | (4) |
|
2.6.9 Conduct Test Readiness Review (TRR) |
|
|
144 | (2) |
|
2.6.10 Conduct engineering peer review of development enabling products |
|
|
146 | (2) |
|
2.6.11 Conduct engineering peer review of program and project safety |
|
|
148 | (1) |
|
|
149 | (4) |
|
3 Systems VVT Activities: Post-Development |
|
|
153 | (70) |
|
|
153 | (1) |
|
3.2 VVT Activities during Production |
|
|
154 | (32) |
|
3.2.1 Participate in Functional Configuration Audit (FCA) |
|
|
154 | (3) |
|
3.2.2 Participate in Physical Configuration Audit (PCA) |
|
|
157 | (2) |
|
3.2.3 Plan system production VVT process |
|
|
159 | (2) |
|
3.2.4 Generate a First Article Inspection (FAI) procedure |
|
|
161 | (4) |
|
3.2.5 Validate the production-line test equipment |
|
|
165 | (1) |
|
3.2.6 Verify quality of incoming components and subsystems |
|
|
165 | (1) |
|
3.2.7 Perform First Article Inspection (FAI) |
|
|
166 | (1) |
|
3.2.8 Validate pre-production process |
|
|
167 | (1) |
|
3.2.9 Validate ongoing-production process |
|
|
168 | (2) |
|
3.2.10 Perform manufacturing quality control |
|
|
170 | (2) |
|
3.2.11 Verify the production operations strategy |
|
|
172 | (2) |
|
3.2.12 Verify marketing and production forecasting |
|
|
174 | (2) |
|
3.2.13 Verify aggregate production planning |
|
|
176 | (1) |
|
3.2.14 Verify inventory control operation |
|
|
177 | (3) |
|
3.2.15 Verify supply chain management |
|
|
180 | (1) |
|
3.2.16 Verify production control systems |
|
|
181 | (2) |
|
3.2.17 Verify production scheduling |
|
|
183 | (1) |
|
3.2.18 Participate in Production Readiness Review (PRR) |
|
|
184 | (2) |
|
3.3 VVT Activities during Use/Maintenance |
|
|
186 | (22) |
|
3.3.1 Develop VVT plan for system maintenance |
|
|
187 | (4) |
|
3.3.2 Verify the Integrated Logistics Support Plan (ILSP) |
|
|
191 | (9) |
|
3.3.3 Perform ongoing system maintenance testing |
|
|
200 | (4) |
|
3.3.4 Conduct engineering peer review on system maintenance process |
|
|
204 | (4) |
|
3.4 VVT Activities during Disposal |
|
|
208 | (13) |
|
3.4.1 Develop VVT plan for system disposal |
|
|
209 | (3) |
|
3.4.2 Assess the system disposal plan |
|
|
212 | (2) |
|
3.4.3 Assess system disposal strategies by means of simulation |
|
|
214 | (1) |
|
3.4.4 Assess on-going system disposal process |
|
|
215 | (4) |
|
3.4.5 Conduct engineering peer review to assess system disposal processes |
|
|
219 | (2) |
|
|
221 | (2) |
|
4 System VVT Methods: Non-Testing |
|
|
223 | (128) |
|
|
223 | (1) |
|
|
223 | (33) |
|
4.2.1 Requirements Verification Matrix (RVM) |
|
|
223 | (3) |
|
4.2.2 System Integration Laboratory (SIL) |
|
|
226 | (4) |
|
4.2.3 Hierarchical VVT optimization |
|
|
230 | (4) |
|
4.2.4 Defect management and tracking |
|
|
234 | (5) |
|
4.2.5 Classification Tree Method |
|
|
239 | (4) |
|
4.2.6 Design of Experiments (DOE) |
|
|
243 | (13) |
|
4.3 Perform VVT Activities |
|
|
256 | (56) |
|
4.3.1 VVT process planning |
|
|
256 | (6) |
|
4.3.2 Compare images and documents |
|
|
262 | (3) |
|
4.3.3 Requirements testability and quality |
|
|
265 | (7) |
|
4.3.4 System test simulation |
|
|
272 | (8) |
|
4.3.5 Failure mode effect analysis |
|
|
280 | (6) |
|
4.3.6 Anticipatory Failure Determination |
|
|
286 | (7) |
|
4.3.7 Model-based testing |
|
|
293 | (9) |
|
4.3.8 Robust design analysis |
|
|
302 | (10) |
|
4.4 Participate in Reviews |
|
|
312 | (34) |
|
4.4.1 Expert team reviews |
|
|
312 | (14) |
|
4.4.2 Formal technical reviews |
|
|
326 | (5) |
|
4.4.3 Group evaluation and decision |
|
|
331 | (15) |
|
|
346 | (5) |
|
5 Systems VVT Methods: Testing |
|
|
351 | (144) |
|
|
351 | (5) |
|
|
356 | (9) |
|
5.2.1 Component and code coverage testing |
|
|
356 | (4) |
|
|
360 | (5) |
|
5.3 Black Box---Basic Testing |
|
|
365 | (13) |
|
5.3.1 Boundary value testing |
|
|
365 | (2) |
|
5.3.2 Decision table testing |
|
|
367 | (1) |
|
5.3.3 Finite State Machine testing |
|
|
368 | (5) |
|
5.3.4 Human-system interface testing (HSI) |
|
|
373 | (5) |
|
5.4 Black Box---High-Volume Testing |
|
|
378 | (10) |
|
5.4.1 Automatic random testing |
|
|
378 | (3) |
|
5.4.2 Performance testing |
|
|
381 | (4) |
|
|
385 | (1) |
|
|
386 | (2) |
|
5.5 Black Box---Special Testing |
|
|
388 | (34) |
|
|
388 | (5) |
|
5.5.2 Security vulnerability testing |
|
|
393 | (9) |
|
5.5.3 Reliability testing |
|
|
402 | (8) |
|
5.5.4 Search-based testing |
|
|
410 | (8) |
|
|
418 | (4) |
|
5.6 Black Box---Environment Testing |
|
|
422 | (21) |
|
5.6.1 Environmental Stress Screening (ESS) testing |
|
|
422 | (2) |
|
|
424 | (2) |
|
5.6.3 Destructive testing |
|
|
426 | (5) |
|
|
431 | (5) |
|
|
436 | (7) |
|
5.7 Black Box---Phase Testing |
|
|
443 | (45) |
|
|
444 | (1) |
|
5.7.2 Exploratory testing |
|
|
445 | (2) |
|
|
447 | (5) |
|
5.7.4 Component and subsystem testing |
|
|
452 | (3) |
|
5.7.5 Integration testing |
|
|
455 | (6) |
|
5.7.6 Qualification testing |
|
|
461 | (2) |
|
|
463 | (3) |
|
5.7.8 Certification and accreditation testing |
|
|
466 | (7) |
|
5.7.9 First Article Inspection (FAI) |
|
|
473 | (4) |
|
5.7.10 Production testing |
|
|
477 | (4) |
|
5.7.11 Installation testing |
|
|
481 | (3) |
|
5.7.12 Maintenance testing |
|
|
484 | (3) |
|
|
487 | (1) |
|
|
488 | (7) |
|
Part III Modeling and Optimizing VVT Process |
|
|
495 | (151) |
|
6 Modeling Quality Cost, Time and Risk |
|
|
497 | (53) |
|
6.1 Purpose and Basic Concepts |
|
|
497 | (3) |
|
6.1.1 Historical models for cost of quality |
|
|
498 | (1) |
|
6.1.2 Quantitative models for cost/time of quality |
|
|
499 | (1) |
|
6.2 VVT Cost and Risk Modeling |
|
|
500 | (21) |
|
6.2.1 Canonical VVT cost modeling |
|
|
500 | (2) |
|
6.2.2 Modeling VVT strategy as a decision problem |
|
|
502 | (3) |
|
6.2.3 Modeling appraisal risk cost |
|
|
505 | (6) |
|
6.2.4 Modeling impact risk cost |
|
|
511 | (5) |
|
6.2.5 Modeling total quality cost |
|
|
516 | (1) |
|
6.2.6 VVT cost and risk example |
|
|
517 | (4) |
|
6.3 VVT Time and Risk Modeling |
|
|
521 | (9) |
|
|
521 | (3) |
|
6.3.2 Modeling time of system/VVT lifecycle |
|
|
524 | (4) |
|
6.3.3 Time and risk example |
|
|
528 | (2) |
|
6.4 Fuzzy VVT Cost Modeling |
|
|
530 | (18) |
|
|
530 | (1) |
|
6.4.2 General fuzzy logic modeling |
|
|
530 | (2) |
|
6.4.3 Fuzzy modeling of the VVT process |
|
|
532 | (9) |
|
6.4.4 Fuzzy VVT cost and risk estimation example |
|
|
541 | (3) |
|
6.4.5 Fuzzy logic versus probabilistic modeling |
|
|
544 | (4) |
|
|
548 | (2) |
|
7 Obtaining Quality Data and Optimizing VVT Strategy |
|
|
550 | (54) |
|
7.1 Systems' Quality Costs in the Literature |
|
|
550 | (4) |
|
7.2 Obtaining System Quality Data |
|
|
554 | (3) |
|
7.2.1 Quality data acquisition |
|
|
554 | (1) |
|
7.2.2 Quality data aggregation |
|
|
555 | (2) |
|
7.3 IAI/Lahav Quality Data---An Illustration |
|
|
557 | (5) |
|
7.3.1 IAI/Lahav pilot project |
|
|
557 | (2) |
|
7.3.2 Obtaining raw system and quality data |
|
|
559 | (1) |
|
7.3.3 Anchor system and quality data |
|
|
560 | (1) |
|
7.3.4 Generating the VVT model database |
|
|
561 | (1) |
|
|
562 | (2) |
|
|
562 | (1) |
|
|
563 | (1) |
|
7.5 VVT Cost, Time and Risk Optimization |
|
|
564 | (36) |
|
7.5.1 Optimizing the VVT process |
|
|
565 | (4) |
|
7.5.2 Loss function optimization---VVT cost |
|
|
569 | (7) |
|
7.5.3 Weight optimization---VVT cost |
|
|
576 | (4) |
|
7.5.4 Goal optimization---VVT cost |
|
|
580 | (4) |
|
7.5.5 Genetic algorithm optimization---VVT time |
|
|
584 | (12) |
|
7.5.6 Genetic multi-domain optimization---VVT cost and time |
|
|
596 | (4) |
|
|
600 | (4) |
|
8 Methodology Validation and Examples |
|
|
604 | (42) |
|
8.1 Methodology Validation Using a Pilot Project |
|
|
604 | (14) |
|
8.1.1 VVT cost model validation |
|
|
605 | (5) |
|
8.1.2 VVT time model validation |
|
|
610 | (7) |
|
8.1.3 Fuzzy VVT cost model validation |
|
|
617 | (1) |
|
8.2 Optimizing the VVT Strategy |
|
|
618 | (21) |
|
8.2.1 Analytical optimization of cost |
|
|
619 | (7) |
|
8.2.2 Cost distribution by phase |
|
|
626 | (1) |
|
8.2.3 Weight optimization of cost |
|
|
627 | (4) |
|
8.2.4 Goal optimization of cost |
|
|
631 | (4) |
|
8.2.5 MPGA optimization for time |
|
|
635 | (2) |
|
8.2.6 SSGA optimization of cost and time |
|
|
637 | (2) |
|
8.3 Identifying and Avoiding Significant Risks |
|
|
639 | (5) |
|
8.3.1 Avoiding critical risks |
|
|
640 | (2) |
|
8.3.2 Conjecture on future risk scenarios |
|
|
642 | (2) |
|
8.4 Improving System Quality Process |
|
|
644 | (2) |
|
Appendix A SysTest Project |
|
|
646 | (11) |
|
|
646 | (2) |
|
|
648 | (1) |
|
A.3 SysTest Pilot Projects |
|
|
649 | (4) |
|
|
653 | (2) |
|
A.5 EC Evaluation of SysTest Project |
|
|
655 | (2) |
|
|
656 | (1) |
|
Appendix B Proposed Guide: System Verification, Validation and Testing Master Plan |
|
|
657 | (14) |
|
|
657 | (1) |
|
|
658 | (1) |
|
B.3 Chapter 1: System Description |
|
|
659 | (1) |
|
B.3.1 Project applicable documents |
|
|
659 | (1) |
|
B.3.2 Mission description |
|
|
659 | (1) |
|
|
659 | (1) |
|
B.3.4 Critical technical parameters |
|
|
660 | (1) |
|
B.4 Chapter 2: Integrated VVT Program Summary |
|
|
660 | (2) |
|
B.4.1 Integrated VVT program schedule |
|
|
660 | (1) |
|
B.4.2 VVT program management |
|
|
661 | (1) |
|
|
662 | (7) |
|
|
662 | (3) |
|
B.5.2 Planning VVT activities |
|
|
665 | (3) |
|
|
668 | (1) |
|
B.6 Chapter 4: VVT Resource Summary |
|
|
669 | (2) |
|
|
669 | (1) |
|
B.6.2 Test sites and instrumentation |
|
|
669 | (1) |
|
B.6.3 Test support requisition |
|
|
669 | (1) |
|
B.6.4 Expendables for testing |
|
|
669 | (1) |
|
B.6.5 Operational force test support |
|
|
670 | (1) |
|
B.6.6 Simulations, models and test beds |
|
|
670 | (1) |
|
B.6.7 Manpower/personnel needs and training |
|
|
670 | (1) |
|
|
670 | (1) |
|
Appendix C List of Acronyms |
|
|
671 | (8) |
Index |
|
679 | |