Book Review of Verification, Validation, and Testing of Engineered Systems – Author: Dr. Avner Engel

As noted in the preface, this book features a total of eight chapters and two appendices organized into three principal sections. It has been written with the practitioner community of test engineers in mind, emphasizing a pragmatic and full lifecycle T&E orientation, yet with sufficient coverage of material to be of interest to program managers, quality assurance managers, and those in specific technical engineering disciplines (e.g., electrical, mechanical). The chapter lengths are somewhat extensive (some running 70-80 pages or more), but numerous sub-sections keep the reader informed of progress through the occasionally lengthy material. A brief description of each of the three topical areas is provided below.

• Part I: Introduction (chapter 1). The book opens with Chapter 1, introducing the VVT process across the entire system lifecycle, including design, development, implementation, integration, and quality assurance phases. Here the author shows how VVT supports the project management and systems engineering processes by providing a framework that supports requirements verification and early analysis of alternative strategies that support design refinement.
• Part II: Verification, Validation, and Testing (VV&T) Activities and Methods (chapters 2-5). This second part of the book proceeds to expand the introductory discussion to a fuller scope of VVT activities across the system lifecycle. Chapter 2 describes the VVT process in system development across the design, implementation, integration, and qualification phases. He dDiscusses the use of system design reviews, acceptance test procedures, and system integration labs for preliminary system assessment. Chapter 3 continues the discussion of VVT processes through the production, maintenance, and disposal phases, including the use of functional configuration audits, integrated logistics support plans, and the use of simulation to reduce environmental impact during system disposal. Chapter 4 considers non-testing related VVT methods including the use of requirements verification matrices, the use and proper timing of system design reviews, as well as a limited discussion of quantitative system risk analysis methods and Design of Experiments methods. Chapter 5 addresses the proper use and application of several types of testing methods, including “white box” and “black box” testing, specialized types of testing (e.g., reliability testing, security vulnerability testing), and environmental testing (e.g., EMI/EMC testing, destructive testing, and weather testing). Additional discussion of testing phases of a system, such as regression testing approaches, component-level testing, acceptance testing, installation and maintenance testing, and disposal testing are also considered.
• Part III: Modeling and Optimizing VV&T Processes (chapters 6-8). Leveraging the foundation laid in Part II, chapters 6-8 consider modeling and simulation in support of the VVT process and various optimization approaches that may be considered. Chapter 6 addresses different approaches for system cost, system risk modeling, and modeling the VVT cost strategy across the system lifecycle; a brief section on fuzzy modeling approaches for VVT cost estimation was particularly interesting to this reviewer. Chapter 7 proceeds to assess the challenge of acquiring and assessing quality test data in support of the VVT process; sections on VVT data collection optimization and goal optimization (including genetic algorithm optimization methods) are quite interesting. Finally, chapter 8 applies the optimization approaches discussed in chapters 6 and 7 to a sample pilot project, with suggestions and recommendations for process improvement also provided.
• Appendices: SysTest and System VV&T Master Plan. The two appendices provide supplemental material on using and applying the SysTest software program for system VVT planning, cost estimation, and test duration planning (Appendix A) and a proposed generic guide and framework for implementing a system VVT master plan (Appendix B.) The SysTest program discussed in appendix A is an interesting capability that could be useful for preliminary test planning and VVT process alignment with broader system engineering and project management performance milestone timelines, but its age and seeming lack of consistent updates may relegate it to another “good idea” for system test plan development and implementation. Regardless, such programs inevitably need consistent updates to maintain their technical relevance. The system VVT master plan template and framework likewise is a helpful introduction to the broader master plan development process, but one which also needs a contemporary update to reflect more recent changes in the system engineering requirements VVT process environment (an alignment of the content in Appendix B with contemporary INCOSE System Engineering Handbook updates might be helpful in this regard.)

This book represents a broad and deep swath of system VVT processes across the system design, development, and deployment lifecycle process. Dr. Engel does a good job of presenting and discussing the differences between verification and validation processes, and their respective emphasis on ensuring that proper system capabilities are developed and confirmed. In addition, the chapter treatments provide a sound and practical application of system requirement verification matrices and VVT management plans. These early system activities are used as the springboard for VVT activities later in the system implementation, integration, qualification, and post-deployment phases. Sufficient technical explanation is provided, but Engel’s breezy discussion prevents the somewhat lengthy chapters from becoming the usual sea of acronyms and overly detailed T&E process details you find in many Government handbooks on these topics. This balance between VVT process description and practical application of these concepts and tools in the system development lifecycle is the principal value of this book. On a personal note, the present reviewer found the book to be an enjoyable read with, perhaps, a little too much emphasis in some places on the administrative and documentation support for the VVT process. Nevertheless, the content is well-organized and clearly presented. The Chapter Five sections on black box phase testing for regression, integration, and acceptance testing are particularly interesting to this reviewer, given his previous T&E work on this level of systems testing.

Besides the technical overview and the prior reviewer comments, one is compelled when doing a book review for the ITEA Journal to ask why a test engineer should read this book. Several reasons for interest and engagement by the test community come to mind. First, the initial part of the book presents a good introduction to the general overview of the T&E process across the sequential scope of system lifecycle phases. This broad summary alone is worth the price of the book, especially for students in introductory subjects in VVT. The second part encompasses most of the content and extends the material presented in the Introduction into a series of chapters with a specific focus on each of the engineering design and development phases. Finally, the third part of the book, with its focus on VVT modeling and optimization processes provides a good survey of VVT cost, time, and risk estimation approaches

The discussion of VVT modeling and optimization approaches presents a good companion to other quantitative analysis methods used in systems engineering for system cost estimation and probabilistic risk analysis. Second, using and applying these modeling tools fosters improved collaboration of the T&E discipline with program management, systems engineering, and physics-based engineering disciplines. Third, the appendix discussing representative VVT master plan development provides a helpful summary of the key features that a proper system testing plan should contain and how these elements help to present a well-rounded approach to system testing through the lifecycle process. Finally, proper consideration of the VVT processes and M&S capabilities by the T&E community can support system testing in environments or system configurations that cannot be safely or conveniently tested at the physical incarnation level.

The book does, however, exhibit some weaknesses in several areas. Given the age of the book, contemporary topics such as testing methodologies for artificial intelligence systems, or a more comprehensive review of statistical test design methods are not broadly represented. In addition, the SysTest software environment described in Appendix A seems a bit outdated and perhaps due for a refresh if the book goes through a comprehensive revision at some future point. Moreover, the systems VV&T master plan process template described in Appendix B is helpful and fairly thorough, yet could benefit from an expansion that could include specific examples for software VV&T and artificial intelligence systems.

Even with the above-noted limitations, this book provides a thorough and robust addition to the systems test and evaluation and provides very useful insights into VV&T processes and their respective application to complex systems testing. One might wish for more discussion of statistical test design methods or advanced testing concepts for artificial intelligence-enabled systems, but the book nonetheless presents a valuable description of the VV&T process through the system lifecycle. The contemporary T&E “shift left” philosophy is effectively described in Engel’s book and thereby presents a useful knowledge cap to enable the test community to implement this test philosophy better.

Engel, Avner. Verification, Validation, and Testing of Engineered Systems. John Wiley & Sons, 2010.