Volume 45, Number 2

June 2024 I Volume 45, Issue 2

Dr. Keith JoinerCSC, CPPD, CPEng, Group Captain, Reserve

Editorial – ITEA Journal – June 2024

Values of Test and Evaluation

T&E Values

This edition of the ITEA Journal focuses on the values of T&E. Within the ITEA Board Dr Mal Tutty is proposing a code of practice for T&E to mirror those codebooks in project management and systems engineering. Such work would improve scaffolding of where T&E supports these two bigger professions. My first argument to Dr Tutty was that codes of practice are built upon first values and second principles. Hence, if you see my first editorial of the Journal the draft of the values of T&E were given, and a survey link. In this edition, several passionate testers have described their values, with three short articles from:

  • Wayne Dumais, My T&E Career, The First 25 Years,
  • Steve Woffinden, The Architecture Analogy in Test Planning: An example of the T&E value of ;’Well-Planned’, and
  • Dr Bob Holcomb, Value and Values of Operational Testing.

Values like integrity, timeliness, context, courage, independence, scoped, representative and many more come to life in these three reflective articles. It was very brave of Wayne, Steve and Bob to be the first to directly address how their values shaped their careers and how they were shaped by the T&E community’s values.

Technical Articles

There are three great technical articles in this edition. The first comes from industry and concerns ‘A Statistical Review of the Cyber Test Process’ by Dr Patrick Lardieri, David Harrison, Dr Sharif Hassan, Michael Hankins, and Govindra Ganesh. These highly experienced and well-educated industry leaders spotlight the metrics needed to decide how much cybersecurity testing is enough in development and acceptance. There remains a real struggle to estimate the amount of cybersecurity testing apriori and to decide which metrics to use. So, ITEA is extremely grateful they share their recommended metrics and a case study. Cybersecurity is a field where industry is usually moving ahead of academia and often Government, so like this outstanding team, please contribute if you have similar insights to share.

The second technical article is from Captain James Winton and Dr John Colombi and concerns ‘Digital Test and Evaluation.’ Digital models using tools like Model-Based Systems Engineering (MBSE) are being developed to assure the development and through-life management of complicated requirements, supply chains, and cybersecurity. These models offer better resilience and a means to create digital twinning. However, there are few articles dealing with how to assure these digital threads and attest to the Authoritative Source of Truth (ASOT) that they store. This insightful paper ‘proposes a model-based validation technique, details a proof-of-concept, and describes how it may enable future acquisition.’ As a fellow educator in T&E, I am especially pleased they describe how they scaffold the validation approach into an open-ended problem-solving pedagogy within a technical acquisition curriculum. The preliminary design workshops at the Air Force Institute of Technology (AFIT) nicely grounds the proposed digital test and evaluation approach.

Our third technical article is from Tom Roltsch concerning the robustness of using the classical mean time between failure (MTBF) test. Reliability standards contain some cautions that his practice and analysis find are overly discouraging. If your test programs contain reliability metrics, you should read his paper to deepen your understanding. He invites anyone with examples of where the classic test does not readily apply to contact him and consider presenting and publishing with ITEA concerning the exceptions.

Conversations with Experts

Luckily for the theme of this Edition, two interviews by Dr Michael Barton with T&E experts were ready and the interviews concern gentlemen with enormous experience in representing the T&E community’s values. Such interviews typically follow a chronology of their careers but if you look carefully there are hints there. For example, Edward Greer states “As the first person to serve in the politically appointed position, it was my task to stand up the organization and implement the other recommendations of the Task Force.” I know from my own experience the sorts of challenges he is likely referring to. He rather understatedly follows with “DT&E was a significant participant at the DAB meetings”, likely meaning his test knowledge and values had a considerable impact, using courage and integrity.

Similarly, the second interview by Dr Barton is titled “A Bourbon with Rusty: A Conversation with Russell L. Roberts.’ ‘Rusty’ has worked at the forefront of challenging technology for decades and balanced the difficulties in being scientific and supporting ongoing R&D while providing effective new capabilities to deter would-be adversaries. That omnipresent balance in testing is to inform further development and yet find the ‘good enough’ for soldiers, sailors, airmen and marines.

Upcoming Editions

Our upcoming September Edition will feature many excellent technical articles from the Defense and Aerospace Test and Analysis (DATA) Workshop held 16-18 April 2024 [https://dataworks.testscience.org/], as these are now under peer-review. The DATA Workshop is the result of a multi-organization collaboration with the Director of Operational Test & Evaluation (DOT&E) within the Office of the Secretary of Defense, National Aeronautics and Space Administration (NASA), the Institute for Defense Analyses (IDA), and the Section on Statistics in Defense and National Security (SDNS) of the American Statistical Association (ASA). Much of the latest developments of this annual workshop deserve to be published for the betterment of all T&E.

If you are presenting at these future ITEA events I urge you to also submit some of your best work to the ITEA Journal for our December edition:

  • Multi-Domain Operations Workshop, https://itea.org/event/2024-annual-
    multi-domain-operations-workshop/, Virginia 16-18 July
  • Cybersecurity Workshop, https://itea.org/event/2024-cybersecurity-
    workshop/, Florida 17-19 September.

Remember we now take Extended Presentations and we welcome summary articles by roving reporters at these events (see last month’s editorial for details).

ITEA_Logo2021
  • Join us on LinkedIn to stay updated with the latest industry insights, valuable content, and professional networking!