Editorial for the ITEA Journal Special Issue on DATAWorks, September 2024

The Defense and Aerospace Test and Analysis Workshop (DATAWorks) is an annual workshop on methodological advancements for testing. DATAWorks is a collaboration between the Director of Operational Test and Evaluation (DOT&E), the National Aeronautics and Space Administration (NASA), the Institute for Defense Analyses (IDA), and the Section on Statistics in Defense and National Security of the American Statistical Association (ASA). The primary mission of DATAWorks is to strengthen the test and evaluation (T&E) community by promulgating rigorous approaches to human-systems interaction, statistical test design, data analysis, cybersecurity, and other fields in defense and aerospace.

DATAWorks 2024 was the ninth annual workshop and the third hybrid workshop. Over 90 speakers contributed keynotes, short courses, mini-tutorials, talks, speed talks, and poster presentations. DATAWorks hosted roughly 225 in-person attendees and around 75 virtual attendees from government, academia, industry, national laboratories, and the military services (see Figures 1–6). The workshop also had excellent student participation from both military and civilian universities.

Note:  This issue’s editorial intersperses several pictures from the conference with the prose to convey the atmosphere of this year’s DATAWorks.

Figure 1. A DATAWorks presenter showcases work to Dr. Shawn Whetstone (IDA)

This special issue of the ITEA Journal is the product of ITEA’s partnership with DATAWorks. It is the second special issue on DATAWorks; the first appeared in 2023. Our impression is that DATAWorks continues to engage participants from an increasingly diverse set of backgrounds and specializations, and therefore the event has become inclusive of analyses and research from many new T&E-adjacent areas, such as artificial intelligence, mission engineering, and joint testing. We are happy to curate these important, technical topics in this special issue, and we hope you will benefit from the research.

Figure 2. Dr. Douglas Schmidt (DOT&E) gives a keynote presentation on artificial intelligence

Some of our workforce development articles have been held until the December edition to make room for the DATAWorks articles. However, one great interview with John Frederick of the US FAA has been included. John’s interview reminds us of the ebb and flow of test policy in areas of national infrastructure outside of Defense. He has had to champion many aspects that Defense test managers may take for granted. Further, his interview reminds us that many testers are subordinate to, or reliant upon, broader systems engineering commitments and policy for verification and validation, with many ramifications thereof.

We are pleased to present 9 (!) technical articles showcasing the incredible work of T&E researchers and practitioners. With one exception (“Development of a Digital Engineering Testing Framework for CubeSat Applications”), each article is based on research presented at DATAWorks.

As with the previous DATAWorks Special Issue, each article has undergone rigorous peer review by multiple researchers from the T&E community.
In addition to these contributions, we encourage you to browse the full archives of the research presented at the workshop. You will likely see something new that you can apply to your work!

1. Mission Engineering

Dr. Judith S. Dahmann and Ms. Gabriela I. Parasidis, MITRE

Dr. Dahmann and Ms. Parasidis present the U.S. DoD’s current mission engineering methodology and describe one approach for applying that methodology to mission engineering analyses. Traditionally, systems engineering has focused on designing systems or systems-of-systems to achieve specified technical performance. Mission engineering goes one step further to evaluate the performance of the system-of-systems in achieving the mission or capability objectives when implemented in a realistic scenario in a simulated environment. Some readers know about mission engineering, but this is largely a novel concept for testers, so we are glad to expand this concept to a new audience.

2. Adaptive Test and Evaluation via Bayesian Decision Theory

Bayesian decision theory is a framework for making decisions based on available data and our prior beliefs.  Dynamo is a paradigm that applies Bayesian decision theory to T&E.  Dr. Ferry and colleagues discuss the principles behind the Dynamo paradigm and the progression of capabilities it provides for T&E as it is deployed in three phases.  They include an example that illustrates how to apply their methodology for test planning and data analysis.

Dr. Jim Ferry, Dr. Nate Crookston, and Dr. Adam Ahmed, Metron

Figure 3. Dr. Tom Donnelly (JMP) explains the benefits of functional data analysis for T&E to Dr. James Wisnowski (Adsurgo) and colleagues

3. Reimagining T&E for the Modern Joint Environment: The Joint Test Concept

Ms. Christina Houfek, Dr. Maegen Nix, and Ms. Natalie Wells,
Virginia Tech Applied Research Corporation

Ms. Houfek and colleagues present their Joint Test Concept (JTC), an effort to change how T&E contributes to the overall assessment of measures and outcomes that are aligned with complex mission webs. The JTC recognizes that systems under test tend to be part of a system-of-systems capability lifecycle, which requires testers to apply a capability lifecycle campaign of learning that delves into critical layers of questions about a system’s functional focus, a service’s mission focus, a system-of-systems capability focus, and a joint all-domain mission focus. The JTC has several primary functions: design a campaign-of-learning strategy, develop a data strategy for the campaign of learning, monitor for decision points, refine the T&E event plan, execute the event, and update the strategies. These functions enable an iterative approach to answering key T&E questions and support milestone decisions during the capability lifecycle.

Figure 4. IDA President Norton Schwartz (Gen, RET) addresses the DATAWorks audience

4. Dodging Pitfalls in Packages for Artificial Intelligence and Machine Learning

Dr. Justin Krometis,
Virginia Tech National Security Institute

Artificial intelligence and machine learning (AI/ML) software packages provide tremendous power and flexibility to users, and greatly facilitate learning and deploying models. However, these packages and the models they include are complicated and may have several subtle but serious pitfalls. Dr. Krometis presents three examples where obscure settings or bugs in AI/ML software libraries dramatically changed model behavior or performance – one from a deep learning regression application, one from reinforcement learning, and one from computer vision classification. These examples illustrate the importance of thinking carefully about the results a model produces and checking each step in the development of the model before trusting its output.

5. Optimizing Performance of Real-Time Detection and Classification for Military Personnel and Weapons

Ms. Karly Parcell, Dr. Brian Choi, and Dr. James Starling,
United States Military Academy

Ms. Parcell and colleagues consider the application of the YOLOv5 model to real-time object detection, testing a YOLOv5 modified to maximize its performance in detecting soldiers and weapons. They demonstrate that the modified YOLOv5 detects and classifies dismounted threats in surveillance footage significantly better than the original YOLOv5 model does. Additionally, their findings indicate that the model has enhanced true positive rates when detecting guns and persons, which reduces the need for extensive human monitoring. Importantly, they discuss ethical considerations and future enhancements for the modified model, including the use of synthetic data to train the model under varied conditions.

6. Development of a Digital Engineering Testing Framework for CubeSat Applications

Mr. Javier Alberto Martell, Ms. Anamika Ahmed Siddique,
Dr. Roberto A. Osegueda, Dr. Angel Flores-Abad, Dr. Ahsan R. Choudhuri, Dr. Sergio Alberto Luna Fong, and Dr. Joel Quintana,
University of Texas at El Paso

Digital engineering (DE) is playing a crucial role in industry, especially as systems become more complex. A prime example is the use of DE tools in the development at the University of Texas at El Paso’s Aerospace Center of the SleeperSat-1 (SPST-1), a modular small satellite. SPST-1, like other CubeSats, requires thorough environmental testing to ensure the success of its mission, the safety of its crew, and the safety of other nearby satellites. Ms. Siddique and colleagues use SPST-1 to investigate how DE tools integrate to form a detailed digital twin.

Figure 5. Dr. Kyle Kolsti and STAT Center of Excellence colleagues discuss presentations

7. From Text to Metadata: Automated Product Tagging with Python and Natural Language Processing

Ms. Aayushi Verma and Mr. Omar Agha Khan,
Institute for Defense Analyses

IDA produces a variety of research products for sponsors, such as reports, memoranda, and slides. Quickly assigning keywords to these products and in a manner that permits researchers to efficiently retrieve information on given topics poses a challenge. IDA has numerous methods for tagging products with terms from IDA-defined taxonomies, but the process is manual and time-consuming and must be repeated periodically to tag newer products. To address this challenge, Ms. Verma and Mr. Khan developed a Python-based automated tagging pipeline. They introduce the mechanics of this pipeline, highlight current results, and discuss future applications of these tags within IDA’s research.

8. Benchmarking ResNet50 for Image Classification on Diverse Hardware Platforms

Mr. Matthew Wilkerson, Ms. Grace Vincent, and Dr. Sambit Bhattacharya,
Fayetteville State University
Dr. Zaki Hasnain and Dr. Emily Dunkel,
Jet Propulsion Laboratory

In edge computing, optimizing deep neural networks within limited computational resources is essential. This study concentrates on improving the efficacy of the ResNet50 model via static quantization. Mr. Wilkerson and colleagues conducted performance evaluations on multiple platforms, including a desktop PC, an Intel Next Unit of Computing mounted on a drone, and an Nvidia Jetson Nano integrated into a custom robot, to assess the impact of quantization on computational efficiency and model accuracy. They demonstrate that quantization reduces model size by approximately 74% and decreases average inference times by 56%–68% with minimal effect on accuracy. Their results corroborate the utility of quantization as a viable approach for deploying complex neural networks in edge computing environments while retaining high accuracy levels.

9. Cobalt Strike: A Cyber Assessment Challenge

Dr. Nathan R. Wray and Mr. Sean Phipps,
SIXGEN and Director of Operational Test and Evaluation

Assessment is a critical part of the cyber tool procurement process for red teams. Once a tool has been approved for use by a red team, it is incorporated into the team’s steady state operations. As a result, newer versions of approved tools may not undergo routine cyber assessments. This oversight is a major concern for the red team community, because new versions can change the operational security of those tools. Similarly, cyber defenders – due to a lack of training or resources – may upload red team cyber tools to commercial malware analysis platforms, which inadvertently releases sensitive information about red team operations. Dr. Wray and Mr. Phipps discuss red team cyber tools, analyze Cobalt Strike versions 4.8+, and provide recommendations on evaluating cyber tooling.

Figure 6. IDA researchers Denise Edwards and Aayushi Verma discuss metadata creation

We are very pleased to introduce Mr. Viruben Watson as the new assistant editor of the ITEA Journal. Mr. Watson retired from the Royal Australian Navy as a senior flight test engineer and is now a senior systems engineer at Thales. He has over 20 years of experience with maritime and aviation systems, T&E, software/systems engineering and development, and modeling and simulation. He will be a valuable asset to the journal.

Mr. Watson joins Dr. Keith Joiner, the chief editor of the journal. Together, they will be the constant stewards of the journal moving forward. They have provided excellent guidance to us during our tenure as guest editors as well as given helpful feedback to the authors who contributed articles to this DATAWorks issue.

The 10th annual DATAWorks will occur April 22–24, 2025, in Alexandria, Virginia. You can find recordings and materials from previous years and stay up to date with DATAWorks by visiting dataworks.testscience.org. The abstract submission window for contributed presentations and posters will open in October 2024, so please help the organizers deliver another blockbuster workshop next year!