Editorial – ITEA Journal – September 2025

On 22 March 2025, individuals gathered in Alexandria Virginia to kick off the Defense and Aerospace Test and Analysis Workshop, commonly known as DATAWorks. DATAWorks is a multi-organization event put on by the Director of Operational Test & Evaluation (DOT&E) within the Office of the Secretary of Defense, the National Aeronautics and Space Administration (NASA), the Institute for Defense Analysis (IDA), and the Section on Statistics in Defense and National Security (SDNS) of the American Statistical Association (ASA). It provides T&E practitioners and data-centric communities a forum to network, mingle, and share their data-driven approaches to evaluation and decision-making. Over the last few months, we have had the pleasure of assembling several stellar pieces that emerged from this event.

The bulk of this special issue is dedicated to technical work presented at DATAWorks 2025. These articles cover an engaging scope of topics, ranging from a Bayesian-inspired skip-lot sampling technique for testing ammunition quality to applications of uncertainty-aware neural networks to spectroscopic data collected by NASA’s Mars rover Curiosity.

This special edition starts out strong with an article by Drs. Carlo Graziani and Marieme Ngom from Argonne National Laboratory (Lemont, Illinois). Their article, “Kernel Model Validation: How To Do It, And Why You Should Care”, explores the challenges associated with testing and validating kernel choices for Gaussian Processes. Graziani and Ngom discuss two quantitative indicators for quantifying the validity of a Gaussian Process kernel – the Mahalanobis distance and fit to normal-mode values – and demonstrate their usefulness in determining whether a model has been misspecified.

Our second article is by Alexander Boarnet, a 2025 graduate of the United States Military Academy at West Point, who received his degree in Mathematical Sciences, and Dr. Mike Powell, an Academy Professor in the Department of Mathematical Sciences at the United States Military Academy at West Point. Their article is titled “Confidence-Based Skip-Lot Sampling”. In their article, Boarnet and Powell explore skip-lot sampling for small-caliber ammunition production. They propose a Bayesian method that ties the probability of skipping a lot to the belief about the current state of the quality of the manufacturing processes. This method creates savings for a production facility while still maintaining the facility’s standards for risk exposure.

Our third article is by Dr. Imène Goumiri, Dr. Jayson “Luc” Peterson, and Dr. Adam Taylor from Lawrence Livermore National Laboratory (Livermore, California). Their article, “Eucalyptus – An Analysis Suite for Fault Trees with Uncertainty Quantification” introduces a Python-based tool, Eucalyptus, that extends Fault Tree Analysis by incorporating uncertainty in component existence and failure. Using Monte Carlo sampling, Eucalyptus generates ensembles of possible fault tress, enabling sensitivity studies, scenario modeling, and visualizations that help analysts quantify the impact of knowledge gaps on overall system reliability.

Our fourth article, “Digital Twins in Reliability Engineering: Innovations, Challenges, and Opportunities” was written by Dr. David Han, an Associate Professor at the University of Texas at San Antonio, and Dr. James Brownlow, a statistician at the Air Force Test Center (Edwards Air Force Base, California). This article explores the rapidly evolving technology of digital twins (DT), which integrates computer science, engineering, and statistics to model and monitor complex systems. The authors highlight DT applications across industries such as healthcare, renewable energy, and national security, emphasizing its role in reliability engineering, preventive maintenance, and quality control. Leveraging data analytics, machine learning, and AI, DT enables simulation, optimization, and better decision-making, improving system understanding and operational practices.

Our fifth article is “Competence Measure Enhanced Ensemble Learning Voting Schemes” by Francesca McFadden, a doctoral candidate in the Department of Mathematics and Statistics at the University of Maryland, Baltimore County.  This article presents an approach to enhance ensemble learning by incorporating base model confidence into voting schemes. This method weighs predictions based on each model’s competence for a given input, ensuring consistency with its prediction space. Demonstrated using random forest classifiers, this approach improves performance over traditional confidence-based selection, highlighting the value of model diversity and informed aggregation in ensemble methods.

Our sixth article is “Advancing the Test Science of LLM-enabled Systems: A Survey of Factors and Conditions that Matter Most” by Karen O’Brien, a senior principal data scientist and the AI/ML practice lead at Modern Technology Solutions, Inc (Huntsville, Alabama). This article examines gaps in testing and evaluation of Large Language Model (LLM) enabled systems between academia and industry, identifying keys factors that affect system performance and providing guidance for rigorous experiments. By incorporating customization and expert input, this approach helps ensure LLM-enabled systems are evaluated with scientific rigor to support mission success.

In our final DATAWorks article, Dr. Natalie Klein, Mark Hinds, Dr. Scott Koermer, and Dr. Michael Geyer from the Statistics Group at Los Alamos National Laboratory explore uncertainty-aware neural network models applied to data collected by NASA’s Mars Curiosity rover. In their paper, “Beyond Accuracy: Evaluating Bayesian Neural Networks in a Real-World Application”, they predict the chemical composition of the surface of Mars using spectral data collected by ChemCam, an instrument onboard the rover. They consider three Bayesian convolutional neural network approaches, and evaluate each using several metrics, such as RMSE, test set coverage and interval width for 95% prediction intervals, and epistemic and total uncertainties.

In addition to our DATAWorks features, this special edition includes an article by Maryam Gracias, a doctoral student in Systems Engineering, and Dr. Erika Gallegos, an Associate Professor in Systems Engineering, both at Colorado State University in Fort Collins, Colorado. Their paper is titled, “Balancing Structure and Flexibility: Evaluating Agile, Waterfall, and Hybrid Methodologies in Aerospace and Defense Projects”. Their work investigates hybrid development frameworks for project management that integrate the rigor of Waterfall methodologies with the flexibility of Agile methodologies. They explore Sustainment Integration for a Legacy Aircraft system as a representative case-study to demonstrate that hybrid approaches offer the optimal balance between stability and adaptability in aerospace sustainment projects.

Finally, we are pleased to feature an article by Drs. Olga Zinovieva and Li Qiao, from the University of New South Wales in Canberra, Australia.  Their article, “Building Confidence, Interest, and Opportunity: A Social Cognitive Career Theory-Based Analysis of the Young Women in Engineering Outreach Program”, stresses the importance of exposing young women to hands-on engineering activities, mentoring, and industry interaction. Annualized intervention initiatives such as the one discussed in this article are crucial as young women consider their career options each year. By providing young women with early exposure to engineering fields and providing them with the appropriate tools, we can create a more inclusive workforce with diverse perspectives and increased problem-solving abilities.

It was truly a privilege to curate this special issue. We thank the team at the ITEA Journal for giving us this invaluable opportunity, as well as the authors who contributed to this issue for sharing their ideas and passions with us. We sincerely hope you enjoy this diverse collection of articles. Each contribution to this issue is intriguing and compelling in its own right, and furthers the T&E field via complex and thoughtful solutions.