Archived Issues

Archived Issues

Members have access to all the archived issues of the ITEA Journal of Test and Evaluation as a benefit of membership. As a non-member, you have the opportunity to access the archived abstracts (below). In 2023, we implemented a few changes in the format and access to the Journal and welcome you to read the most recent issue. We hope you will join your colleagues and peers and become an ITEA member. 

We invite you to read the archived journals from 2023.

Abstracts 2022 

December 2022 – Cyber T&E – Testing for Resilience

To close out 2022 we have two new additions for our Workforce of the Future Column. The first article comes from a Navreeta Singh, Princeton University, who interned with Dr. Jeremy Werner at Director Operational Test and Evaluation (DOT&E).  In their article, “Uncertainty Analysis Demonstration: A Missile Case Study” they highlight that validation is a key component of modeling and simulation (M&S) and role uncertainty quantification plays in the validation using sample missile test.  The second student article comes from a team at the Virginia Tech National Security Institute.  In her internship Cassidy Honea, Virginia Tech work with researchers at VTNSI to develop the article, “Synthetic Aperture Radar (SAR): Utilizing Radon Image Transformations to Enhance the Detection of Ship Wakes.” Mr. Erik Higgins, Mr. Daniel Sobien, and Dr. Justin Kauffman worked with Ms. Honea to conduct the research where they show using simulated SAR that Radon Transformation improves the probability of ship wake detection.

The first technical article on “Chaos Engineering” is authored by Ms. Jenn Bergstrom, Fellow at Parsons.

  • Abstract: Chaos engineering, developed at Netflix, is a new approach for software testing at scale in an operational environment. The adoption of chaos engineering follows the pattern of other now popular public domain applications and tools, like Hadoop, TensorFlow, and PyTorch. Large tech companies dependent on web traffic and distributed software applications for their success have developed chaos experiment tools and released them to the open-source community. Early adopters have created as-a-Service chaos tools that are used by companies across multiple domains to validate the resiliency of their systems. The background and principles of chaos engineering are introduced and demonstrated for distributed systems operating in public cloud environments. Application to other domains, such as security, personnel, and to organizations, is also described.

In the second technical article Mr. Michael Leite, Test Area Manager, R&D Programs at Department of Homeland Security, Science and Technology Directorate discusses, “Testing of Research and Development Projects.”

  • Abstract: The conduct and management of testing for research and development (R&D) programs involves different goals from the test and evaluation (T&E) efforts typically applied within acquisition programs. The Department of Homeland Security (DHS) has established a separate testing group to work with R&D program managers to ensure that testing of their projects is adequate to support the transition to acquisition, procurement, or commercialization.

Our final technical article comes from Dr. William “Data” Bryant at Modern Technology Solutions, Inc.  and discusses, “Measuring the Measurers: Using Test to Validate Cyber Risk Assessments.”

  • Cyber risk assessment processes claim to be able to predict the success of cyber-attacks against weapon systems and platforms in a range of highly-contested cyber environments, but what evidence do we have that they are more accurate than random guesses or tea leaves? Fortunately, on systems that are also undergoing cyber test, we can leverage this test to validate that the risk process is effective by having risk assessors also provide predictions on what test cases will be successful and comparing the predictions and test results.

Enjoy the Issue!

September 2022 – Digital Engineering’s Impact on Test and Evaluation

This month’s issue starts off with another great Conversations with Experts column in which J. Michael Barton, Ph.D., interviews Malcolm Tutty, Ph.D. Dr. Tutty has served in the Royal Australian Air Force, Public Service and Industry in a multitude of test, operations, engineering, staff, project management and command roles. Dr. Tutty shares his experiences in and out of T&E. He also calls back to numerous quotes that I am sure will keep ITEA readers engaged. A quick line of advice that he slips in, “Enjoy your service, make friends everywhere, and I hope you have successful careers in and out of the T&E space along with having as much fun as I have had with many of y’all.”

In our Workforce of the Future column this month we have contributions from undergraduate and graduate student-led research at the University of Memphis. The first article, “Testing the Control Software in Autonomous Cyber-Physical Systems,” by Bhavana Garimilla and Mark L. Gillenson, Ph.D., examines how to test software in autonomous cyber-physical systems. The authors highlight the need for ensuring safety and the critical role of testing software extensively when it incorporates neural networks. In the next article, “Managing the Security of Data in the Cloud,” Venkata Akshara Motati and Mark L. Gillenson, Ph.D., discuss the security of cloud-based data storage. They provide recommendations that merit consideration before moving data to the cloud. In our third article, “Simulation and Testing of AI/Autonomous Systems,” Faizan Zafar and Eddie Jacobs, Ph.D., report on state-of-the-art ­practices and conduct simulations using the Mississippi State University’s Autonomous Vehicle Simulation (MAVS) library.

Our first technical article comes from Nick Jones and Wayne Adams at the Scientific Test and Analysis Techniques Center of Excellence (STAT COE), Air Force Institute of Technology. Their article entitled, “Model Selection and Use of Empiricism in Digital Engineering,” addresses the important role that modeling plays in test and evaluation. The article outlines a process based in STAT by which a digital engineering test program should leverage the digital engineering (DE) environment by first utilizing existing resources on the digital thread and then first-principles and empirical data to build new models when needed.

In our next article Donald H. Costello III, Ph.D., from the United States Naval Academy and Huan Xu, Ph.D., from the University of Maryland discuss “Using a Run Time Assurance Approach for Certifying Autonomy Within Naval Aviation.” The article illustrates how to use run time assurance as the framework for a safety of flight certification of autonomous behavior. They leverage an aerial refueling task as a use case for the framework for certification.

In our final article a research team from Virginia Tech including Laura Freeman, Ph.D.; Justin Kauffman, Ph.D.; Erin Lanus, Ph.D.; Daniel Sobien; and Tyler Cody, Ph.D., summarizes some best practices they have adopted in testing Artificial Intelligence Enabled Systems (AIES). As the lead author on the article entitled, “Best Practices for Addressing New Challenges in Testing and Evaluating Artificial Intelligence Enabled Systems,” I would like to highlight that these practices might prove useful to others as they embark on developing test programs for AIES, but they are surely not complete and will need to be updated as our understanding of testing AIES evolves.

Enjoy the Issue!

June 2022 – Test and Evaluation and Digital Data Strategy

In this month’s Conversations with Experts column J. Michael Barton, Ph.D., interviews Eileen Bjorkman, Ph.D., Executive Director at the Air Force Test Center. In the article, “Lofty Goals, A Celebration of Flying,” Dr. Bjorkman discusses her career in the US Air Force, the decision to pursue a Ph.D., and her career in Test and Evaluation (T&E). In her advice to people interested in T&E careers, Dr. Bjorkman recommends that individuals “try out different types of jobs early on and don’t be afraid to try something outside your area of expertise.” She highlights how technologies change and to think of your academic degree as just a starting point.

In our Workforce of the Future column with three articles written by students that participated in the High Performance Computing Modernization Program Internship Program:

  • Richard Healy from Rensselaer Polytechnic Institute and Phuriwat Anusonti-Inthra, Ph.D., Matthew W. Floros, Ph.D., and Hao Kang, Ph.D., round out the articles with their contribution on “Air-Launched Effect Trajectory Prediction using Computational Fluid Dynamics.”
  • Aaron Crawford from the Georgia Institute of Technology and Matthew W. Floros, Ph.D., and Phuriwat Anusonti-Inthra, Ph.D., both from the Army Research Laboratory author an article titled, “Computational Fluid Dynamics Investigations of an Unmanned Aircraft Systems Fuselage and Rotor Aerodynamics.”
  • Isabel F. Fernandez from the Georgia Institute of Technology and Michael Avera from the US Army DEVCOM Army Research Laboratory author an article titled, “Assessing the Performance of the Lattice Boltzmann Method for Rotorcraft Flow Applications.”

In our first technical article Alexander Hillman, a major in the US Air Force and a Ph.D. candidate in aeronautical and astronautical engineering at the Massachusetts Institute of Technology, discusses how testers are uniquely equipped to lead new data strategy efforts. Hillman’s article, “Leveraging a Tester’s Experience to Build a Successful Enterprise Data Strategy,” describes the years of experience, knowledge, and the challenges that testers bring to the data strategy discussion. He outlines the critical elements of data strategies and provides astute observations on how Department of Defense processes and protections can inhibit progress in achieving strategic objectives.

In our next technical article Everett H. Oliver, Ph.D., and Shahram Sarkani, Ph.D., P.E., and Thomas A. ­Mazzuchi, D.Sc., discuss the value of systems thinking and model-based systems engineering in assessing the resilience of critical infrastructure sectors. In their article, “Assessing the Resilience of Critical Infrastructure Systems and Supply Chains Against Cascading Disruptions,” they provide a case study to demonstrate the approach leveraging the historical example of Superstorm Sandy and the food supply chain.

In our final technical article Frank Cruz from the US Air Force, Applied Spectrum Technology Research Office (ASTRO), at Edwards AFB discusses, “Optimizing Antenna Placement Using Modeling and Simulation.” Mr. Cruz’s paper showcases how modeling and simulation can be used to enable trade-space exploration in order to optimize hardware design. He highlights the potential cost savings when compared to the cost of chamber testing.

Enjoy the issue!

March 2022 – Accelerating T&E to the Speed of Need

The March 2022 issue focuses on Test and Evaluation (T&E) at the speed of need. In the Conversations with Experts, J. Michael Barton, Ph.D., interviews Larry Leiby. In the column, Mr. Leiby provides: A Historical View of Army Test and Evaluation. Mr. Leiby brings to the column his experiences across a 42-year career in T&E to include his time with Deputy Under Secretary of the Army (DUSA) and Army Test and Evaluation Command (ATEC). As a former recipient of the ITEA Junior Achiever award, I personally was also interested to learn about Mr. Leiby’s role in creating the award. Mr. Leiby shares his passion for developing the next generation of the T&E workforce.
Speaking of the next generation, we are continuing our Workforce of the Future column with three articles written by students that participated in the High-Performance Computing Modernization Program Internship Program. This program supports undergraduate and graduate students with internship opportunities at Department of Defense (DoD) laboratories and test centers and allows them to learn high-performance computing skills not readily available at their colleges and universities.
• In our first article, Nicholas Kendall, a sophomore at the University of Wyoming, discusses “3D Printing High Performance Computing (HPC) Simulation Results.” Mr. Kendall’s paper also includes Ethan Eldrige, David Kleponis, and Stephen Schraml from DEVCOM Army Research Laboratory. They compare simulations of shaped charge jet formation and penetration to experimental measurements.
• In the next article, Patrick Mortimer, a Ph.D. student at the University of Texas at Austin, discusses “Nonlinear Response of a Coaxial Rotor to Blade Pitch Change.” The work in partnership with George Jacobellis, Ph.D. from US Army Research Laboratory, uses computational fluid dynamics (CFD) simulations to explore nonlinear responses of rotor inflow.
• In the final student article of this issue, Nikos Trembois, a Ph.D. student at the University of California, Davis, also leverages CFD simulations in “Broadband Noise Prediction from CFD Wall Spectrum” which is also a collaboration with Dr. Jacobellis.
Our first technical article comes from Naval Air Warfare Center Aircraft Division (NAWCAD) and is titled, “Risk-Based [Verification, Validation, and Accreditation] VV&A: A Systematic Approach.” In the article, James Elele, Ph.D., David Hall, and David Turner from NAWCAD Integrated Battlespace Simulation and Test (IBST) share the systematic framework their organization has developed to support cost-effective, risk based VV&A process for models and simulations (M&S). They highlight how the process has evolved from the original process and the successes their organization has had in employing the process.
In our final technical article, Colonel Matthew ­Magness from Air Force Operational Test and Evaluation Center (AFOTEC), speaks directly to the theme of this issue. His article entitled, “Accelerate or Lose: Four Ways to Streamline T&E Process Across Multiple Domains,” provides food for thought on how test and evaluation organizations can support the acceleration of getting new capabilities in the hands of warfighters. His reflections and proposals provide valuable insights for T&E organizations at the current “inflection point” he observes between old and new methodologies.
Enjoy the issue!

Archived Issue Abstracts