2021 ITEA Journal Issues Abstracts

ONLINE Issues (ITEA Members Only)

December 2021 – Success Stories in T&E

In this month’s Conversations with Experts column J. Michael Barton, Ph.D., interviews Fred K. McCoy. Mr. McCoy shares his experiences from a “Half a Century of T&E” reminding us how the field has changed. He also reminds us of the value of membership in ITEA.

We are continuing our Workforce of the Future column with two more articles written by students that participated in the High-Performance Computing Modernization Program Internship Program. This program supports undergraduate and graduate students with internship opportunities at DoD laboratories and test centers and allows them to learn high performance computing skills not readily available at their colleges and universities. In our first article Katherine M. Breczinski, Dale Shires, and Michael An cover, “Tools and Processes for Health Analysis Related to Blast Over-Pressure. This article provides an overview of python code and graphics that characterize data from Blast Over-pressure Studies. In our second student article, Mariya Occorso, Jacob LeBlanc, Jamie Infantolino, and Vincent Perry discuss, “Distributed Processing and Analysis for Large-Scale Heterogeneous Data: Text Classification and Analysis through Clustering.” The authors preprocess data and take us though a rigorous analysis of identifying clusters in text data.

This issue focuses on best practices and success stories, allowing us the opportunity to reflect on our past and how best practices have evolved. In a special feature for this issue J. Michael Barton, Ph.D., reproduces a Panel of ITEA Presidents. This article is the result of a panel discussion held on 16 September 2021 as part of the 38th International Test and Evaluation Symposium. The panel was moderated by Matt Reynolds, the 7th president of ITEA. Matt posed a series of questions to panelists resulting in a lively discussion. The discussion is revealing and entertaining on personal and professional levels. The panel participants included Patricia Sanders, Ph.D., Rusty Roberts, Stephanie Clewer, Mark Brown, Ph.D., Gene Hudgins, Bill Keegan, and Pete Crump.

For those who were not able to join the symposium this year, our next feature by J. Michael Barton, Ph.D., provides a summary of the 38th International Test and Evaluation Symposium, whose theme was “Evolution of T&E in an Age of Rapid Technological Change.” The Symposium was held online from 13 – 17 September 2021, echoing the theme from the 1991 Symposium, Impacts of Emerging Technologies. The message reached 170 registered attendees through 12 invited speakers, 6 panels, 42 technical presentations in 12 technical tracks, and 7 tutorials. The “I” in ITEA was especially visible through participants representing diverse geography and time zones across the US, Canada, Germany, the UK, Italy, Dubai, and Australia. We are looking forward to meeting in person next year for the 39th International T&E Symposium, which will be held 19 – 22 September 2022 in Virginia Beach, VA. Its theme is “Forging the 21st Century T&E Tools in an Era of Great Power Competition” and will be chaired by Erwin Sabile.

We have two Historical Perspectives columns this month. In the first one, J. Michael Barton, Ph.D. reflects on 41 years of ITEA history. In his article, “Is the Past Prologue? A Retrospective and Prospective for ITEA,” Dr. Barton explores the origins, evolution, and current state of ITEA.

In our second Historical Perspectives column Robert Kaper writes, “WW II Aircraft Flies Again for NAWCAD Telemetry.” Robert shares with us how this rebuilt WW II aircraft with upgraded telemetry has unique advantages over the modern aircraft.

In our first technical article, Robin Deiulio from the Test Research Management Center discusses, “Creating Efficiencies in Cyber Policy for Secure Software Development in Agile Environments.” The article discusses challenges in current implementations of the risk management framework and potential. As the article notes, “The TRMC Cybersecurity team is dedicated to solving the RMF and cybersecurity issues the RDT&E community is facing while focusing on secure software development, developing streamlined cybersecurity processes, offering certified products, cybersecurity reciprocity, and training.” The article provides insight to their current efforts in achieving that objective.

In our next technical article, “Looking at Legacy I & V through an Agile Lens,” Scott Goodman, Ph.D. and Doug Moshier discuss how we can rethink Integration and Verification (I&V) efforts with an agile lens to improve insight while decreasing life cycle cost.

In our final technical article, “A Software Connected Approach for Facing the Challenges Associated to New EMSO and Digital Transformation,” Raffaele Fiengo and Massimo Sciotti discuss how Modern Electro-Magnetic Spectrum Operations (EMSO) require the evolution of test and evaluation principles and architectures. They advocate for digital engineering processes as a key element of improving T&E outcomes.

I would also like to provide a special thank you to Michael Barton for his efforts to capture memorable moments from the ITEA Symposium for all of ITEA in this issue!

Enjoy the issue!

September 2021 – Testing Artificial Intelligence and Collaborative Autonomous Systems

In this month’s Conversations with Experts, I had the opportunity to interview Jane Pinelis, Ph.D., Chief, Test and Evaluation, Department of Defense (DoD) Joint Artificial Intelligence Center. In this article, Dr. Pinelis reflects on her work in Test and Evaluation (T&E). She highlights the importance of interdisciplinary teams in T&E. She also highlights how Artificial Intelligence (AI) is bringing attention and excitement to T&E careers and the opportunity for new research.

In the next section, we continue our Workforce of the Future column with four more articles written by students that participated in the High-Performance Computing Modernization Program Internship Program.

This program supports undergraduate and graduate students with internship opportunities at DoD laboratories and test centers and allows them to learn high-performance computing skills not readily available at their colleges and universities. This month’s articles include:

Selena Hamilton, James Hughes, and Virginia To, “Magical Machinations: A Virtual Reality Outreach Tool”
Tony Cruz, Chase Carbaugh, and Matthew Dwyer on “HPC Workload Characterization: Optimizing Data Ingestion and Initial Analysis of SCOUT Workloads”
Sean Leonard, Zhanping Liu, Ph.D., Simon Su, Ph.D., Luis Bravo, Ph.D., Anindya Ghoshal, Ph.D., Muthuvel Murugan, Ph.D., and Alison Flatau, Ph.D., “Application of CinemaScience Methods to Visualize and Analyze Complex Multiphase Flow Processes Supporting Next Generation Future Vertical Lift (FVL) Concepts”
Alexandra Le Moine, Alison Flatau, Ph.D., Simon Su, Ph.D., Luis Bravo, Ph.D., Anindya Ghoshal, Ph.D., and Muthuvel Murugan, Ph.D., “Deposition Models for CMAS (calcium-magnesium-alumino-silicate) Attack in Gas Turbine Engines.”

In our first technical article of this issue, Douglas Flournoy, Bianica Pires, Ph.D., Rob Wittman Jr., Ph.D., and Rob Layden, Ph.D., from the MITRE Corporation, discuss “Simulating the Operational Impact of AI-Enabled Systems: Concepts and Considerations.” In the article, the authors explore the current state of AI simulations and how they might be used in assessing mission effects when AI-enabled systems are deployed into operational missions. They propose a set of tenets “to guide the selection and/or necessary upgrading of these simulations for AI-enabled systems analysis.” They propose a conceptual framework, Beliefs, Desires, and Intentions (BDI) framework, as “well suited for quickly creating transparent, theory-based, credible models of both artificial and human intelligence.”

In the second technical article, Edward Repetski, Shahram Sarkani, Ph.D., and Thomas A. Mazzuchi, D.Sc. discuss “Managing Complexity in Operational Test and Evaluation of Integrated Systems.” The article reviews how systems thinking and corresponding tools can prove useful in planning and integrating information from operational testing. They also highlight how given the cost of each tool can be defined, their values can be compared for overall efficiency and efficacy.

In the final technical article of this issue, Thomas Klemas, Ph.D., Sean Atkins, Ph.D., Nazli Choucri, Ph.D., and Rebecca K. Lively address “Accelerating Cyber Acquisitions: Introducing a Time-Driven Approach to Manage Risk with Less Delay.” The article addresses the unique challenges associated with the cyber domain in that system technologies and threats are constantly evolving. The paper proposes “a time-shifting approach to simultaneously accelerate capability delivery while maintaining traditional rigor, and achieve optimal balance between fiscal, performance, and time risks.”

Enjoy the issue!

June 2021 – Training the Future T&E Workforce

Before we launch into The Issue at a Glance, I would like to take the time to thank our associate editors and publication team at The ITEA Journal. Associate editors provide recommendations for articles, conduct interviews, and keep ITEA’s content relevant to the Test and Evaluation (T&E) profession. We are currently looking for new associate editors. If you are interested in joining our team, or would like to learn more about what is involved, please let me know at journal@itea.org.

This month in our Conversations with Experts, J. Michael Barton, Ph.D., talks with Pete Christensen. Pete reflects on a career in the DHS and DoD. His advice to us all is, “Stick with It and Volunteer for Everything.”This issue, I am especially excited about our Workforce of the Future column.

Since this is a workforce issue, we are launching this new column, but it will remain a fixture for future issues. Our goal is to capture more papers from students (our future workforce) in The ITEA Journal. These papers could come from internship experiences, research projects, or experiential learning projects. The only requirement is that a student should be the lead author on papers in this column. This issue’s column features articles written by interns from the High Performance Computing Modernization Program (HPCMP) Internship Program. This program supports undergraduate and graduate student summer internship opportunities at DoD laboratories and test centers where students learn high performance computing skills. Virginia To provides a great introduction to this column. In “Future with Computing,” Virginia provides an overview of the DoD HPCMP Internship Program. The three articles that follow all come from last year’s summer internship program and include:
• James Hughes, Selena Hamilton, and Virginia To, “Immersive Environments for Visual Analytics”
• Aiden Kenny, Mariya Occorso, and Vincent Perry, “Visual Analytic Application Toolset for Large-Scale Data”
• Sage Leone, Jaime Infantolino, Adam Childs, and Cleon Anderson, “Topic Modeling and Visualization for Test Incident Report Data”

In our first technical article, “How to Train Your Space Tester: Big Picture Challenges Facing Space Test Training,” Michael Nayak, Ph.D., Christina Straight, Evelyn Kent, and Jarred Langhals discuss four of the big challenges that need to be addressed for a possible future “Space Test Pilot School,” and their impact on US Space Force operators.

In our second technical article, William Rowell, Ph.D., Steven Oimoen, Ph.D., Leonard Truett, Ph.D., and Timothy Shelton discuss, “Applying Scientific Test and Analysis Techniques in the Agile/DevSecOps T&E Environment.” The article provides guidance and illustrative examples.

In technical article three, Vincent Perry, Mariya Occorso, Aiden Kenny, and Dale Shires discuss “Integrating Analysis and Visualization for Heterogeneous T&E Data.” Their work is motivated by the convergence of autonomous systems and big data.

In “Test in the 21st Century: Enabling a Digital Workforce to Support Tomorrow’s T&E Needs,” Alexander Hillman and Rebeka Melber discuss the need for a data savvy workforce in this new digital era. They highlight the need for leadership support in developing, recruiting, and educating the evolving T&E workforce.

In our final article for this workforce issue, Shokoufeh Mirzaei, Ph.D., Erich Brownlow, James Brownlow, Ph.D., and Robert Poulson, Ph.D. discuss new methods for modeling tracking errors in “Stochastics Differential Equation Modeling of Target Tracking Errors.” They showcase how stochastic differential equations offer a solution to modeling systems whose output is autocorrelated and also converges to a specific value over time.

March 2021 – New Initiatives in Developmental and Integrated T&E

This month’s issue focuses on new initiatives in developmental and integrated T&E. In this month’s Interview with Expert’s column, J. Michael Barton, Ph.D. talks to Matthew Reynolds, a founding member of ITEA in 1980! Matthew Reynolds discusses how T&E has changed over the course of his career to emerge into a discipline distinct from systems engineering. He highlights how ITEA shaped his career and is an excellent organization for workforce development and sharing of ideas, tools, and best practices. He concludes noting it is truly a golden age for T&E! This column is a great reflection on the rewards of a career in T&E.

We have a book review this month from Mark London, Ph.D. on Doing Bayesian Analysis, 2nd Edition by John R. Kruschke. Dr. London provides great insights into how the book might be useful to practitioners both by providing an accessible introduction to Bayesian analyses and providing numerous examples with code.

Our first article comes from Arjuna (AJ) Pathmanathan, Director of the National Cyber Range Complex (NCRC). AJ notes, “the NCRC provides Department of Defense (DoD) and other government and industry users with representative environments, a distributed network infrastructure, and other tools and services to support cybersecurity testing and evaluation, research and development, mission rehearsal, and workforce training events.” The article summarizes the origins of the NCRC and its evolution to the capability it provides today, which includes complex cybersecurity testing in real-world scenarios to include dynamic input from users.

Our next article is a combined effort from Air Force Test Center (AFTC) and Air Force Operational Test and Evaluation Center (AFOTEC). The article highlights the operational need to rethink the contractor/developmental/operational testing sequence to meet the need of modernization and new acquisition strategies designed to accelerate weapon systems development. Colonel Keith M. Roessig, Vice Commander of the AFTC, and Colonel Matthew T. Magness, Vice Commander of the AFOTEC illustrate the effectiveness of a Combined Test Force (CTF) under one chain of leadership and how it benefited the Ground Based Strategic Deterrent (GBSD) test program. They highlight that by “capitalizing on unity of command, the GBSD CTF will promote real-time combined test planning, execution, and reporting to meet developmental and operational test objectives.” This article provides a great example for potential future test force structures.

In our next article, authors from the UKL Ministry of Defence, Defence and Science Technology Laboratory (Victoria va der Byl, Richard Cawthorne, Brian Gillett, and John O’Hara) describe their innovative approach for maximizing the impact of an industry day by capturing data. Their goal is to facilitate the future of UK T&E capabilities. They designed a unique approach that leverages data captures coupled with the structured brainstorming technique, world café. The results allowed them to clearly capture in numerous charts the challenges, capabilities, and investment needs for the future of T&E. Notably, they also captured non-technology challenges. The article is a great illustration of how government organizations can bring clarity to the vast amount of input provided by industry days.

Tom Roltsch’s article on the sequential probability ratio test (SPRT) introduces a Bayesian approach for designing reliability tests. The SPRT and the Bayesian SPRT are used to estimate the mean time between failures, 50th percentile (MTBF50), and stopping rules for testing given the MTBF50 is significantly above or below a threshold level. Tom provides an intuitive explanation of the test, the mathematical basis, and a great example illustrating the stopping rules. These types of methods provide new ways of determining how much testing is enough in an integrated test paradigm.

Our final article comes from a team from the Institute for Defense Analyses. Matthew Avery, Ph.D., Jim Simpson, Ph.D., and Heather Wojton, Ph.D. examine the question of “how much testing is enough?” They compare and contrast current challenges and capabilities of the T&E community with the report that resulted from a joint Military Operations Research Society (MORS) and the International Test and Evaluation Association (ITEA) symposium in 1994. The paper serves as a great reference for publications in T&E, (citing 68 publications!), that tackle changes in statistical methods, modeling and simulation, pooling data, and costs and benefits. The article is a great summary of what has changed and what challenges still remain. The final section highlights new challenges in T&E to include software intensive systems, cybersecurity, artificial intelligence, and autonomy. A larger analysis is available directly from IDA.

I would like to thank all of this month’s authors for their excellent contributions and encourage the ITEA readership to contribute to our upcoming issues. In June, we will cover “Training the Future T&E Workforce,” September should be an exciting issue where we will tackle “Testing Artificial Intelligence and Collaborative Autonomous Systems,” and December gives us the opportunity to show off our best practices and accomplishments with “Success Stories in T&E.” Please consider contributing your work to one of

these issues.