Editorial – ITEA Journal – December 2024

This edition of the ITEA Journal first celebrates the ITEA Award Winners with our first article by Associate Professor Robin Poston who chairs the ITEA Awards Committee. The article covers all the major award winners announced at the ITEA Annual Symposium in Huntsville in November. Please check out who was honoured with lots of smiling faces and photographs from the event.

Our second ITEA business article comes from Dr Raymond O’Toole summarising the ‘Best Outcomes of the ITEA Annual Symposium’ in Huntsville during the November US election week. It has been an ambition of ITEA to capture the summary of the ITEA conference since Mr Duma used to do these verbal summaries at the symposium close. We are grateful for Dr O’Toole for continuing the tradition and for publishing his summary for posterity and wider awareness.

Our third ITEA business article covers the ‘Best 2024 Research Articles in T&E’ as assessed by ITEA Publications Committee and reported by me (Dr Keith Joiner). Here we cover not only the winner of the ITEA Publication Award but the equal runners up and the third place article. The Committee is widely read in T&E and so they sought and tried to rank the most significant articles in T&E from the many hundreds written across the twelve-month qualification period. Regardless of your role in T&E, we encourage you to read these influential articles across the Christmas period.

Our first peer-reviewed research article comes from Australia where they are undergoing somewhat of a renaissance in developmental capabilities in the area of new robotic autonomous systems. Examples include the Speartooth [1], Ghost Bat [2] and Ghost Shark [3] vehicles. This article from Australian postgraduate students, Yu Ning, Sebastian Russell, James Flawith and Jake Nicholas, is titled, ‘Evaluation of Technological Readiness in Mixed Maturity Sub-Systems of Large Uncrewed Underwater Vehicles’ (LUUVs). According to the authors, ‘TRLs are a commonly used framework to examine developing technologies and products in the context of project management and system engineering disciplines to gauge risk and inform decisions.’ They employ a literature review and Technological Readiness Levels (TRLs) to examine ‘the evolution of LUUVs’ and the ‘influx of new and mature technologies.’ Such approaches can assist Australian Defence navigate their early ‘pre-second-pass’ life cycles which are analogous to the US DoD phase of ‘Technological Maturation and Risk Reduction.’

The second research article is by Dr Jose Alvarado and Professor Thomas Bradley of Colorado State University concerning, ‘A Case Study-Based Assessment of a Model-Driven Testing Methodology for Applicability and Metrics of Model Reuse.’ This research continues earlier research by these authors whereby it ‘expands upon the existing grey box model-driven test design (MDTD) approaches by leveraging model-based systems engineering (MBSE) artifacts to generate flight test scenarios.’ They introduce ‘a methodology developed by the Air Force Operational Test and Evaluation Center (AFOTEC) to guide the operational test and evaluation (OT&E) test planning process using an MBSE framework.’

I was fortunate to attend the Old Dominion University for the last five months teaching for the US Fall Semester and attending what conferences I could involving T&E. The US has a rich offering of events educating best practices in T&E by associations like ITEA and the National Defense Industry Association (NDIA). My only criticism of these association symposia is that none of the four I attended provided ready access (yet) to the rich presentation material. I wondered at the many attendees taking screenshots of the presentations but afterwards, I discovered why. I am not sure if this limitation is a fear of litigation through inadequate screening of slides, digital storage, digital security (i.e., to limit access to attendees) or a lack of administrative capacity. Whichever organisation can address this decline will be more influentially competitive in the future. This deficiency needs to be addressed if attendance at the events is to speed up best practices, avoid the experience being ephemeral, and reward the institutions that invest in sending delegates.

By comparison to the US diet of T&E-related symposia, had I stayed home in Australia, I would have had just one week in September of the Systems Engineering T&E (SETE) conference coupled with a one-day Defence T&E symposium. Several US experts in T&E have made the pilgrimage ‘down-under’, such as Chris Collins and Sarah Standard in recent years. This year it was our outgoing ITEA President, Mark Phillips, who attended SETE. After much discussion and planning, next year in May 2025 our UK chapter will run a symposium event for T&E, while the next Australian SETE event is expected in March 2026. I encourage anyone wanting to vary their diet on advances in T&E to consider attending these events ‘outside the continental US (OCONUS).’ The presentation slides might even be available to attendees!

Keeping abreast of the growing alignment of Australia, UK and US in an ‘AUKUS’ technological and test partnership is important for Defence testers who follow advanced strategic capabilities. For example, consider the very recent and significant agreement on hypersonic testing [4]. The 2021 AUKUS Pillar One agreement concerned nuclear-powered submarine agreements but the agreement is now extending under ‘Pillar Two’ [5]. According to Dolan [6] there are ‘eight functional areas that comprise Pillar 2: hypersonic missiles and long-range weapons, artificial intelligence, undersea capabilities, advanced cybersecurity, quantum technologies, autonomous weapon systems, information sharing, and innovation’ with the measure of success being in balancing China. The impetus to expand testing of advanced capabilities could include Canadian, New Zealand, Japan and South Korean testers as there is active debate about including such allies in Pillar Two [7-10]. A great impartial read on this topic is by Markowski, et al. [11] who concludes, “that the AUKUS arrangements stand to provide less scope for free-riding by the parties, while giving them greater incentive to invest in interoperable military capabilities, thereby creating more contingent (real) deployment options for the future, US-led combined military operations, to be exercised by the parties in response to emergent military contingencies.” The Markowski, et al. [11] article maintains membership requires appropriate investment sharing which I would argue extends to a critical mass of expertise in T&E at a base level and then within each new technology and domain.

With the above T&E obligation in mind, the third peer-reviewed research article is by me, (Dr Keith Joiner), on ‘Australia’s Pentagon Wars Moment.’ Following concerning press coverage [12], recent testimony by flight testers regarding an MRH-90 helicopter crash [13] is analysed against previous audits and inquiries into Defence procurement in Australia and the accounts of the so-called US Pentagon Wars period of the early 1980s [14, 15]. The article concludes that new legislation before the Australian Parliament [16] is critical to reforming T&E to account for new technologies and advanced capabilities.

The remaining research articles are in the chronological order of the symposia where they debuted.

Our fourth research article is one that remained from our DATAWorks collection because we had not finished editing before the September edition. This article is from the US Westpoint Military Academy by Cadet Felter, Instructor Sustaita, and Associate Professor Starling on using ‘Synthetic Data for Target Acquisition.’ Testers are concerned by the representativeness of test data and maybe reluctant to use synthetic data to help train AI machine learning. However, research threads like Professor Freeman’s across the last four years ([17-20] have shown the importance of factorial balance in the training and test data. As such the Felter et al. article the results serve to proffer that synthetic imagery can be cost-effectively used to augment real imagery to produce more effective models, especially early in prototype or minimum viable product (MVP) of an AI ML application.

The fifth research article is a review by me (Dr Keith Joiner) of a presentation track from ITEA’s Cybersecurity Workshop [21] in Orlando in September. The track was chaired by Dr Mike Shields and titled ‘Fuzzing to Find Unknown Vulnerabilities’ with four presentations:

• ‘Vader Modular Fuzzer (VMF) Now and in the Future’ by Josh Weader, Deputy PM and Technical Director, Draper Lab
• ‘G-QEMU (GQ),’ Now and in the Future, again by Josh Weader
• ‘Hybrid Fuzzing,’ by Kaleb Senator, Research Scientist, GTRI
• ‘Measuring Fuzzer Performance,’ by Dan Koch, TRMC and Vigilent Cyber Systems

The review provides an introduction to fuzz testing developments, overviews the ontology, and gives links to key literature. We hope to follow up this review with a write-up by the presenters themselves, as fuzzing is an important new T&E approach in industry and Government to find vulnerabilities and build digital sovereignty through better cyber-resilience. The effort to provide coverage of fuzzing in our Journal is due to its effectiveness, rapid pace of development, and because the Test Resource Management Centre (TRMC) are providing enhanced fuzz test benches for the US DoD.

NDIA held a workshop on 18 October at Mitre’s Reston Buildings in Washington. There is particular excitement that Generative AI along with digital engineering, can assist acquisition bureaucracies speed up their capability development. Themes explored how AI in a digital engineering environment can:

1) reduce person-to-person document transfer, and

2) accelerate software engineering.

The workshop covered many tools developed to assist acquirers speed up their work. Project Gordian is examining contracting officer support. Contract writing tools include CON Write, CON IT, CON Clause, and AcqBot. The latter helps with requirements writing including for Integrated Logistics Support (ILS) and testers. Such tools can lack complex reasoning, contextual understanding, and regulatory nuance; however, with qualified human teaming they offer reduced development times. A tool called Atlas has been developed to help market research and request for proposal (RFP) development. A sister tool is Tally to evaluate proposals using a rubric based on the RFP, including autogenerating the evaluation portal and initial evaluation report. Also highlighted by Trac Bannon was recent work by MITRE Corporation concerning the use of AI in the software development cycle, as shown in Figure One [22] and the recent article Bannon and Laplante [23]. Bannon’s team are developing decision-support tools and guidance to help build calibrated trust in GenAI-augmented Software Development Life-Cycle (SDLC) tools by software developers to address the quality and security needed. We hope to have an article from the team in a future edition explaining the implications on software testers.

Figure 1: Mitre Corporation diagram concerning AI use in software development [22]

This NDIA conference was held in Norfolk, Virginia. The technical tracks in this jam-packed conference[1] covered the following 12 themes, six tracks at a time, with an incredible 156 presentations, each lasting at least 30 minutes:

• Artificial Intelligence
• Education & Training
• System Security Engineering
• Mission Engineering & Specialty Engineering
• Digital Engineering and its Environment
• Model-Based Systems Engineering
• Test & Evaluation
• Physics-Based Modeling and Simulation
• Integrated Program Management
• Adapt (Agile or Hybrid)
• Safety & Environmental Engineering
• Architecture / Modular Open Systems Architecture (MOSA)

Coming from this conference our sixth research article covers two of the tracks above with impacts into many others. It is by Maryam Gracias and Dr Erika Gallegos, concerning ‘Transitioning Perspectives: Agile and Waterfall Perceptions in the Integration of MBSE within Aerospace and Defense Industries.’  According to Maryam and Erika, ‘The Aerospace and Defense (A&D) industry, characterized by its complexity, high-stakes, and stringent regulations, has traditionally relied on Waterfall methodologies for project management’ but . ‘Agile methodologies, emphasizing flexibility, collaboration, and iterative development, have emerged as promising alternatives.’ Using comprehensive surveys they explore the integration of Model-Based Systems Engineering (MBSE) in the sector to ‘understand the impact of MBSE on project management processes, efficiency, and time-to-market in Agile and Waterfall environments.’ Such research is important to understand if MBSE and digital engineering environments are the bridge that finally facilitates agile approaches to juxtapose hardware and software development with sufficient rigour to speed up traditional waterfall techniques around MVP [24]. In a similar vein, we hope to have a future technical article on the work of Dallas Rosson who also presented at the NDIA Conference on a structured hybrid approach known as ‘Systems Engineering Focused Agile Development (SEFAD)’ which has been trialled in about six different US DoD projects.

[1]  https://www.ndia.org/events/2024/10/28/systems-mission-engineering-2024

Several presenters at ITEA’s annual symposium in November promised technical articles, especially those who persistently attended their technical posters in the sponsored break-out area throughout the three days. Also, we hope to have a future write-up from Mr Kenneth Senechal of the US DoD Developmental T&E and Assessments. Ken presented keynote addresses at the NDIA conference and the ITEA Symposium on a major experimental workshop by the US DoD to evolve their acquisition through digital engineering, known as ‘Hackathon 2.0.’ We encourage all such presenters to follow through for a wider readership of their work (see https://itea.org/submissions/ ).

Have a merry Christmas and please look for more on T&E research and practices in our 2025 editions. I am ably supported in this role by the members of the ITEA Publication Committee and two excellent sub-editors, John Haman and Viruben Watson. We are always interested in widening that network of editors to better cover T&E, especially if you can cover specific ITEA and NDIA events throughout 2025; please email the three of us (journal@itea.org) if you are interested.

[1] T. Fish, “Latest Speartooth trials complete and production-ready,” in Australian defence Magazine, ed. Online: ADM, 2023.

[2] V. Insinna, “US Air Force to start new experiments with Boeing’s MQ-28 Ghost Bat drone,” in Breaking Defense, ed. Online: Breaking Media, 2022.

[3] R. Manuel, “Anduril Ghost Shark Submarine Drone Arrives in US,” in The Defense Post, ed. Online: X, 2024.

[4] (2024). AUKUS Partners Sign Landmark Hypersonics Agreement. Available: https://www.defense.gov/News/Releases/Release/Article/3966986/aukus-partners-sign-landmark-hypersonics-agreement/

[5] P. Parrish and L. A. Nicastro, “AUKUS Pillar 2 – Background Issues for Congress,” in Targeted News Service. Washington, D.C., ed. Proquest: Congressional Research Service, 2023.

[6] C. Dolan, “AUKUS Pillar 2 – Technology, Interoperability, and Advanced Capabilities in the Evolving Trilateral Security Partnership,” in National Security in the Digital and Information Age, S. Burt, Ed. London, United Kingdom: Incitech Open, 2024, pp. 121-140.

[7] S. Carvin and T. Juneau, “Why AUKUS and not CAUKUS? It’s a Potluck, not a Party,” International Journal, vol. 78, no. 3, pp. 359-374, 2023.

[8] R. Steff, “The strategic case for New Zealand to join AUKUS Pillar 2,” Australian Journal of International Affairs, pp. 1-9, 2024.

[9] (2023). Tilting Horizons: The Integrated Review and the Indo-Pacific. Available: https://publications.parliament.uk/pa/cm5803/cmselect/cmfaff/172/summary.html

[10] A. Wyatt, J. Ryseff, E. Yoshiara, B. Boudreaux, M. Black, and J. Black, “Towards AUKUS Collaboration on Responsible Military Artificial Intelligence.,” RAND Australia, Online2024, Available: https://www.rand.org/content/dam/rand/pubs/research_reports/RRA3000/RRA3079-1/RAND_RRA3079-1.pdf.

[11] S. Markowski, R. Wylie, and S. Chand, “The AUKUS agreement: a new form of the plurilateral defence alliance? A view from downunder,” Defense & Security Analysis, vol. 40, no. 3, pp. 430–449, 2024.

[12] L. Gwynn and L. Lavelle, “Retired Australian Defence Force test pilot apologises to families of airmen killed in Taipan helicopter at inquiry in Brisbane,” in ABC News, ed. Online: Australian Broadcast Commission, 2024.

[13] “INSPECTOR-GENERAL AUSTRALIAN DEFENCE FORCE INQUIRY INTO THE CRASH OF A MRH-90 TAIPAN HELICOPTER IN WATERS NEAR LINDEMAN ISLAND ON 28 JULY 2023,” in Inspector-General of the Australian Defence Force, The Brisbane Convention & Exhibition Centre, Plaza Level Room P11, Merivale Street, South Brisbane QLD 4101 ed. Online: Australian Government, Defence, 2024.

[14] J. G. Burton, The Pentagon wars: Reformers challenge the old guard. Naval Institute Press, 2014.

[15] R. Benjamin, “The Pentagon Wars,” ed. United States: HBO, 1998, p. 1 h 43 min.

[16] Defence Capability Assurance and Oversight Bill 2024, P. N. 47, 2024.

[17] E. Lanus, L. J. Freeman, D. R. Kuhn, and R. N. Kacker, “Combinatorial testing metrics for machine learning,” in IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW), Online, 2021, pp. 81-84: IEEE.

[18] T. Cody, E. Lanus, D. D. Doyle, and L. Freeman, “Systematic training and testing for machine learning using combinatorial interaction testing,” in IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW), Online, 2022, pp. 102-109: IEEE.

[19] L. Freeman, “Best Practices for Addressing New Challenges in Testing and Evaluating Artificial Intelligence Enabled Systems,” ed. https://itea.org/professional-development/webinars/: International Test and Evaluation Association, 2023.

[20] J. Chandrasekaran, T. Cody, N. McCarthy, E. Lanus, L. Freeman, and K. Alexander, “Testing Machine Learning: Best Practices for the Life Cycle,” Naval Engineers Journal, vol. 136, no. 1-2, pp. 249-263, 2024.

[21] “Exploring Cyber Test Ranges: Past Present and Future Perspectives,” in Cybersecurity Workshop 2024 Orlando, Florida, 2024, vol. Abstracts only, Online: International Test and Evaluation Association, 2024.

[22] T. Bannon, “Infusing Artificial Intelligence Into Software Engineering and the DevSecOps Continuum,” Computer, vol. 57, no. 9, pp. 140-148, September 2024.

[23] T. Bannon and P. Laplante, “Generative AI in the Software Development Lifecycle,” Computer, vol. 57, no. December, pp. 27-34, 2024.

[24] B. Kazakevich and K. F. Joiner, “Agile Approach to Accelerate Product Development using an MVP Framework ” Australian Journal of Multi-Disciplinary Engineering, 2023: submitted.