41st International Test and Evalution Symposium

It was my pleasure to present the closing Keynote Address reviewing the 41st International Test and Evaluation Symposium, which was held in Huntsville, Alabama 4 – 7 November. I have adapted those notes for this article. I’d like to start by noting the Technical Director and Deputy Commander of Huntsville’s U.S. Army Redstone Test Center, Mr. Kenny Chenoweth, said he had the “coolest job in the Army,” when he spoke early in the week. In addition, the Director of T&E for the Air Force and Space Force, Mr. Chris Wilcox, claimed to have “the coolest job that exists on earth,” at the Symposium. Well, I hate to brag but as the Principal Deputy Director of DOT&E, which oversees all the services including the U.S. Space Force, I have the greatest job in the galaxy. However great jobs come with great responsibility, which was the only reason why I was not present for all the Symposium. I want to thank Barbara Ware and Kelly Horst who attended the conference all week for me so that I have some observations to share.

Once again, ITEA did an outstanding job with this year’s Symposium. Thanks to all the ITEA staff, speakers, panelists and other participants who made it a success. I am also appreciative of 300 plus attendees who took time out of their busy schedules to explore this year’s Symposium theme of T&E Challenges of Evolving Digital Ecosystems.

The Honorable Tommy Battle’s remarks opening the Symposium signaled the high caliber of the presentations to come. Each day began with presentation of an ITEA award. On day 1, Colonel Eileen Bjorkman, USAF, Retired received the Dr. Allen R. Matthews Award; on day 2, Mr. Terry Murphy of the Department of Homeland Security, T&E Directorate received the Board of Directors’ Award; and on day 3, Mr. Tim Morey of KBR and former ITEA Chairman received the Chairman’s Award. The remaining awards were presented during the Wednesday luncheon; you can read about all of them here: https://itea.org/awards/recent-recipients/. Congratulations to all recipients. Thank you for all you do for ITEA and for our warfighters.

We had excellent global participation this year, with attendees from Australia, Austria, Brazil, Korea, Sweden and the UK. This is crucial to our success because we must continue to nurture and broaden our international partnerships. My friend and colleague, Andrew Caldwell, from the UK Ministry of Defence, made some excellent points in his T&E Transformation Programme for Evolving Digital Ecosystem, including his messages on the system of systems architecture which, incidentally, supports the first pillar of the DOT&E Implementation Plan, “Test the way we fight.” He also discussed linking live and digital systems during testing. His talk included one of my favorite observations which is “When you live test you always find something you didn’t expect.”

Tuesday’s Senior Leadership panel, Challenges of Emerging Technology, led by Mark Phillips, Raytheon’s chief engineer, included panelists from the U.S. DoD,  DHS, NASA, and the FAA. The leaders, many of whom serve different customers or stakeholders, discussed the similar challenges they face when working with digital ecosystems.   During that panel, HON Dr. Douglas Schmidt, Director, Operational Test and Evaluation, shared his three top challenges:

1. The present disconnect between expectations for technology maturity and reality.
2. Recoding software late in a program’s acquisition pathway.
3. The pressing need to get the workforce up to speed on digital technology.

The same panel discussed autonomy and the acceleration of autonomy in integrated systems, and concluded:

• There is a need to understand trust and how to measure it in testing.
• We need to figure out what we are using autonomy for and how to leverage these systems in our favor
• We are not ready for autonomy systems that have not been tested – which reflects Pillar 1 of DOT&E’s Strategy Implementation Plan, “Test the way we fight.”
• We need to adapt testing to address the threat’s ability to change, thus T&E is not an end state.
• The “campaign of learning,” and “the continuum of test” are the latest buzz words or phraseology. What I think we need is a campaign of education on the value of T&E.

On Wednesday, Chris Wilcox, Director for T&E for Department of the Air Force and Department of the Space Force, in his T&E in the Digital Ecosystem remarks, provided a definition for digital ecosystem as “the complex system and environmental digital components and models functioning as a representation of the physical world.”   He also said the T&E community needs to communicate the uncertainty in the digital models to decision makers and develop plans to mitigate their associated risks; and the government needs to assess and use industry digital models and vice versa. He also provided four vignettes on how to leverage the digital ecosystem in the future. Mr. Wilcox also discussed the Joint Simulation Environment as an example of a digital range for test and training and shared the slide of the five DOT&E pillars of our ecosystem.

Representatives from the academic community proposed methods to inform, educate and attract the best talent to our industry in the panel Attracting the Next Generation T&E Workforce. Increasing our workforce is a top DoD priority. It is captured in DOT&E’s Strategic plan Pillar #5, Foster an agile T&E enterprise workforce. It is always heartening to see our school age ITEA participants at the Symposium participating in the STEAM competition and awards. ITEA works with local chapters throughout the year to provide STEM-focused scholarships and host STEAM-a-thon competitions for elementary to high school students.

During the AI & Autonomous Systems track, Chair Vincent Perry, an Army Research Laboratory Data Scientist, introduced Nilo Thomas, a DOT&E Software and AI Advisor, to discuss A Companion Guide to the DOT&E DoD Manual. Other topics included an Introduction to DoD Machine Learning System Safety, Maturing Adversarial Testing in T&E for AI/ML Systems, and A Natural Language Processing Pipeline for Evaluating Test Incident Reports. All of the speakers expressed a desire to get ahead of future T&E needs for AI and autonomous systems, including testing and experimenting with pilot programs. These AI capabilities are supporting the warfighter on oversight  and helping testers do their jobs more quickly and efficiently.

The Model vs. Digital Shadow vs. Digital Twin Cycle Phases in T&E track chaired by Erwin Sabile of Booz Allen Hamilton continued the work started at the 2022 ITEA Cybersecurity Workshop. The workshop participants published a definition of digital twins in T&E in the March 2024 issue of the ITEA Journal. They used this year’s Symposium to forge ahead with expanding that definition to four phases: (1) Design & Development, (2) Integration, (3) T&E, and (4) Operations & Sustainment. DOT&E’s Nilo Thomas revealed DOT&E is working on rolling out T&E policy and guidance in five major areas, to include AI and autonomous capabilities. If you are interested in helping this team with authoring articles or expanding its Use Case Scenarios, contact Erwin Sabile.

In the Emerging Technologies & Systems track, Chair Dr. Mark Kiemele, President & Founder of Air Academy Associates, facilitated a discussion with representatives from DOT&E, Johns Hopkins University Applied Physics Laboratory, and University of Maryland Applied Research Laboratory for Intelligence and Security. They shared presentations on T&E automated tools, techniques and lessons learned related to emerging technology and operational environments. Topics included an independent automated verification and validation testbed, a mathematical framework for CJADC2, and a tool for conducting Mission Based Risk Analysis.

Dr. Keith Krapels, Director, U.S. Army Space & Missile Defense Command Technical Center, challenged us to change sensing so we do not give away our location, by using  optical versus radio frequency sensors. He asked us to work “hand in glove,” to help our nation move forward. John Frederick, Manager for Verification & Validation Strategies and Practices Branch, at FAA’s William J. Hughes Technical Center, called Validation & Verification professionals superheroes. I want to say I could not agree more. Following John Frederick, Thomas Tomaiko, T&E Executive for the Transportation Security Administration, introduced TSA and its Test and Evaluation Division (TED), summarized digital transformation initiatives at DHS and TSA. Mr. Tomaiko  described the current and future state of TSA TED T&E digital ecosystems. He highlighted improvement initiatives including requirements development and feasibility studies and related development and training efforts tailored to meet TSA T&E needs.

This was followed by three panels, the first being An Acquisition View of T&E, and I want to make two key points from it. The first is What We Do Matters. The second point is we need to change how we approach T&E. The way DOT&E thinks T&E should be changed is outlined in the Spring 2024 issue of the Naval Engineers Journal    https://www.navalengineers.org/Publications/Naval-Engineers-Journal. DOT&E staff authored this issue, and I highly recommend reading it. If ordering a copy of this journal is not practical, the DOT&E implementation and strategy plans that can be found at https://www.dote.osd.mil/Publications/ are also a good roadmap of DOT&E’s future plans. Ken Senechal chaired the Hackathon 2.0 Outbrief. A hackathon is an event where people engage in rapid and collaborative engineering over a relatively short period of time. This one focused on turning the developmental T&E as a Continuum concept into reality. The highly successful event included more than 200 participants from the Office of the Secretary of Defense, the services, industry, and academia. The panel itself included several of the key hackathon contributors. Look for 3.0 next year.

We were reminded that next year the Symposium will be held in Destin, Florida in November and it promises to be the zenith of ITEA’s 45^th anniversary celebration. The newly elected ITEA Board officers were also announced: Chairman, Dr. Michael Barton; Vice Chairman, Mr. Erwin Sabile, Secretary, Ms. Cathy O’Carroll; and Treasurer, Mr. John Rafferty. I want to congratulate them, along with all our young and our more experienced award winners. We always need innovative ideas to continue to produce the information our warfighters, decision makers, and the public deserve. So, we need everyone engaged in Symposiums such as this one.

In closing, ITEA has continued to enlarge its role as the major player in T&E by bringing together some of the finest academia, industry and government thought leaders to this year’s Symposium. All of you should be proud to support this great organization’s work which helps keep our warfighters and the world order that we enjoy, safe. I look forward to seeing you in the future. Until then, may you be well and safe travels.

Biographies