December 2016 – Statistical Methods in T&E

The theme for this issue is “Statistical Methods in T&E,“ and the issue includes a Guest Editorial, two Inside the Beltway features, Hot Wash-Up, President’s Corner, a Book Review, and seven technical articles.

Our Guest Editorial is titled “Statistics in Defense: A Guardian at the Gate,” and David Chu, Ph.D., states that investment in statistical practice has accelerated in the T&E community. Improvements due to this focus may include a more precise sense of likely performance and a better sense of how confident we are in that estimate of performance. Statistical tools now allow the decision makers to better weigh the importance of information against the cost of the test.

Our first of two Inside the Beltway features titled “Better Buying Power, Developmental Testing, and Scientific Test and Analysis Techniques,” is authored by Darryl Ahner, Ph.D., and he states that scientific test and analysis techniques (STAT) focus on efficient and effective testing that is critical to Better Buying Power initiatives. He also proposes that STAT helps bring effective, innovative, and better understood game-changing technologies to the field to support warfighters.

Our second Inside the Beltway feature, “Rigorous Test and Evaluation for Defense, Aerospace and National Security: Panel Session Summary,” is authored by Laura Freeman, Ph.D., and she has captured a summary of the technical leadership panel presentations from the April 2016 workshop organized by NASA, DoD, and IDA. The lengthy question and answer session after the panel presentations illustrated the interest of the audience with many questions related to incorporating statistical thinking into organizational processes for organizations that plan, support, and evaluate T&E events.

In the Hot Wash-Up feature, “A Broad Role for a Different Kind of Statistical Analysis in the Testing of Complex Systems,” Wilson Felder, Ph.D., states that the evolution of complex systems can no longer be constrained to the traditional stage-gate life cycle. Some issues include the delays in handoffs between acquisition phases, the development speed of new threats, prematurely outdated requirements, and the reality of increasing digital interactions in complex systems.

The President’s Corner, written by Gene Hudgins, discusses the 33rd Annual ITEA Symposium, key guest speakers, technical tracks, and the awards ceremony. He also highlights the new ITEA Board of Directors and summarizes the State of ITEA delivered during the Symposium.

As a special feature of this issue, we have a Book Review authored by Mark London, Ph.D., for the book Engineering Philosophy written by Louis Bucciarelli. The book begins with the proposition that engineering design is essentially a social process. As the report states, T&E is, in a sense, a social-type process too because testers must complete their tasks keeping in mind many perspectives and completing many subtasks related to financial, systems engineering, safety, program management, and system effectiveness.

Our first of six technical articles “Improving Reliability Estimates with Bayesian Statistics,” by Kassandra Froncyk, Ph.D., and Laura Freeman, Ph.D., is a peerreviewed article. The authors offer a discussion of the advantages of the Bayesian approach of combining information from various subsystems and components and other sources to estimate full system reliability. An example case study for ship reliability based on the multi-mission ship program is included.

The second technical article in this issue, “Management, Mechanics, and Math (M3): An Enhanced Methodology for Future T&E of Complex Information Systems,” written by Stephen Conley and Jennie Lenig- Schreffler, proposes the M3 methodology and associated briefing format as a way to think through and create a testing approach that includes a more effective analysis. The authors have developed this approach based on art and science in recognition that information technology programs are growing in complexity.

In “GPS-based TSPI Instrumentation….Past, Present, Future – a Three-Part Series. Part 3, Future Technology,” Tom Macdonald presents the third and last segment of his three-part series. He addresses many of the future capabilities that must be realized from a number of Global Navigation Satellite Systems. For instance, the author describes the need for a receiver that integrates measurements from a variety of sources, multiple frequency GPS, improving the accuracy and reliability, and reduction in size, weight, and power.

In our fourth technical article, “Modernizing Flight Test Safety to Address the Human Agency,” Daniel Javorsek states that we must better prepare for unpredictable effects in order to continue to reduce flight mishap rates. The author proposes that complexity theory focusing on adaptive systems (like aircrew members) might influence designs of safer aerospace systems; however, the new focus must be pushed by flight test professionals with a healthy dose of skepticism.

In the fifth technical article, “M&S Requirements and VV&A Requirements: What’s the Relationship,” James Elele, Ph.D., et al, discuss methods for developing requirements for modeling and simulation (M&S) and how verification, validation, and accreditation (VV&A) of M&S – particularly early in the development phase of a program – can improve stated requirements. The authors also state that VV&A record keeping and reporting are vitally important.

The sixth technical article is our first refereed article for The ITEA Journal of Test and Evaluation. In “Frequency-Agile and Power-Scalable Multi-Band Telemetry Applications,” Tony Quach, et al, discuss the first reported frequency-agile gallium  nitride microwave monolithic integrated circuit power amplifier design for telemetry applications. The goal of this research is to further the development of miniaturized, power-efficient, and reconfigurable telemetry transmitter technology 
capable of frequency agility, power scaling, and waveform diversity to assist the T&E community.

Our last technical article of the issue, “Developing Novel Statistical Analyses to Diagnose Cognitive Impairment,” written by Nikhil Patel and Charles Hughes, Ph.D., looks at early detection and intervention of Alzheimer’s disease in order to treat patients and slow the progression of the disease. The authors developed tools that can identify whether or not a subject has cognitive impairment with an estimated 97% accuracy. Such a test can be a simple, early replacement or supplement for more complex, expensive, and less available diagnostic tests.

I hope you enjoy this last issue of 2016 for The ITEA Journal of Test and Evaluation. Please remember that a one-year membership in ITEA (and 4 issues of The ITEA Journal of Test and Evaluation) can go to any student on your gift list for just $25! By the time you receive issue 37-4 in December, the March 2017 issue 38-1 is being finalized. That theme is “Blending Systems Engineering, Lifecycle Support, Reliability, and Testing.” For the next issue (the second issue of 2017), 38-2, the deadline for submissions is just after March 1, 2017, and the theme will be “Training the Future T&E Workforce.” We have posted all themes for the remainder of 2017- early 2021 on the ITEA web site. Please provide feedback on the choice of themes, and please write early and often.

September 2016 – Changes to Developmental and Integrated Testing

The theme for this issue is “Changes to Developmental and Integrated Testing,“ and the issue includes a Guest Editorial, Inside the Beltway, Hot Wash-Up, President’s Corner Tribute to Dr. Ernest Seglie, Historical Perspective, and six technical articles.

Our Guest Editorial is titled “Developmental Test and Evaluation and Test Resource Management Center Initiatives” and C. David Brown, Ph.D. states that he has defined and implemented a set of initiatives to change the way T&E is viewed, planned, executed, and analyzed within the acquisition community. He groups his initiatives into three general categories and provides a synopsis of each initiative.

Our Inside the Beltway feature titled “Turns Out that DT&E is Really Important,” is authored by Steve Hutchison, Ph.D., and his feature discusses how we can invest in success by funding robust, independent T&E throughout the program lifecycle. In particular, he states that early in the program we should gain significant confidence that the system works – we must make the investment up front prior to granting approval for programs to begin production.

In the Hot Washup feature, “Integrated Developmental and Operational Testing as Part of the New Theory of Test,” Wilson Felder, Ph.D., states that the theme of the issue is part of a major change affecting the practice of T&E, and more generally, system development. Dr. Felder addresses new, improved, or repurposed approaches designed to get the system acquisition cycle back inside the technology development cycle.

As a special addition to this issue, we share a Tribute to Dr. Ernest Seglie. Ernie passed away on May 1, 2016. We shared a short obituary notice in the May issue of The ITEA Journal, but now we have more time (plenty of time before issue release) to highlight his contributions to T&E and his other interests. As an added tribute, we have included a reprint of his article “DOT&E Leaders Speak Out About Past and Future Challenges and Priorities” as a Historical Perspective feature from 2009.

The President’s Corner, written by Gene Hudgins, discusses our feature articles and the upcoming symposium, the symposium theme, and the list of scheduled dynamic speakers. The symposium is jointly hosted by the ITEA George Washington, Hampton Roads, Francis Scott Key, and Southern Maryland Chapters.

Our first of six technical articles “Counteracting Harmful Incentives in DoD Acquisition through Test and Evaluation and Oversight,” by Nathan Smith, Captain, United States Air Force, et al, offers an extensive review of past T&E studies and concludes that incentives drive the acquisition process. The author offers multiple recommendations that may counter the incentives that drive unsuccessful results.

The second technical article in this issue, “Table Top Wargaming: Cost Effective Planning for Cybersecurity Test and Evaluation: Or, How I Learned to Stop Worrying and Love the Six-Phase Cybersecurity T&E Process,” written by Paola Pringle., et al, offers one way to navigate the process to identify credible vulnerabilities and develop actionable requirements – the Cybersecurity Table Top (CTT). This method was first used before there were policies and guidebooks to describe the cybersecurity process. Now, it helps as one of the first cybersecurity steps to coordinate the combined acquisition, testing, and engineering looks at cybersecurity for the system under test and its interconnected systems.

In “GPS-based TSPI Instrumentation….Past, Present, Future – a Three-Part Series – Part 2, Present Technology,” Tom Macdonald, presents the second part of his three-part series. He addresses the current state of  Global Positioning System-based instrumentation. The author describes the enhancements being funded for the T&E community and  recommends specific near term actions. He sets the stage for his follow-on article next issue where he discusses future enhancements.

In our fourth technical article, “Defending the Virtual Walls: Active Cyber Defense of Weapons Systems,” Col. William Bryant, Ph.D., United States Air Force, states that defense in cyberspace must be built on a solid passive defense augmented by an active maneuvering defense. This defense in depth presents multiple barriers for an enemy to overcome, and adding human leadership in the loop will increase the defense against any enemy cyber attack. These defenses can be planned to be built into new weapons and supporting systems.

In the fifth technical article, “DOE in DT: The Place to Be!,” Francisco Ortiz and Michael Harman discuss that waiting until operational testing to apply Design of Experiments misses the option for developmental testing (DT) to provide useful and relevant data for the operational testing (OT) evaluators. At a minimum, DT provides a longer data collection opportunity, and use of DOE in DT integrates scientific principles early in the program and socializes the use of DOE across the testing and acquisition team.

Our last technical article of the issue, “Facilitating V-Model Analyses: Data Visualization for Test and Evaluation,” written by Philip Amburn, Ph.D., et al, looks at the current state of information transfer procedures and the common problems of effectively communicating the results, conveying intelligence, giving direction, or conducting analyses. The authors propose that increasing the speed and improving the insights gained from the data will be of great help in systems engineering, testing, and training.

I hope you enjoy this third issue of 2016 for The ITEA Journal of Test and Evaluation. By the time you receive issue 37-3, the December 2016 issue 37-4 is being finalized. That theme will be “Statistical Methods in T&E.” For the next issue (the first issue of 2017), 38-1, the deadline for submissions is just after December 1, 2016, and the theme will be “Blending Systems Engineering, Lifecycle Support, Reliability, and Testing.” We have posted all themes for the remainder of 2016-2019 on the ITEA web site. Please provide feedback on the choice of themes, and please write early and often.

 June 2016 – Inter-Agency and International T&E

The theme for this issue is “Inter-Agency and International T&E,“ and the issue includes a Guest Editorial, Inside the Beltway, Hot Wash-Up, President’s Corner, Historical Perspective, Workforce Development feature, and six technical articles.

Our Guest Editorial is titled “Multinational Test and Evaluation is a Reality!” and Dave Duma states that the Department of Defense has made significant use of a series of bilateral international agreements to conduct tests with allies. He also discusses the Multinational Test and Evaluation Program International Cooperation Agreement that expands and simplifies test and evaluation beyond two-party agreements.

Our Inside the Beltway feature titled “Test & Evaluation for Enhanced Security,” is authored by Steve Hutchison, Ph.D., and his feature discusses how the Department of Homeland Security has an enormously challenging mission space for which inter-agency collaboration and cooperation is essential. He summarizes the safety and security tasks for Super Bowl 50 as an example. Additionally, Dr. Hutchison describes the contributions of several components that contributed staff and technology for that effort.

In the Hot Washup feature, “Collaboration and Transparency – Two Key Ideas in Interagency and International Test and Evaluation,” Wilson Felder, Ph.D., reviews the numerous benefits of collaboration and transparency. Dr. Felder states that the benefits include more rapid advancement of technical and scientific knowledge and more efficient use of shared resources.

The President’s Corner, written by Gene Hudgins, discusses the importance of international and interagency cooperation in T&E, summarizes key components of the recent 4th Cyber Security Workshop, and invites readers to attend the upcoming 20th Test Instrumentation Workshop in Las Vegas May 10-12.

For our Historical Perspective, “National Partnership for Aeronautical Testing (NPAT) History and Background,” Michael Mastaler describes the cooperation between the National Aeronautics and Space Administration and the Department of Defense starting in the mid-1990s that helped lead to the NPAT that began in June 2007. The partnering over the past 20 or more years has led to improved and sustained national aeronautical capabilities in T&E.

As a special addition to this issue, we include a Workforce Development feature. That theme for The ITEA Journal is planned for June 2017, and this feature will give us one aspect of how the hypersonic workforce is being re-invigorated. In “Retooling the T&E Workforce – A Pilot Program to Re-Invigorate the Hypersonic Workforce through Novel Partnership with S&T,” Dan Marren, et al, discuss how the T&E workforce for hypersonics can be developed to have the right mix of people and skills to adapt to the  hypersonic testing regime that is, in some part, learned on the job. He describes a novel pilot program.

Our first of six technical articles “Australia’s First Official Use of Design of Experiments in T&E: User Trials to Select Rifle Enhancements,” by Keith Joiner, Ph.D., et al, leads the technical section of this international theme with a Peer Reviewed article on how Design of Experiments (DOE) techniques were used to test and evaluate the potential enhancements to Australia’s main military rifle. The key responses of interest were time-to-engage and probability-of-hit.

The second technical article in this issue, “Global Positioning System (GPS)-Based Instrumentation Past, Present, and Future,” written by Tom Macdonald, is part one (the past) of a three-part article. Mr. Macdonald discusses the beginning of GPS-based Time-Space-Position Information (TSPI) that was developed in the recent past for T&E applications. He summarizes the major developments in GPS that became a highly-accurate means of providing TSPI for nearly all test applications, and he sets the stage for part two in the next issue.

In “Quantifying Test Risk Using Design of Experiments,” Michael Harman, et al, present how Design of Experiments (DOE) can be used to quantify risk associated with test design changes. The advantage of DOE is that it provides upfront analysis steps that require determination of responses, factors, design types and sizes, signal to noise ratios, and replications or sample sizes. These same steps can also document consequences of changes to the program or funding.

In our fourth technical article, “Ridit Analysis for Cooper-Harper and Other Ordinal Ratings Given for Sparse Data,” Arnon Hurwitz, Ph.D., and David Stetson describe the correct method to analyze ordinal categorical data based on a method of probability scoring (or “ridits”), enabling generation of statistically valid hypothesis tests. The methods presented are often needed when observed data is sparse, as is often the case in flight tests.

For our fifth technical article in this issue, “Bootstrap Methods for Estimating Operational Availability Conference Interval,” Tom Roltsch, describes how a limited data set gathered at the beginning of a system’s lifetime can be used to make the prediction of operational availability to meet requirements. The method presented is one testers can use to make an acceptably accurate prediction.

In the sixth and final technical article, “Cost Effective Security Progression from Unclassified to System High to Multi-Level Security,” Jack Edington, et al, discuss the need on test and training ranges for a flexible architecture to dynamically support future missions. Test and training will need ranges that can handle a smooth progression from unclassified to mixed levels of classified data. The authors propose a integrated layered architecture for these needs.

I hope you enjoy this second issue of 2016 for The ITEA Journal of Test and Evaluation. By the time you receive issue 37-2, the September 2016 issue 37-3 is being finalized. That theme will be “Changes to Developmental and Integrated Testing.” For the next issue (the last/fourth issue of 2016), 37-4, the deadline for submissions is just after September 1, 2016, and the theme will be “Statistical Methods in T&E.” We have posted all themes for the remainder of 2016-2019 on the ITEA web site. Please provide feedback on the choice of themes, and please write early and often.

March 2016 – Leveraging Training and Experimentation Infrastructure and Events for T&E

The theme for this issue is “Leveraging Training and Experimentation Infrastructure and Events for T&E,“ and the issue includes a Guest Editorial, Inside the Beltway, Hot Wash-Up, Historical Perspective, President’s Corner, Tribute to Mr. Gary Carter, and seven technical articles.

In our President’s Corner by Gene Hudgins, he lists three upcoming ITEA events and then mentions that ITEA is always looking for members to participate in workshops and the symposium. ITEA needs and welcomes your participation at every level as planners, presenters/speakers, session chairs, panel members, tutorial instructors, sponsors, and participants. So, he asks that you let ITEA know if you want to help or have new ideas to enhance or expand the agendas.

As a special feature of this issue, we include a tribute to a proud T&E professional and a great friend of ITEA and the T&E profession at large, Mr. Gary Carter, retired United States Navy Commander. Gary left this Earth on October 17, 2015 due to a tragic accident.

Our Guest Editorial is titled “Overcoming Barriers: A Tester’s Perspective – Collaboration among the Test, Training, and Experimentation Communities,” and Derrick Hinton states that test, training, and experimentation communities can work together to wield a powerful force greater than the sum of its parts. In his feature, Mr. Hinton addresses the hindrances to more widespread test and training collaboration, and he offers solutions.

Our Inside the Beltway feature titled “Defense System Complexity: Engineering Challenges and Opportunities,” is authored by Kristen Baldwin and Scott Lucero. Their feature summarizes the compounding factors that drive complexity in engineering, testing, and evaluation of defense systems. The authors then present the challenges caused by complex systems and the progress the Department of Defense is making in meeting those challenges.

In the Hot Washup feature, “Systems Engineering, Reliability, Life Cycle Support and Testing: a New Theory of Test for Complex Systems, Part II,” Wilson Felder, Ph.D., reviews the purpose of the previous issue (36-4), for which Dr. Felder was the Guest Editor, to explore the consolidation of systems engineering with verification and validation and other disciplines used across the life cycle of programs. Dr. Felder summarizes the articles from Issue 36-4 and highlights articles that continue the topical coverage in Issue 37-1.

For our Historical Perspective, “Beep Beep, and We’re Off to Live-Virtual-Constructive Operating Environment: Project ROADRUNNER ‘98,” James Welshans, Ed.D., describes how Project Roadrunner developed and how it paved the way to the VIRTUAL FLAG and RED FLAG live-virtual-constructive events of today. Ironically, Project ROADRUNNER was developed in no small measure by T&E experts.

Our first of seven technical articles “Empowering T&E with Agile,” by Joshua Strain and Amar Modi, describes successfully applying the Agile methodology to T&E within the systems engineering life cycle. The authors conclude that the application of Agile can create a significant reduction in rework, provide repeatable and predictable results, and allow flexibility in testing complex programs.

The second technical article in this issue, “National Air Space Capability Test Strategy” written by John Frederick and Howard Shectman, discusses the steps included in the National Aerospace System Capability Test strategy document. This strategy document was developed to describe the approach in detail and establish integration and testing constructs to facilitate delivery of the Next Generation Air Transportation System.

In the third technical article, “Systems Engineering a Scalable Cyber Warfare Test Environment,” Ke-Thia Yao, Ph.D., et al, present how solid systems engineering and high performance computing was used to develop a low-cost, fully operable simulated cyberspace environment. The methods discussed will help ensure future cybersecurity.

In our fourth technical article, “Universal Beamforming Technology: Application and Tests,” Anand Kelkar, et al, describe the technology development and test of a steerable multi-band, multi-beam antenna subsystem suitable for airborne and ground applications. The paper describes how universal beam-forming technology was advanced to technology readiness level 6. Part of this research helped develop integration methods, troubleshooting techniques, and other engineering tools and applications.

Our fifth technical article in this issue, “Thoughts on Use of University-Scale Rocket Models to Study Launch Acoustics – A Case Study,” Krish Ahuja, Ph.D., et al, describes how progress in the field of launch acoustics can be expedited by leveraging university-scale testing. This study highlights how this type of testing may address spacecraft program risk, decrease testing (and program) costs, increase the data available to program systems engineering, and provide a testbed for feasibility evaluation of unique designs.

In the sixth technical article, “The Coordinated  Implementation of Digitally-Aided Close Air Support: An Integrated Systems Engineering and Test and valuation Approach,” Walter Ott, et al, identify an integrated systems engineering and test and evaluation approach to address the interconnected challenges related to systems-of-systems interoperability and end-to-end functionality assurance. The authors propose that their approach, as examined, is shown to be effective in addressing these challenges.

Our seventh technical article of the issue, “How to Properly Size Response Surface Method Experiment (RSM) Designs for System Optimization,” written by Wayne Adams, et al, looks at the challenges of sizing experimental designs for systems engineering and T&E when some factors are necessarily modelled at powers higher than linear. This paper describes other methods that can be used when non-orthogonal designs are required to model a system or subsystem.

I hope you enjoy this first issue of 2016 for The ITEA Journal of Test and Evaluation. By the time you receive issue 37-1, the June 2016 issue 37-2 is being finalized. That theme will be “Inter-Agency and International T&E.” For the next issue (the third issue of 2016), 37-3, the deadline for submissions is just after  June 1, 2016, and the theme will be “Changes to Developmental and Integrated Testing.” We have posted all themes for the remainder of 2016 and 2017-2018. Please provide feedback on the choice of themes, and please write early and often.