Current Issue of The ITEA Journal of Test and Evaluation

CLICK HERE for access to the FULL ONLINE issue (Members Only)

September 2017 - Test and Evaluation of Cybersecurity and Readiness

sept2017 ITEA Journal cover web

The theme for this issue is “Test and Evaluation of Cybersecurity and Readiness,” and the issue includes two Guest Editorial features, an Inside the Beltway feature, a History feature, the President’s Corner, and nine technical articles.

Our first Guest Editorial, written by Derrick Hinton et al., is “Cyber Warriors: DoD’s Most Advanced Weapons.” This is an update of one of his previous editorials. In this feature, he lists the statutory and regulatory missions of the Test Resource Management Center, and then Mr. Hinton discusses the buildout of the National Cyber Range Complex to address increased demand. He states that the cyber workforce is a make-or-break factor in establishing, maintaining, and providing operationally realistic cyber events in the National Cyber Range infrastructure. He ends his feature discussing Science, Technology, Engineering, and Math initiatives and the ultimate goal: workforce excellence.

For our second Guest Editorial, Chip Ferguson, in “The Test Resource Management Center National Cyber Range Complex and the Cyber Test and Evaluation Infrastructure,” lists the five strategic goals and implementation objectives of the Department of Defense (DoD) Cyber Strategy. He then describes the need to increase cyber range capacity and to provide a cyberspace test and evaluation infrastructure. Mr. Ferguson then covers the role and future responsibilities of the DoD Executive Agent for Cyber Test Ranges.

Our Inside the Beltway is titled, “Improving Cybersecurity DT&E through Mission-Based Cyber Risk Assessments,” and Sarah Standard and Rhiannon Hutton, PhD, state that the Deputy Assistant Secretary of Defense, Developmental Test and Evaluation (DASD (DT&E)) has the authority to assess system performance across the Major Defense Acquisition Program (MDAP), Major Automated Information System (MAIS), and other special interest acquisition programs. As one of several cyber DT&E improvement initiatives to help acquisition programs plan for and increase mission context in cybersecurity DT&E, DASD(DT&E) is encouraging the use of mission-based cyber risk assessments (MB-CRA) as a tool to plan mission-focused cybersecurity testing. Specifically, DASD (DT&E) is promoting the Cyber Table Top (CTT) MBCRA methodology that the authors discuss.

For the Historical Perspectives feature, “70 Years of Scientific Computing in the Army,” Michael Barton, PhD, and Raju Namburu, PhD, describe the evolution of these powerful tools of scientific computing, partly due to miniaturization of electronic components. They state that at Aberdeen Proving Ground (APG), testing, research, and scientific computing are inextricably connected. The authors describe the first four stages of scientific computing, and they highlight innovations at APG: the first general purpose scientific computer, the first stored program computer, the first campus area network, the first parallel computer, and more. Finally, they hint at a possible stage 5 of scientific computing at APG.

The President’s Corner, written by Gene Hudgins, mentions that this feature is his final President’s Corner. He states his appreciation to all the contributors to The Journal and to all the volunteers and contributors to the ITEA Workshops and Symposia. He then summarizes the articles in this issue and highlights the two final workshops for 2017: this year’s Symposium in October with the theme “T&E in a Time of Risk and Change” and the Disruptive Technology in Test and Evaluation Workshop in November. He also looks ahead to the Symposia in Oxnard, California in 2018 and in Lihue, Kauai in 2019.

Our first of nine technical articles, “Improving the T&E Workforce’s Understanding of the New Approach to Developing Warfighting System Cyber Requirements,” written by William Rowell, PhD, and Darryl Ahner, PhD, has a goal of helping the DoD T&E workforce understand the cyber survivability requirements development process. The key is cyber survivable system capabilities that are engineered properly through development of testable and measureable requirements, reducing the risk of not achieving the desired capabilities to an acceptable level.

The second technical article in this issue, “Cybersecurity Test and Evaluation: A Look Back, Some Lessons Learned, and a Look Forward!” written by Pete Christensen, reviews his 3-year assignment as Director of the National Cyber Range (NCR). He discusses the status at the beginning of his tour, progress made, lessons learned, and then looks to the future. He ends his article stating that the most limiting resource is the Cybersecurity T&E Workforce.

For our next article “Test Capabilities Directory (TCD) On-Line” by Denise De La Cruz and Kenneth Wheeler, the authors state that the TCD is an on-line, web accessible repository of capabilities to support Program Managers and their test planners with information covering the wide range of test capabilities within the Department of Defense (DoD). The authors cover descriptions of the TCD, search capabilities, use cases, and future versions of the TCD.

In our fourth technical article, “Advancing Developmental Evaluation Frameworks and Statistical Test and Analysis Techniques for DT&E,” George Axiotis calls on the evaluation community to further advance application of the Development Evaluation Framework (DEF) and Scientific Test and Analysis Techniques (STAT) for planning effective developmental test and evaluation (DT&E). He encourages updating and adjustment of DEF and STAT to better address DoD system maturation and performance in the development life cycle.

In the fifth technical article, “A Pathway to ‘File and Fly’ Operations for Remotely Piloted Aircraft in the National Airspace System,” George Harrison and Wilson Felder, PhD, look at opportunities and challenges in the Remotely Piloted Aircraft (RPA) arena, and they separate the challenges into the categories of policy and technical. They propose a pathway to speed RPA operations in the National Airspace System in terms of months vice years.

The sixth technical article was authored by Mark London, PhD, and Robert Schaller, PhD, with the title “Musing of a Manager: Thoughts on Training the Future T&E Workforce.” The authors explore some of the pending challenges (such as demographic and financial) to growing the future workforce of test engineers. And, they offer some key trends that seem to portend a bright future for the T&E workforce.

The seventh technical article “Interface Management in Systems Engineering” by George Gardner, PhD, describes the principal users of interface management (one of the technical management processes in systems engineering) and discusses their approach to the process. Interface management seeks to ensure smooth integration, improve interoperability, and provide for easier and cheaper technical enhancement for programs with large systems of systems.

Our next technical article “Ridit Analysis for Cooper-Harper & Other Ordinal Ratings for Sparse Data—A Distance-Based Approach” was written by Arnon Hurwitz, PhD, and discusses how probability scoring (ridit) analyses and related methods can be used to handle ordinal categorical data (OCD). The author developed a new method of comparing the results of OCD data distributions using a distance-based metric and randomization tests to derive inferences on distributions for statistical measures.

For the last technical article for this issue, “A Case Study in Understanding and Evaluating Live, Virtual, and Constructive (LVC) Command and Control Training Effectiveness,” the authors, Andrew Roberts et al., state that a credible means to evaluate the use of LVC simulation as a training tool for warfighters is lacking. They present an approach for understanding and measuring C2 training effectiveness and trainee performance, and they show how it could be applied to a large-scale Air Force Command and Control LVC training event.

I hope you enjoy this third issue of 2017 for The ITEA Journal of Test and Evaluation. By the time you receive issue 38-3 in September, the December 2017 issue 38-4 is being finalized. That theme will be “T&E for Enhanced Security.” For the next issue (the first issue of 2018), 39-1, the deadline for submissions is just after December 1, 2017, and the theme will be “Testing Using Facilities Around the World.” We have posted all themes and descriptions for the remainder of 2017–2021 on the ITEA website. Please provide feedback on the choice of themes, and please write early and often.


International Test and Evaluation Association (ITEA) | E-mail: | © 2015 | All Rights Reserved.
4400 Fair Lakes Court, Suite 104, Fairfax, VA 22033-3899 | Phone: 703-631-6220 | Fax: 703-631-6221
See ITEA on Facebook Visit ITEA on LinkIn Web Graphic Design of Warrenton, VA - Hosting and Construction by Moe Technologies (MoeTec)