T&E in Practice: Applying the Defined Discipline

Systems engineering is a relatively young human activity, with genesis stories clustered around British and US defence-industries in the 1940s. Bell Telephone Laboratories is credited with coining the term, with practice maturing through the Cold War and the space race [1]. Evolving from systems engineering, Test and Evaluation (T&E) is still a young discipline, with less than a century for the accumulation of knowledge toward practice. Compare that with the thousands of years of accumulated knowledge underlying stone masonry, evident in the comparatively massive lexicon [2] that carries its practice. Little wonder that the activities and value of T&E remain inconsistently understood in contemporary industry, frequently conflated with generic notions of testing, treated as a late-stage project obligation, or justified by nothing more than superficial reference to organisational policy.

Recent work [3] addressed this gap to establish T&E as a systems engineering discipline defined independently of application. That foundation resolves long-standing, problematic definitions in the literature, and this article now addresses the complementary problem: how the discipline is understood by practitioners and stakeholders. The need for this clarification lies in the changing nature of systems. When systems were complicated, as distinct from complex, [4, 5] the inconsistent framing of T&E was of little consequence because the assurance question was resolvable. [6] The static nature of a complicated system lends itself to a more-is-better approach to T&E. [7] However, contemporary systems increasingly exhibit tight coupling, dynamic configuration, [5] emergent behaviour, and blended human-technical control, [8-10] such that behaviour cannot be inferred from component performance alone. [7] T&E needs to evolve in line with the systems it serves, yet the T&E professional discourse has not kept pace with the systems it now supports.

This article is scoped to professional practice: how individuals describe, scope, and apply T&E within projects and engineering teams. It does not revisit the ontological argument for T&E as a discipline, which was addressed previously, [3] nor does it address the organisational and governance implications of that discipline, which remain for future work. Three observed sources of confusion are addressed in turn: the language used to describe T&E, the proliferation of project-specific labels attached to its two forms, and its relationship with project delivery frameworks.

This article builds on the definition of T&E presented previously in this Journal: [3]

T&E is the deliberate collection of objective evidence, using the scientific method, to assess baselined attributes of an integrated system, in order to inform decisions.

Three implications follow directly. First, selectivity is intrinsic to the discipline: T&E addresses attributes where uncertainty is material to a decision, it is not an exhaustive census of system behaviour. Second, T&E is independent of the tools used to generate its evidence, whether test, inspection, demonstration, modelling, or analysis. Third, following directly from the framework established in Defining T&E as a Discipline, [3] the requirement for controlled variation of a dependent variable yields exactly two fundamental forms of T&E: Developmental T&E (DT&E), where the system itself is varied while its operating context is held stable, and Operational T&E (OT&E), where the operating environment is varied while the system configuration is held stable.

These three implications, selectivity, tool-independence, and reduction to two foundational forms, are the basis this article builds on. They are argued, with their theoretical grounding, in a prior issue; here they are the starting point for practice, not the destination.

3.1.Overloading the word Test

In everyday usage, test is a broad and ambiguous term. Students sit tests at school; scientists test hypotheses to advance theory; quality teams test products against a specification; a car can be test-driven before purchase. Each use is legitimate in its own context, but each differs in purpose, method, scope and standard of evidence. A lay person brings their personal experience of test into a systems program, typically diluting the discipline and rigour that distinguish T&E within systems engineering.

When T&E is reduced to generic concepts of test rather than being recognised as a distinct discipline, it is conveyed as backward-looking overhead, or worse, a casual sampling of individual experience, like the casual test-drive of a car. That perception is not malicious; it is the lived experience of wider stakeholders: the project director, the scientist, the system operator. These stakeholders bring their perspective with them and that is entirely valid on its own terms. Denying it invites argument that is distracting. From a lay perspective, testing can be an unnecessary cost, an activity dedicated to finding fault, one that delays a team otherwise focused on producing components and integrating a system that works. Through that lens, T&E practitioners would be perceived as standing outside the team, working contrary to successful team outcomes. A former US Department of Defense (DoD) Principal Deputy for Developmental T&E addresses this when he writes that T&E professionals should not be gatekeepers, but partners providing constructive feedback [11]. Crediting the idea to Crispin and Gregory [12], Hutchison criticises the entrenched gatekeeper culture because it does not support the success of the parent project.

Where a system is complicated enough to require systems engineering, and increasingly where it becomes complex, T&E carries a disciplined meaning distinct from casual usage. As established, [3] T&E requires a baselined expectation, an integrated system with a defined boundary, and the disciplined collection of evidence against that expectation, to apply the scientific method. Recognising this tight definition draws a boundary around what constitutes T&E, distinct from other activities described as “test”, to prevent the discipline from being diluted by activities that serve other purposes. Examples of test that are not T&E include its use in science, where testing is exploratory, developing theory rather than evaluating a defined system; a car’s test-drive advertises features rather than generating evidence against a baseline. Counterintuitively, a tight definition identifies activities that are not T&E, enabling T&E to stick to its swim lane, concentrating effort and value rather than contesting ownership of a common word.

3.2.Test Is Not the Value Proposition

A second problem in language arises when test is treated as the value T&E delivers. When practitioners describe their work by test points executed, events conducted, or scenarios exercised, the focus shifts from outcome to activity. Test becomes something a project endures, rather than something it seeks. The US Government Accountability Office (GAO) rejected this practice when reviewing weapon test programs that managed test activities completed rather than anchoring in knowledge attained. [13] INCOSE’s measurement guidance suggests that an individual measure only earns its place if it traces back to a decision someone actually needs to make. [14] Counting effort doesn’t count.

What projects actually require from T&E is assurance: confidence, grounded in evidence, that an integrated system will behave as expected under defined conditions. For design engineers, assurance provides trusted feedback on design and integration decisions. For system owners and operators, T&E provides trusted feedback on operational capability and fit within the wider organisation. Establishing this feedback loop is consistent with Aven [15], who similarly characterises risk assessment as existing to support decisions under uncertainty, rather than as an end in itself.

Reframing T&E around assurance explains why it exists at all. Where no uncertainty matters and no question about system attributes remains, there is no requirement for T&E. Where uncertainty exists and the consequences are meaningful, assurance is sought as the basis for a decision. T&E is a means to that end and the analogy to the drill-bit, recently attributed to Levitt, is apt. [16] No one wants T&E. What they want is assurance. This reframing sharpens thinking and communication, moving discussion away from compliance and toward purpose: why a T&E activity is necessary, why a particular tool was selected, and why some questions warrant evaluation while others do not.

3.3.Process Versus Tool

A third source of confusion is a historical quirk in which the name of one of the tools used by T&E has come to stand for the discipline itself. Test is not what T&E is, test is one of the tools T&E uses; alongside modelling, demonstration, inspection, and analysis. [3] Each tool is capable of contributing evidence when applied appropriately. What distinguishes T&E is not the tool selected, but the scientific method under which the tools are applied to provide a disciplined evaluation of an integrated system against a baselined expectation, answering a specific assurance question at the required fidelity. T&E can be conducted using tools other than test.

Different assurance questions demand different tools, different fidelity, and can tolerate different degrees of intrusion on the system under evaluation. They are valued by stakeholders differently, leading to differences in allowable budget to answer the question. Treating test as synonymous with T&E encourages tool-driven practice, to reach for the same instrument regardless of the question, an example of framing assumption bias identified by Kendall [17]. Recognising T&E as a process enables different tooling options, empowering practitioners to exercise judgement in selecting and combining tools to scale the assurance delivered to the decision at hand. Each of the tools recognised by NASA [18] are defined as equal methods, with tool attributes to be considered against the assurance requirement. Deliberate tool choice provides options, recognising that not all assurance questions require the fidelity and overhead of testing. Figure 1 summarises the three distinctions using logic operators.

Figure 1. T&E is not colloquial test. T&E provides assurance. Test can be a process or a tool.

The distillation of T&E to two fundamental forms, DT&E and OT&E, [3] provides a stable foundation though practitioners rarely encounter T&E in such simple terms. Acceptance T&E (AT&E), Preview T&E (PT&E), Built-In T&E (BIT&E), Qualification T&E (QT&E), Assembly and Integration T&E (AIT&E), Hardware-in-the-Loop (HITL), Initial OT&E (IOT&E), Live-Fire T&E (LFT&E), Follow-on OT&E (FOT&E), and a growing list of project-specific titles circulate across programs, each apparently implying its own method or rule. Timing sequence within a project can be the only difference between some of these, and Figure 2 places a representative set of these titles in a typical sequence, categorised against the two forms of T&E.

Figure 2. Derivative forms of T&E are conducted sequentially across the project schedule, but each reduces to one of two forms, determined by the locus of the dependent variable: internal to the system (DT&E) or in the operating environment (OT&E).

Examined against the scientific foundation of the discipline, these derivative titles are not additional forms of T&E. They are overlays imposed by contractual, organisational, or programmatic requirements; indicative of a stage in a project schedule, a transfer of project authority, a different source of funding, a specific decision point authorising further development, or similar. These distinctions are important to the project, but they do not alter how evidence is generated or interpreted to assure the underlying system. New labels are coined by projects for project purposes, and the list is not exhaustive. It will keep growing. Defense Acquisition University’s Test and Evaluation Management Guide [19] and the US DoD Directorate of OT&E, Test and Evaluation Enterprise Guidebook [20] published seventeen years apart, illustrate this. Both guides contain derivative labels specific to the US DoD, with the later Guidebook carrying a larger vocabulary of derivative labels; CT (Contractor Testing) and CVPA (Cooperative Vulnerability and Penetration Assessment) being among the additions. Comparisons between organisations would illustrate a different set entirely.

For practitioners, this conceptual reduction allows recognition of project-specific terminology alongside the T&E discipline; it allows alignment of evaluation design with the assurance question being asked rather than the project label attached to the activity. The conceptual reduction avoids unproductive arguments about “types of test” that arise from local convention rather than any real difference in method. Returning to the location of the dependent variable, inside the system or in its environment, reconciles the proliferation of titles encountered to a single, coherent framework, independent of domain, lifecycle model, or organisational convention.

Two of the most common derivative labels illustrate the reduction well. PT&E is a form of OT&E: the system configuration is held fixed, as offered by the vendor, while the variable of interest is the prospective client’s operating environment. AT&E, by contrast, holds the environment fixed at the contracted specification, to examine the latest iteration of the system for acceptance; where successful, it is properly understood as the final iteration of DT&E.

For a T&E practitioner moving between projects, between employers, or between a traditional acquisition program and a rapid, iterative project, this conceptual reduction highlights their transferrable skills. They do not need to relearn T&E at each transition as only the labels change, along with the cadence and fidelity appropriate to context.

When lay observers or T&E practitioners describe T&E itself as Agile, they are conflating a project delivery method with the T&E discipline it should merely host. Agile, capitalised noun, denotes software delivery approaches, now also project methodologies, derived from the Agile Manifesto. [21] Conversely, agile, lower-case adjective, describes a management philosophy undertaken in an incremental manner so as to be responsive to change, whether formally Agile or not. As projects increasingly interleave T&E activity with other requirements to speed up delivery, projects risk corrupting the T&E discipline when it is subsumed into the surrounding delivery framework, rather than being hosted as an independent discipline serving the project. As a subordinate activity, testing can be compressed, deferred, or cut when the project demands it. Especially when T&E is seen as misaligned with project objectives, then rationalising it away starts to sound like good project management.

Agile makes this acute for a structural reason as T&E requires a baselined expectation be held stable while the variable of interest is exercised. [3] Yet, genuinely Agile projects continuously increment requirements to satisfy evolving design demand. That is difficult to reconcile with T&E as the baseline keeps changing, but it is manageable where the T&E team is engaged as a cooperative partner. That requires an understanding of the difference between the project methodology and their role to implement T&E to provide assurance. The T&E team then incrementally re-baseline deliberately as the system matures. If the T&E team are subordinated, the impact of project methodology eroding T&E discipline is unlikely to be recognised, and the low quality assurance provided by T&E won’t be valued.

The label Agile attracts this risk more visibly than most, since being seen as agile is treated as a positive attribute in its own right, worth claiming whether or not the practice behind it is real. The United States GAO has twice found that US DoD programs describing themselves as Agile fail to meet the practices the label implies, falling well short of the iterative development cadence the methodology assumes. [22, 23] A label claimed without the practice behind it is evidence of the label’s pull, not of the practice itself. That same pull toward appearing agile is what makes trimming a discipline like T&E look like virtuous cost cutting rather than the loss of assurance.

Delivery frameworks change the cadence, timing, and intensity of T&E, not the discipline itself. Projects often label different periods of T&E distinctly, in line with their scheduling. An Agile project may call for more frequent, small-scope, lower-fidelity evaluations to keep pace with rapid iteration; a more traditional program may concentrate T&E effort into fewer, large-scope, higher-fidelity events. T&E may look agile in the casual sense, being responsive and scalable, but that responsiveness is intrinsic to serving a project, not a consequence of adopting Agile methods. To describe T&E itself as Agile is to misattribute a property of the project to the discipline that serves it, and the same misattribution can occur under any delivery framework. Swap Agile for waterfall or spiral and the erosion happens the same way, under a different name. Preserving the distinction from the host project protects the coherence of T&E as a discipline, rather than letting it be absorbed into whichever project delivery fashion happens to be current.

Understanding T&E as a discipline grounded in the scientific method gives practitioners a stable anchor for reasoning: what is the assurance question, what baseline expectation is being evaluated, where does uncertainty genuinely matter, and how can it be reduced? This framing distinguishes activities that contribute to assurance, from activities for other project purposes, leading to clear communication of the concepts.

Projects vary widely in structure, from tightly staged programs to iterative, adaptive environments. Without a clear conceptual foundation, that variation can be misread as requiring different things from T&E, to necessitate different forms of T&E. Whereas sticking with the tenets of the discipline (the scientific method, a baselined expectation, and controlled evaluation) lets a practitioner adapt cadence, fidelity, and timing to the context while the discipline itself stays constant. While projects can change shape, the underlying discipline does not. Yet, after investigating the US DoD in 2025, the US GAO [24] still finds that the underlying tenets of T&E were “largely not reflected in the policies for programs in the major capability acquisition and middle tier of acquisition pathways.”

Much of the difficulty in communicating the value of T&E follows from misaligned expectations. Described as testing, it reads as fault-finding or compliance. Framed as an assurance activity grounded in evidence, it becomes possible to explain to engineers, project managers, and other stakeholders why a particular activity is necessary, why a tool was chosen over an alternative, and why a given question warrants evaluation at all. That clarity grounds the conversation in purpose rather than precedent, and consistency of purpose builds trust in what T&E produces.

Separating T&E from its tools has implications for professional autonomy, and the responsibility that comes with that. Where T&E is equated with test execution, a practitioner can be constrained to apply one tool regardless of its suitability to the question at hand. Though when framed as a scaled process, T&E allows the tools to be selected and combined in proportion to the assurance required. That autonomy carries an obligation: a practitioner must be able to justify a choice of tool against the uncertainty being addressed and the resources it consumes. This reinforces T&E as a discipline grounded in judgement, not procedure.

There is a corresponding implication for scope. When T&E is understood as a discretionary set of activities, particularly associated with physical testing, that scope is vulnerable to expansion and contraction based on perception: additional activity added without clear purpose, or essential evaluation removed under pressure on cost or schedule. This mirrors the drift Dekker [25] describes in organisations that erode margins incrementally under pressure, one locally reasonable decision at a time. Instead, anchoring scope to the assurance question being answered gives a practitioner a clear basis for both inclusion and exclusion, shifting the conversation with stakeholders from the mechanics of running tests to the consequence of not having the evidence a decision requires.

Taken together, these distinctions reduce a great deal of the friction practitioners encounter day to day. Debates about whether testing is necessary, confusion over terminology, and misalignment with a project’s delivery framework can usually be traced to the absence of a shared conceptual foundation, rather than to any disagreement about the evidence itself. Building outward from the discipline established in Defining T&E as a Discipline [3], rather than from local habit or convention, empowers practitioners with the outcome that project stakeholders seek: assurance.

A discipline becomes useful when it is applied and this article uses the definition of T&E established previously in this Journal [3] to resolve three observed sources of confusion in the application of T&E:

1. the overloaded language in the word:
2. the proliferation of project-specific labels appended to T&E’s two forms.
3. the tendency to conflate T&E with the delivery frameworks that surround it.

None of these confusions reflect a deficiency in T&E principles. Instead, they reflect an inconsistency in how the discipline has matured in different contexts. Addressing them does not require new constructs, only consistent application of the foundation already established. It does not resolve every difficulty a practitioner will encounter on a project, nor does it need to; its purpose is to provide a stable conceptual anchor from which project-specific judgement can be exercised consistently.

This article is deliberately bounded to the practitioner level, to be read with Defining T&E as a Discipline. [3] The organisational implications of a coherent T&E discipline remain for future work, particularly the role of T&E governance, risk management in complex systems, and the optimisation of complex socio-technical systems. A complementary avenue for future work would be direct empirical validation of the confusions addressed here. While this article grounds its arguments in institutional reporting, practitioner texts and organisational documents, a structured practitioner validation activity, whether survey, structured interview or workshop, would test these confusions against the lived experience of a defined T&E population directly, rather than through the secondary evidence assembled in this treatment. Such primary research sits outside the scope and resourcing available to this article, but would usefully sharpen the boundaries of the concepts to establish whether the prevalence and consequence vary by technology domain or organisation.

Projects don’t need T&E for its own sake. Yet applied in line with the discipline, from the language practitioners use through to the labels a project attaches to it, the T&E discipline gives projects what they actually wanted from the start: assurance.

This article originated as material presented at the Australian Systems Engineering and Test & Evaluation 2026 (SETE 2026) conference. The author thanks conference attendees whose feedback and interest encouraged its preparation for The ITEA Journal of Test and Evaluation.

Large language models were used to assist with editorial tasks, including grammar and language refinement. Generative tools were used to render figures based on the author’s original sketches. All concepts, content, analysis, and interpretations remain with the author.

This research received no external funding.

The author declares no conflict of interest.

All data supporting the findings of this study are contained within the article; the reference list provides comprehensive coverage of relevant sources.

1. INCOSE, Systems Engineering Handbook. 5 ed, ed. D.D. Walden. 2023, San Diego CA: John Wiley & Sons, Incorporated.
2. StoneWorld, Dimension Stone Design Manual.Stone World, 2016. 25(8).
3. Luther, B., Defining T&E as a Discipline.ITEA Journal of Test and Evaluation, 2026. 47(2).
4. Snowden, D.J. and M.E. Boone, A Leader’s Framework for Decision Making.Harvard Business Review, 2007. 85(11): p. 68 – 75.
5. Sterman, J., Business dynamics: systems thinking and modelling for a complex world. 2000, Boston, USA: McGraw-Hill Higher Education.
6. Luther, B.S., I. Gunawan, and N. Nguyen, Identifying effective risk management frameworks for complex socio-technical systems.Safety Science, 2023. 158.
7. Luther, B., I. Gunawan, and N. Nguyen, Evolving Risk Management Frameworks for Complex Systems—An Empirically Grounded Systems Thinking Approach.Systems research and behavioral science, 2024.
8. Rasmussen, J., Risk management in a dynamic society: a modelling problem.Safety science, 1997. 27(2): p. 183-213.
9. Leveson, N., A new accident model for engineering safer systems.Safety science, 2004. 42(4): p. 237-270.
10. Leveson, N., et al., Moving Beyond Normal Accidents and High Reliability Organizations: A Systems Approach to Safety in Complex Systems.Organization studies, 2009. 30(2-3): p. 227-249.
11. Hutchison, S., Test and evaluation: myths and misconceptions.Defense AT & L, 2015: p. 6.
12. Crispin, L. and J. Gregory, Agile testing : a practical guide for testers and agile teams. 1 ed. The Addison-Wesley signature series. 2009, Upper Saddle River, NJ: Addison-Wesley.
13. Office, U.S.G.A., Best practices : Using a knowledge-based approach to improve weapon acquisition. 2004, U.S. General Accounting Office: Washington DC.
14. INCOSE, Systems Engineering Measurement Primer. 1998, International Council on Systems Engineering.
15. Aven, T., Risk assessment and risk management: Review of recent advances on their foundation.European Journal of Operational Research, 2016. 253(1): p. 1-13.
16. Christensen, C. and S. Cook. What Customers Want from Your Products. 2006 16 January 2006 11 August 2026]; Available from: https://www.library.hbs.edu/working-knowledge/what-customers-want-from-your-products.
17. Kendall, F., Risk and Risk Mitigation—Don’t Be a Spectator. Defense Aquisition Technology and Logistics, 2015. 44(1): p. 2-5.
18. Hirshorn, S.R., ed. NASA Systems Engineering Handbook. 2 ed. 2016, National Aeronautics and Space Administration: Washington, DC.
19. University, D.A., Test and Evaluation Management Guide. 2005, Defense Aquisition University.
20. DOT&E, Test and Evaluation Enterprise Guidebook. 2022, US Department of Defense.
21. Beck, K., et al., Manifesto for Agile Software Development.2001.
22. Office, G.A., DOD Faces Risks and Challenges in Implementing Modern Approaches and Addressing Cybersecurity Practices. 2021, Government Accountability Office: Washington DC.
23. Office, G.A., Defense Software Acquisition – Changes to Requirements, Oversight, and Tools Needed for Weapon Programs. 2023, Government Accountability Office: Washington DC.
24. Office, G.A., Weapon Systems Testing: DOD Needs to Update Policies to Better Support Modernization Efforts. 2025, Government Accountability Office: Washington, DC.
25. Dekker, S., Drift into Failure: From Hunting Broken Components to Understanding Complex Systems. 1st edition. ed. 2011, London: CRC Press.