JUNE 2026 I Volume 47, Issue 2
JUNE 2026
Volume 47 I Issue 2
IN THIS JOURNAL:
- Issue at a Glance
- Chairman’s Message
Technical Articles - 2026 AI in T&E Forum
- Decision Assurance for AI-Enabled Mission Systems: From Test Evidence to Operational Authority
- Accelerating Test & Evaluation with AI Across the Systems Engineering Lifecycle
- SECC: An AI-Powered Assurance Agent for Complex Government System Integration
- Human Oversight for AI-Generated Test Artifacts
- Toward an Integrated T&E Framework for AI-enabled Systems: A Conceptual Model
Technical Articles
- Retrieval-Augmented Generation for Departmental Test & Evaluation
- Avoiding Vendor Lock-In in AI Procurement
- Developing Winning Proposals through the Lens of Test and Evaluation
- Defining T&E as a Discipline
News
- Association News
- Chapter News
- Corporate Member News
![]()
Human Oversight for AI-Generated Test Artifacts

Andrew Pollner
President, American Software Testing Qualifications Board (ASTQB)
Chevy Chase, MD
![]()
![]()
Abstract
Generative artificial intelligence can draft test cases, expand suites quickly, and refresh artifacts as requirements change, but acceleration is not the same as evidence. A recent global developer survey found that artificial intelligence-assisted coding is now routine, test generation is widely adopted, yet trust in artificial intelligence-generated output remains low and verification lags. For the test and evaluation community, the central issue is no longer whether artificial intelligence can produce test artifacts; it is whether those artifacts provide trustworthy evidence for release and acceptance decisions. This paper synthesizes current survey data, recent research on artificial intelligence-generated unit tests, and risk-management guidance to identify common failure modes—happy-path bias, missing boundary and state coverage, nonfunctional omissions, and false confidence from volume—and proposes a practical oversight model for independent verification and validation of artificial intelligence-generated test artifacts.
Keywords: artificial intelligence; software testing; independent verification and validation; risk-based testing; test artifact review
Introduction
Generative artificial intelligence is now part of ordinary software engineering practice. In a 2026 global survey of 1,149 developers, 72 percent of developers who had tried artificial intelligence coding tools said they use them every day, and respondents estimated that 42 percent of the code they commit is generated or significantly assisted by artificial intelligence (SonarSource 2026, 4–5). The same report found that generating tests is one of the most common artificial intelligence-assisted development tasks: 75 percent reported that their team or company uses artificial intelligence for test generation, although only 59 percent rated it as extremely or very effective for that task (SonarSource 2026, 7). In other words, adoption is high, but confidence in the quality of the resulting test artifacts is notably lower.
That gap matters to the test and evaluation community because a test artifact is not valuable merely because it exists or executes. It is valuable because it supports a claim: that a requirement was exercised, that a risk was challenged, that a defect would likely be exposed before release, or that a mission-relevant behavior was validated under defined conditions. If artificial intelligence increases the volume of tests without increasing the evidentiary strength of those tests, then the result is not stronger assurance. It is a more convincing appearance of assurance.
In this paper, assurance refers to justified confidence that test artifacts provide trustworthy, reviewable, and risk-relevant evidence suitable for release, operational, acceptance, or mission-related decisions. The term is used in this sense throughout the paper to distinguish evidentiary strength from simple artifact volume or execution success.
Recent guidance from the National Institute of Standards and Technology reinforces why this distinction is important. NIST’s Artificial Intelligence Risk Management Framework explicitly elevates test, evaluation, verification, and validation across the artificial intelligence life cycle and notes that actors with evaluation-specific expertise should be integrated throughout that life cycle (National Institute of Standards and Technology 2023, 9–11). NIST also identifies separation between those building models and those verifying and validating them as a best practice (National Institute of Standards and Technology 2023, 10). For independent verification and validation in software-intensive systems, that principle is directly relevant. Artificial intelligence may help draft a test case, but it cannot remove the need for independent judgment about whether that test case is adequate, complete, traceable, and fit for use as evidence.
Methodological Approach
This paper uses a structured synthesis approach combining practitioner guidance, industry survey data, academic research, and risk-management frameworks related to artificial intelligence-assisted software testing and verification. Sources were selected based on four criteria: relevance to test artifact generation, applicability to independent verification and validation, empirical or operational insight into artificial intelligence-generated testing behavior, and alignment with established software and systems assurance practices.
The analysis emphasizes recurring failure patterns and oversight concerns that appear consistently across multiple categories of sources, including developer surveys, academic evaluations of large language model-generated tests, National Institute of Standards and Technology guidance, and practitioner-oriented testing governance recommendations. Rather than evaluating a specific tool implementation, the paper synthesizes current evidence to identify common oversight requirements and proposes a structured framework for evaluating whether artificial intelligence-generated test artifacts are sufficiently trustworthy to support release, acceptance, or mission-relevant decisions.
The intent is not to discourage the use of generative artificial intelligence in testing workflows. Instead, the paper focuses on how test and evaluation organizations can integrate artificial intelligence-generated artifacts into existing verification and validation processes while preserving evidentiary rigor, traceability, and independent review.
The New Verification Gap
The present challenge is not that artificial intelligence fails to save time. On the contrary, practitioners consistently report meaningful acceleration. Industry guidance aimed at testing organizations describes shorter drafting cycles, easier scaling across large backlogs, and more consistent formatting of test assets when generative tools are used to draft artifacts (Pollner n.d.; ASTQB n.d.). Although this discussion focuses primarily on software testing and software-intensive systems, many of the same oversight concerns increasingly apply to enterprise workflows, integrated platforms, and broader operational test environments that incorporate artificial intelligence-assisted artifact generation. The problem is that velocity is easier to measure than correctness, relevance, or coverage adequacy.
The same survey makes this point clearly. Although developers reported productivity gains and widespread daily use, 96 percent said they do not fully trust that artificial intelligence-generated code is functionally correct, and only 48 percent said they always check artificial intelligence-assisted code before committing it (SonarSource 2026, 9–10). In the same survey, 61 percent agreed that artificial intelligence often produces code that looks correct but is not reliable, and the same percentage agreed that substantial effort is needed in prompting, fixing, and refinement to obtain good output (SonarSource 2026, 11). This is a critical finding for test and evaluation professionals because test artifacts generated from questionable code, incomplete requirements, or shallow prompts can inherit those weaknesses while simultaneously masking them behind high test counts and polished structure.
The survey also shows that this problem is not limited to code generation. Developers already describe review and validation as the most important skill in the artificial intelligence-assisted environment, outranking prompting itself (SonarSource 2026, 13). Static analysis tools are increasingly being applied to review artificial intelligence-generated code, suggesting that practitioners are already turning toward more deterministic controls to compensate for artificial intelligence’s variability and the review burden it creates (SonarSource 2026, 27–28). The broader implication is that the bottleneck has moved downstream. Generating artifacts is faster; determining whether those artifacts deserve to influence release or acceptance decisions is harder.
Similar concerns appear in broader discussions of generative artificial intelligence reliability and oversight. Research on large language model behavior has identified tendencies toward plausible but incorrect outputs, inconsistent reasoning, and context-sensitive variability, all of which complicate the evaluation of generated software and test artifacts. These concerns reinforce the need for independent review processes that focus on evidentiary quality rather than apparent fluency or artifact volume.
For testing organizations, this shift creates a subtle danger. If schedule pressure remains unchanged while artifact production accelerates, review depth may erode precisely when it is most needed. A suite of 300 automatically generated tests can appear more defensible than a human-authored suite of 40 tests, even if the smaller suite is better aligned with operational risk, interfaces, edge conditions, and failure modes. Volume can therefore distort judgment by substituting activity for evidence.
Common Failure Modes in Artificial Intelligence-Generated Test Artifacts
The most important oversight task is understanding where artificial intelligence-generated test artifacts tend to fail. Several failure patterns appear consistently in practitioner guidance and emerging research.
Table 1. Common Failure Modes in AI-Generated Test Artifacts
| Failure Mode | Typical Weakness | Risk to T&E Evidence | Reviewer Focus |
| Happy-path concentration | Overemphasis on nominal flows | Operational failures remain untested | Negative and degraded scenarios |
| Boundary and partition gaps | Missing edge values and invalid partitions | Hidden defect conditions | Boundary value and equivalence analysis |
| Incomplete state-transition coverage | Missing retries, rollback, recovery paths | Stateful defects escape detection | Workflow and transition validation |
| False confidence from coverage and count metrics | High coverage or test volume despite weak assertions or shallow validation logic | Misleading assurance and undetected defects | Oracle quality, assertion validity, evidentiary strength |
| Nonfunctional omissions | Missing security, performance, resilience, or interoperability checks | Incomplete operational assurance | Risk-based nonfunctional testing |
These recurring failure modes suggest that artificial intelligence-generated test artifacts require systematic oversight before they can be treated as trustworthy evaluation evidence.
Happy-path concentration
Artificial intelligence-generated tests often overrepresent nominal behavior. Pollner’s poster on human oversight of artificial intelligence-generated test artifacts describes a “happy path bias” in which ideal user journeys receive disproportionate attention while negative conditions, exceptional handling, and complex state behavior are under-specified (Pollner n.d.). This companion paper makes the same point more plainly: artificial intelligence tends to produce clean, expected flows while underrepresenting the situations in which software most often fails (ASTQB n.d.).
Generative models may also exhibit sycophantic tendencies by favoring expected or implied behavior patterns over contradictory, adversarial, or negated conditions that are often central to rigorous testing. This can reinforce an overemphasis on nominal behavior while underrepresenting conditions designed specifically to challenge assumptions, expose faults, or stress operational boundaries.
This tendency is unsurprising. Large language models are optimized to predict plausible continuations from common patterns, and nominal flows are the patterns most frequently expressed in requirements, examples, training material, and public code. Yet mission-relevant assurance usually depends on the opposite. Failures often appear at the seams: invalid data, partial completion, asynchronous timing, resource exhaustion, degraded dependencies, unexpected user behavior, and combinations of conditions that do not occur in textbook examples.
Boundary and partition gaps
The poster also highlights boundary value analysis and equivalence partitioning as techniques that artificial intelligence frequently fails to apply without explicit guidance (Pollner n.d.). This observation aligns with current empirical work. Wang et al. found that large language models struggle to infer test inputs needed to cover all conditions in complex methods, resulting in missing lines and branches unless the problem is decomposed to make coverage goals more explicit (Wang et al. 2024). A model can generate many syntactically valid tests while still missing the values and classes most likely to expose defects.
For independent verification and validation, this has two implications. First, artificial intelligence-generated tests require review not only for execution success but also for input selection logic. Second, human reviewers must ask whether the suite reflects how the system could fail, not only how it is expected to work. This is where trained testers add disproportionate value. Boundary values, invalid partitions, and domain-specific partitions are rarely obvious from natural-language prompts alone. They emerge from analysis.
Similar limitations appear in more rigorous forms of test design, including combinatorial testing, covering arrays, and statistically informed input selection strategies. Generative models may produce plausible collections of test cases while failing to systematically represent interaction effects or higher-order condition combinations unless those objectives are explicitly engineered into the prompting and review process.
Incomplete state-transition coverage
Stateful systems pose a related problem. Transactions, sessions, workflows, safety interlocks, and recovery logic all depend on transitions between states rather than on isolated input-output pairs. Artificial intelligence-generated test suites often cover obvious transitions but omit less visible ones, such as retries after timeouts, partial rollback, interrupted authorization flows, or restoration from degraded modes (Pollner n.d.; ASTQB n.d.).
Research on project-level unit test generation shows why this matters. In ProjectTest, Wang et al. found that even frontier models exhibit significant basic yet critical errors, including compilation and cascade errors, when asked to generate project-level test suites (Wang et al. 2025). Those problems are noteworthy not just because some generated tests fail to run, but because complex, multi-file, stateful systems are exactly where test adequacy becomes hardest to judge by superficial inspection. A suite that compiles and covers straightforward functions may still leave temporal and state-dependent defects untouched.
False confidence from coverage and count metrics
Coverage is useful, but it is not self-interpreting. Bhatia et al. found that generated unit tests achieved coverage comparable to an established automated generator for some classes of programs, yet about one-third of the assertions generated by a large language model were incorrect in some categories (Bhatia et al. 2023). This is a cautionary result. Coverage can suggest progress even when expected results are wrong, oracles are weak, or assertions merely confirm the same flawed assumption used to generate the code.
Variability between generated outputs introduces an additional challenge. Two executions of the same prompt may produce different test suites, assertions, or coverage patterns, complicating repeatability and evidentiary consistency. In higher-assurance environments, this variability increases the importance of provenance tracking, version control, and human review of accepted artifacts.
For a test and evaluation audience, this distinction is central. A test case is not sound evidence simply because it executes a line, branch, or path. Evidence requires a defensible oracle, meaningful pass-fail logic, and traceability to the claim being evaluated. If the oracle is wrong, then a passing test may strengthen false confidence rather than reduce uncertainty.
Nonfunctional omissions
Functional checks are not the only concern. Practitioner guidance on artificial intelligence-generated tests consistently warns that security, performance, accessibility, and other nonfunctional risks are often absent unless a skilled tester explicitly requests them and reviews the result (ASTQB n.d.; Pollner n.d.). The broader developer data point in the same direction. In SonarSource’s survey, developers identified exposure of sensitive company or customer data as their top concern related to artificial intelligence-generated code, while concerns about subtle and severe security vulnerabilities were also widespread (SonarSource 2026, 29–32).
NIST’s secure software development guidance for generative artificial intelligence reinforces the need for broader testing policy. The guidance recommends that organizations include artificial intelligence models and related components within code review and code testing policies and notes that testing may include unit, integration, penetration, use case, red-team, and adversarial testing, depending on risk (Booth et al. 2024, 17). In other words, the relevant question is not whether a generated test suite exists, but whether it addresses the quality characteristics and threat conditions that matter for the system’s intended use.
Implications for Independent Verification and Validation
These failure modes change how independent verification and validation should treat artificial intelligence-generated test artifacts. Traditionally, the primary concern has been whether the system under test satisfies requirements and whether the test process was sufficiently rigorous and independent. With generative artificial intelligence in the toolchain, the test artifacts themselves become objects of verification. They can no longer be assumed to be neutral instruments.
This does not imply that every artificial intelligence-generated test must be discarded or rewritten manually. It means that such artifacts should be treated as drafts until they have passed a defined acceptance process. The structured oversight framework introduced later in this paper operationalizes that acceptance process through staged review activities tied to evidentiary quality and system risk. At minimum, that acceptance process should establish five things: traceability to a requirement, interface, hazard, or risk statement; a clear and reviewable oracle; evidence that negative and edge conditions were considered; documentation of assumptions and constraints embedded in the prompt or generated artifact; and a record of human review proportional to mission or business criticality.
Independence becomes especially important in high-consequence environments. NIST’s artificial intelligence risk guidance emphasizes pre-deployment testing, content provenance, governance, and risk management for generative artificial intelligence deployments (National Institute of Standards and Technology 2024, 1–2). In the test and evaluation context, provenance includes more than knowing which tool produced a test. It includes knowing which prompt or source requirement informed it, what context window was provided, what follow-up edits were made, and who approved it for inclusion in an official suite. Without that chain, later failures are difficult to analyze and lessons are difficult to institutionalize.
The same logic explains why a NIST pilot code challenge now focuses specifically on measuring and evaluating artificial intelligence-generated unit tests (National Institute of Standards and Technology n.d.). The quality of generated tests is itself a measurable problem. That recognition is important because it moves the conversation beyond anecdote. Artificial intelligence-generated tests are no longer merely a convenience feature inside development tools. They are becoming formal artifacts whose adequacy can and should be evaluated.
In mission-level T&E environments, AI-generated test artifacts may influence operational readiness assessments, digital twin evaluations, hardware/software integration testing, and autonomous system validation. For independent verification and validation teams, this leads to a practical reframing. Artificial intelligence should be used to accelerate candidate artifact production, not to make final acceptance judgments. Acceptance decisions should remain tied to independently reviewable criteria. In low-risk contexts, that may mean team-level peer review and automated checks. In higher-risk contexts, it should mean independent review, explicit sign-off, and stronger evidence of traceability and risk coverage. The more safety-critical, security-sensitive, or mission-dependent the system, the less appropriate it is to treat artificial intelligence-generated tests as self-authenticating.
A Structured Oversight Framework for AI-Generated Test Artifacts
The recurring failure modes discussed earlier suggest that artificial intelligence-generated test artifacts should not be treated as self-validating evidence. Instead, they should pass through a structured oversight process that evaluates whether the generated artifact is trustworthy, reviewable, risk-aligned, and appropriate for inclusion in an official evidence base.
This paper proposes a six-stage oversight framework intended for software testing, enterprise verification, and independent verification and validation environments that increasingly incorporate generative artificial intelligence into test design workflows.
The framework does not prohibit artificial intelligence-generated testing. Rather, it formalizes the distinction between:
- candidate artifacts generated by artificial intelligence
- accepted evidence approved through human oversight
The framework is designed to scale according to system criticality, organizational risk tolerance, and operational consequences.
Stage 1 — Artifact Origin and Provenance Review
The first oversight task is establishing provenance. Reviewers should understand:
- which tool or model generated the artifact
- what prompts or source requirements informed generation
- what contextual information was provided
- whether post-generation modifications occurred
- who approved the artifact for continued review
This stage establishes traceability and supports later root-cause analysis if the artifact proves inadequate.
Stage 2 — Traceability and Risk Alignment Review
Generated tests should be mapped to:
- requirements
- interfaces
- hazards
- operational risks
- business or mission-critical behaviors
The purpose is to verify that the artifact supports a meaningful evaluation claim rather than merely exercising code paths.
Oversight review should evaluate generated artifacts against an explicit test strategy defining risk priorities, coverage objectives, operational constraints, and evidentiary standards. The strategy provides the rationale for what should be tested, what may be deprioritized, and what degree of rigor is appropriate for the system context.
In enterprise environments, this may include transaction integrity, workflow continuity, or data-handling requirements. In higher-consequence environments, it may include safety constraints, degraded operational conditions, or mission-critical workflows.
Stage 3 — Coverage Adequacy Review
Reviewers should assess whether the generated suite adequately exercises:
- boundary conditions
- invalid partitions
- negative scenarios
- degraded dependencies
- state transitions
- concurrency or timing behavior where applicable
Particular attention should be given to conditions that generative models frequently underrepresent, including partial failures, recovery logic, asynchronous processing, and unexpected user or system behavior.
Coverage review should focus on evidentiary adequacy rather than raw test volume or execution count.
Stage 4 — Oracle and Assertion Validation
A test artifact provides meaningful evidence only if its pass-fail logic is trustworthy.
Reviewers should verify:
- correctness of assertions
- validity of expected outcomes
- appropriateness of tolerances and thresholds
- meaningful failure detection capability
- determinism and repeatability where required
This stage is especially important because large language models can generate assertions that appear plausible while encoding incorrect assumptions.
Stage 5 — Nonfunctional and Enterprise Risk Review
Generated artifacts should be evaluated for coverage of nonfunctional concerns relevant to the system context, including:
- security
- performance
- resilience
- accessibility
- interoperability
- scalability
- auditability
Enterprise and operational environments increasingly depend on complex workflows and distributed dependencies that may not be represented in nominal functional tests.
Reviewers should verify that omitted nonfunctional areas are intentionally excluded rather than unintentionally ignored.
Stage 6 — Human Acceptance and Evidence Authorization
The final stage is a human acceptance decision proportional to system criticality.
Low-risk internal tooling may require only peer review and automated checks. Enterprise-critical or externally audited systems may require formal approval workflows. Safety-critical or mission-relevant systems may require independent verification and validation review prior to acceptance into the official evidence base.
At this stage, the artifact transitions from:
- artificial intelligence-generated candidate output
to:
- human-approved evaluation evidence
The framework therefore preserves the acceleration benefits of artificial intelligence while maintaining independent accountability for the evidentiary quality of test artifacts.
Table 2. Oversight Framework for AI-Generated Test Artifacts
| Oversight Stage | Primary Question | Typical Reviewer Concern |
| Artifact Origin Review | Where did the artifact come from? | Prompt provenance, model context, modification history |
| Traceability Review | What claim does the test support? | Requirement, interface, or risk alignment |
| Coverage Adequacy Review | What conditions are exercised? | Boundary gaps, negative paths, state behavior |
| Oracle Validation | Would this test detect meaningful failure? | Incorrect assertions or weak pass/fail logic |
| Nonfunctional Review | Are broader operational risks addressed? | Security, performance, resilience omissions |
| Human Acceptance Review | Is this artifact evidence-worthy? | Approval authority proportional to criticality |
Example: Oversight Review of an Artificial Intelligence-Generated Test Suite
Consider an enterprise authentication platform supporting customer account access for a financial services application. A generative artificial intelligence tool is used to create an initial suite of login and password-management tests based on requirements documentation and interface specifications.
The generated suite includes:
- successful login validation
- invalid password rejection
- password reset workflow testing
- account lockout after repeated failures
- multi-factor authentication verification
At first inspection, the suite appears comprehensive because it contains more than one hundred generated tests with high execution success rates and broad line coverage.
Applying the proposed oversight framework reveals several important deficiencies.
Traceability and Risk Alignment Findings
Reviewers determine that the generated suite traces successfully to most functional authentication requirements but lacks explicit traceability to:
- fraud-detection workflows
- session timeout handling
- audit logging requirements
- recovery from partially completed authentication flows
The suite therefore demonstrates functional coverage but incomplete operational-risk coverage.
Coverage Adequacy Findings
Boundary-value and partition analysis identifies several missing conditions:
- maximum-length credential handling
- malformed token processing
- simultaneous login attempts
- degraded network conditions
- interrupted multi-factor authentication workflows
State-transition analysis further reveals that retry behavior following timeout or partial authentication failure is underrepresented.
Oracle Review Findings
Several generated assertions merely confirm expected response codes without validating:
- backend authorization state
- session invalidation
- audit-event creation
- privilege assignment integrity
Although the tests execute successfully, reviewers determine that some assertions would fail to detect meaningful security defects.
Nonfunctional Review Findings
The generated suite contains little evidence of:
- performance stress testing
- accessibility verification
- resilience under dependency degradation
- adversarial or abuse-oriented testing
These omissions are especially important because the platform supports externally accessible interfaces.
A Practical Oversight Model for Test and Evaluation Organizations
An effective response is neither to ban generative artificial intelligence nor to treat it as a drop-in replacement for test design expertise. A more durable approach is to combine artificial intelligence acceleration with structured oversight. The poster and this companion paper frame the choice as a shortcut path versus a strategic path. The shortcut path treats artificial intelligence primarily as a cost-reduction mechanism and judges success by speed and artifact volume. The strategic path preserves human expertise, uses explicit review criteria, and aligns testing effort with actual risk (Pollner n.d.; ASTQB n.d.). For test and evaluation organizations, only the strategic path is compatible with independent assurance.
The first element of that path is governance. Organizations need clear policy for what artificial intelligence may generate, where it may be used, what systems are in scope, what provenance must be recorded, and what approval level is required before an artifact is accepted. Governance should distinguish between informal exploratory use, internal engineering use, and artifacts that support external claims, acceptance, or release decisions. High-consequence artifacts deserve stronger controls.
The second element is review standards. A generated test should not be accepted because it is readable or because it executes successfully. Review criteria should address requirement traceability, risk relevance, oracle quality, setup validity, data representativeness, maintainability, and determinism. For stateful behavior, reviewers should verify that transitions, preconditions, and postconditions are adequately exercised. For interfaces and boundary-sensitive functions, reviewers should verify that valid, invalid, and limit conditions are intentionally represented. For nonfunctional concerns, reviewers should verify that the suite includes or explicitly excludes security, performance, resilience, accessibility, and interoperability checks based on system risk.
The third element is metrics. Test organizations often default to what is easiest to count: number of generated tests, line coverage, execution pass rate, and elapsed time to produce a suite. Those metrics are still useful, but they are insufficient. Better indicators include defect detection effectiveness, mutation resistance where applicable, percentage of high-risk requirements with reviewed traceability, escaped defect rates, percentage of generated artifacts requiring substantive human rework, and review time per accepted artifact. The goal is to measure whether artificial intelligence improves the quality of evidence, not merely the quantity of output.
The fourth element is skill development. The most effective reviewers of artificial intelligence-generated test artifacts will be people who understand both classical test design and the strengths and weaknesses of generative tools. That includes boundary value analysis, equivalence partitioning, state-transition testing, risk-based prioritization, and nonfunctional test design, as well as prompt construction, artifact review, and tooling limitations. Formal training and certification can support this capability, but the essential issue is demonstrable competence rather than branding. Artificial intelligence changes the mechanics of drafting; it does not remove the need for analytical testing skill. Classical test science disciplines remain essential because artificial intelligence changes the mechanics of drafting artifacts without replacing the analytical rigor required for effective test design and evaluation.
The fifth element is role design. Developers, test engineers, analysts, and managers are not interchangeable in an artificial intelligence-assisted workflow. The developer may be well positioned to use artificial intelligence for rapid scaffolding and local checks. The independent tester or analyst is better positioned to challenge assumptions, identify missing partitions, and assess operational realism. The test manager is better positioned to define review gates, escalation criteria, and evidence standards. Oversight is strongest when those roles are aligned rather than collapsed.
These elements can be implemented incrementally. A mature organization does not need to perfect every element before obtaining value from artificial intelligence. It does, however, need to resist the temptation to substitute faster artifact generation for stronger evidence. Artificial intelligence can help draft a requirement-based test matrix, propose negative cases, generate harness code, or refresh cases when interfaces change. Those are meaningful gains. But the artifact should become part of the official evidence base only after it has been reviewed by a human with the authority and expertise to question it.
Conclusion
Generative artificial intelligence has changed the economics of test drafting, but it has not changed the fundamentals of assurance. A trustworthy test artifact still requires traceability, sound oracles, appropriate coverage, and independent judgment about adequacy. Current survey data shows that artificial intelligence-generated output is widely used but not fully trusted, and current research shows that even strong models still miss conditions, produce incorrect assertions, and generate basic errors that require human correction. For the test and evaluation community, the appropriate response is not skepticism for its own sake and not blind adoption. It is disciplined integration. Artificial intelligence should accelerate the production of candidate test artifacts, while independent verification and validation ensures that only reviewed, risk-aligned, and evidence-worthy artifacts influence mission or release decisions. Artificial intelligence adds speed. Human oversight is what turns that speed into trustworthy evaluation.
References
ASTQB. n.d. “Why AI-Generated Tests Need Human Oversight.” Accessed March 25, 2026. https://astqb.org/ai-generated-tests-need-human-oversight/.
Bhatia, Shreya, Tarushi Gandhi, Dhruv Kumar, and Pankaj Jalote. 2023. “Unit Test Generation using Generative AI: A Comparative Performance Analysis of Autogeneration Tools.” arXiv. https://doi.org/10.48550/arXiv.2312.10622.
Booth, Harold, Murugiah Souppaya, Apostol Vassilev, Michael Ogata, Martin Stanley, and Karen Scarfone. 2024. Secure Software Development Practices for Generative AI and Dual-Use Foundation Models: An SSDF Community Profile. NIST SP 800-218A. Gaithersburg, MD: National Institute of Standards and Technology. https://doi.org/10.6028/NIST.SP.800-218A.
National Institute of Standards and Technology. 2023. Artificial Intelligence Risk Management Framework (AI RMF 1.0). NIST AI 100-1. Gaithersburg, MD: National Institute of Standards and Technology. https://doi.org/10.6028/NIST.AI.100-1.
National Institute of Standards and Technology. 2024. Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile. NIST AI 600-1. Gaithersburg, MD: National Institute of Standards and Technology. https://doi.org/10.6028/NIST.AI.600-1.
National Institute of Standards and Technology. n.d. “GenAI: Code Challenge (Pilot).” Accessed March 25, 2026. https://ai-challenges.nist.gov/code.
Pollner, Andrew. n.d. “Human Oversight in AI-Generated Test Artifacts: Implications for Independent Verification and Validation.” Poster presentation at 2026 ITEA AI in T&E Forum.
SonarSource. 2026. State of Code Developer Survey Report. https://www.sonarsource.com/state-of-code-developer-survey-report.pdf.
Wang, Yibo, Congying Xia, Wenting Zhao, Jiangshu Du, Chunyu Miao, Zhongfen Deng, Philip S. Yu, and Chen Xing. 2025. “ProjectTest: A Project-level LLM Unit Test Generation Benchmark and Impact of Error Fixing Mechanisms.” arXiv. https://arxiv.org/abs/2502.06556.
Wang, Zejun, Kaibo Liu, Ge Li, and Zhi Jin. 2024. “HITS: High-coverage LLM-based Unit Test Generation via Method Slicing.” arXiv. https://doi.org/10.48550/arXiv.2408.11324.
Author Biographies
Mr. Andrew Pollner Since 2004 he volunteers as a director of the American Software Testing Qualifications Board (ASTQB) where he currently serves as president, standardizing testing certifications in the US based on the global International Software Testing Qualifications Board (ISTQB). Mr. Pollner is vice-chair over a broad portfolio of ISTQB specialist certifications and is the Product Owner for Performance Testing and Test Automation syllabi. Additionally, Mr. Pollner is President and CEO of ALP International Corporation, a leading organization in the area of test process improvement and automated testing, bringing effectiveness and efficiency to testing, since 1993.
Dewey Classification: L 681 12

