Breaking Agents: a Testing Framework for Red Teaming LLM-enabled Systems

Enterprise adoption of large language models (LLM) has moved beyond conversational assistants into agentic systems that retrieve internal knowledge, call tools, and execute multi-step plans that can change enterprise state such as sending messages, updating tickets, modifying documents, initiating workflows, or triggering infrastructure actions. In this regime, the central security problem is no longer whether the model can be coaxed into producing unsafe text but whether the entire agentic stack preserves authorization, confidentiality, integrity, and auditability when adversaries attempt to manipulate its context, its tool choices, or its decision loop. The shift matters because an agent’s mistakes can become durable state changes, and durable state changes can propagate into other systems that interpret them as legitimate operations rather than as model errors (NIST 2024).

The motivation for AI agents is straightforward: language interfaces compress the friction of complex software by translating user intent into actions across heterogeneous systems. Yet the same translation that creates productivity also creates a new class of vulnerabilities in which untrusted language is interpreted as privileged instruction. Traditional software security has lived for decades with the principle that data must never be executed as code, and that untrusted inputs must be isolated, validated, and authorized before they can influence control flow. Agentic systems challenge this principle because they intentionally blend natural language instructions, retrieved artifacts, and tool outputs into a single context that the model uses to decide what to do next. When the model cannot reliably distinguish which tokens are instruction and which tokens are merely data, the system inherits a structural ambiguity that adversaries can exploit (OWASP 2024). Another reason why this problem is hard is because the behavior of an LLM-centered agent is partly stochastic, emergent, and sensitive to minor changes in phrasing, retrieval context, and tool results. This makes evaluation by one-prompt-one-run fundamentally inadequate. A system can appear safe in a deterministic demo yet fail under adaptive multi-turn pressure, or when the retrieval pipeline returns an unexpected document that injects new goals (Manès 2021). Security engineering therefore needs a methodology that respects what is unique about LLM systems (context dependence, probabilistic behavior, and tool-mediated action) while borrowing what is durable from established security practice (threat modeling, failure oracles, coverage, reproducibility, and regression).

Here we argue that agent red teaming should be treated as a formal testing method rather than as a creative writing exercise. The point of red teaming is not to collect entertaining jailbreaks but to discover and document boundary violations that are observable, repeatable, and actionable. In a tool-integrated agent, the relevant boundary is not the model’s willingness to say no but the system’s ability to prevent an unauthorized read, an unsafe write, a cross-tenant or cross-user data flow or a policy-inconsistent state change even when the model is manipulated (ISO/IEC 2023). A test-and-evaluation (T&E) program therefore begins by translating policy into invariants, mapping the agent’s operational authority, and then designing campaigns that systematically explore how that authority can be induced to cross boundaries through direct prompting, indirect influence via retrieved content, or manipulation of tool responses; see Figure 3 below.

¹ An invariant is a security property that must hold across every execution trace of the agent, regardless of prompt wording, retrieved content, tool outputs, or intermediate model reasoning. Invariants function as failure oracles. A red-team test succeeds when it induces a violation of an invariant, whether that violation appears in the agent’s narrative output, an attempted or executed tool call, or a downstream state change.

Figure 1: The three-layer failure model for AI agents – observable failures in narrative output, tool invocation, and downstream state change

The methodological core of this work is a three-layer failure model that distinguishes narrative output, tool invocation, and downstream state change; see Figure 1 above. This separation is not an academic nicety. It is the difference between measuring compliance theater and measuring safety. A system can produce a policy-compliant response while still attempting a dangerous tool call, and an attempted tool call can be blocked at middleware even though the model would have proceeded if allowed. Conversely, a system can appear safe at the text layer while a downstream automation consumes its output and executes it as code, producing harm outside the LLM boundary (Yao 2024). Therefore, we emphasize instrumentation, evidence standards, and metrics that privilege execution traces over textual impressions.

A subtle challenge is that many agents are embedded into environments, where identity and authorization are complex such as role-based access control, project-based permissions, or attribute-based policies. In such environments, even small errors in how an agent propagates identity (such as using a service account rather than a user account, or caching an authorization decision) can create cross-user leakage that is not visible in a single interaction. This helps explain why leadership fears that agents will become data exfiltration machines, yet also why simplistic mitigations such as forbidding certain phrases are ineffective (ISO/IEC 2023). The correct question is whether the system enforces authorization at every boundary transition and whether its logs can prove that it did so.

To keep the discussion grounded, we draw on several streams of research and practice. Adversarial machine learning (AML) shows how systems can fail under crafted inputs and how evaluation must be threat-model driven (Biggio 2018). Software security testing demonstrates that systematic exploration requires explicit failure oracles and coverage, and that fuzzing succeeds because it pairs input generation with deterministic triage and minimization (Manès 2021). The rapidly growing LLM security literature provides evidence of prompt injection, tool manipulation, and privacy leakage risks that arise from deployment patterns rather than from model internals alone (Yao 2024). Finally, privacy research such as membership inference and differential privacy clarifies why “the model didn’t intend to leak” is not a defense when information can be inferred from outputs or from system behavior (Hu 2022).

The contribution of this work is not a new attack recipe but a cohesive testing methodology suitable for AI agents. It proposes a way to profile authority, define measurable failure, design safe but realistic campaigns, and convert findings into controls and governance gates that continuously reduce risk. We also propose a measurement strategy that moves beyond single success rates to boundary-aware metrics, coverage, and severity indices aligned to impact. It should be noted that this paper intentionally avoids operational exploit payloads and focuses instead on what constitutes a rigorous, evidence-driven evaluation. This choice follows an ethics stance: the goal is to help builders test and harden systems, not to accelerate misuse. In practice, organizations can adopt the proposed approach while keeping sensitive details such as proprietary tool schemas and internal corpora within secure evaluation environments. The remainder of the paper develops as follows. We first frame agent risk as a systems problem, then propose a red teaming method. We show how measurement and evidence make results actionable, and finally connect findings to controls and governance that can be integrated into the enterprise lifecycle.

Figure 2:   AI agent risk in a systems-level architecture – trust boundaries and enforcement points across input, instruction, retrieval, tools, and state

The security of an agent cannot be understood by inspecting the model alone because the agent is an assemblage: a base model wrapped in prompts and policies, connected to retrieval systems and connectors, equipped with tools that embody operational authority, and governed by orchestration logic that decides when to retrieve, when to call tools, and when to stop. In practice, the system’s behavior is shaped as much by the retrieval index, the tool schemas, and the middleware validators as by the model’s latent knowledge (Yao 2024). This is why failures emerge at the seams, and why an effective threat model must enumerate not only user inputs but all channels by which untrusted content can influence the model’s context and decisions.

A useful starting point is to decompose the agent into layers that correspond to different trust boundaries; see Figure 2 for illustration. At the top is the interaction layer, which includes user messages and conversational context. Beneath that is the instruction layer, where system prompts, developer prompts, tool descriptions, and safety policies are encoded, often as text but sometimes as structured metadata. Beneath that is the retrieval layer, where documents are selected, chunked, embedded, and inserted into context. Beneath that is the tool layer, which comprises callable functions and APIs that can read, write, or execute. Finally, there is the state layer: the external world the agent can change, including tickets, documents, databases, and operational systems, and the internal state of the agent itself such as memory and conversation history (OWASP 2024).

As described, Figure 2 provides the architectural decomposition of the agent stack in Figure 1, identifying the trust boundaries through which untrusted content, model reasoning, and operational authority interact. Figure 1 provides the corresponding evaluation lens over that architecture into three security-relevant failure surfaces, distinguishing whether a boundary failure appears in the output layer, the tool layer, or the downstream state layer. In a nutshell, Figure 2 answers “where risk lives in the system” while Figure 1 answers “how failures manifest and where they should be detected.” Specifically, Figure 2’s layers 1 through 4 map to Figure 1’s layer 1 because this is where context assembly, instruction/data confusion, and unsafe content shaping first appear. Figure 2’s layer 5 maps to Figure 1’s layer 2, where the main concern is attempted or actual unsafe tool calls and parameter misuse. Figure 2’s layer 6 maps to Figure 1’s layer 3, where harm becomes persistent through external writes, workflow triggers, or durable misconfiguration. The enforcement points in Figure 2 become the detection points in Figure 1. A key insight is that an attacker does not need to break the model’s alignment if they can instead manipulate the boundaries between these layers so that untrusted data is treated as instruction or privileged capability.

2.1 Indirect Prompt Injection & Tool Integration

Indirect prompt injection illustrates this boundary confusion in its simplest form. If an agent retrieves an email, a document, or a web page, and the retrieved artifact contains instruction-like text, the model may treat those tokens as part of the task directive rather than as content to be summarized or analyzed. The risk is amplified when retrieved artifacts are not clearly labeled as untrusted when the system prompt does not enforce strict role separation, or when the agent is encouraged to follow any instructions encountered in context. What makes the threat particularly enterprise-relevant is that indirect injection does not require direct access to the agent (Liu 2024). It only requires influence over something the agent might later retrieve, which could be a shared document, a support ticket, a vendor message, or a public page that is indexed by a search connector.

Prompt injection research has been valuable not only because it shows attacks but because it clarifies the conceptual anatomy of the vulnerability. When instruction and data share the same representational channel, the system has a semantic ambiguity that can be exploited by adversaries who craft tokens that look like instruction. That ambiguity is compounded by modern deployment patterns such as retrieval-augmented generation (RAG), where the system explicitly places externally sourced content into the model’s context (Liu 2024). In this setting, the attack surface is less a set of endpoints than a set of context assembly rules that determine how untrusted strings are framed and weighted in the model’s decision process.

Tool integration turns injection from an information hazard into an action hazard. A model that hallucinated a fact in a chat window can be corrected but a model that hallucinates a parameter in a tool call can trigger an irreversible action or create a persistent misconfiguration. The operational danger arises from three facts that engineers sometimes overlook. First, tool calls are often executed with real credentials and real permissions, so the model’s choice becomes a security decision. Second, tool calls are frequently compositional: one action can unlock another, and the agent can chain tools in a loop. Third, many AI tools have wide parameter surfaces, so the difference between benign and harmful behavior can be a minor change in an argument string (OWASP 2024). The agent therefore becomes a control plane for operations, and the security question becomes whether the system enforces authorization and validation at the control plane rather than relying on the model’s judgment.

2.2   Tool Ecosystem & Adversarial Capabilities

Recent work on vulnerabilities in tool-calling ecosystems highlights a further systems risk (Zhang 2025a). The tool catalog itself can become an adversarial influence. If an attacker can introduce or manipulate tool metadata, they can bias tool selection or cause the model to invoke tools in ways the developer did not intend. Even when the base model is well aligned, the dynamic selection of tools based on natural language descriptions creates a routing problem that resembles security-sensitive intent classification with the added complexity that the classifier is also the planner. This reinforces the need to treat tool catalogs and routing policies as security-critical code rather than as convenience features.

A systems threat model for agents is best articulated in terms of adversary capabilities rather than adversary identities. An adversary may be an external actor who can send messages to an agent-facing interface but in many organizations the more plausible adversary is someone who can influence what the agent reads. A marketing vendor who emails a crafted document, a contractor who uploads a file into a shared repository, or an attacker who poisons a web page that is later retrieved can all act as context suppliers. The system is also exposed to adversarial tooling in the form of compromised plugins, misconfigured connectors, or malicious third-party APIs whose responses are treated as ground truth. These diverse adversary channels explain why a red teaming program that focuses solely on direct user prompting can miss the failures that matter most in deployments.

The systems view also helps reconcile two apparently conflicting truths. On the one hand, many agents perform well in benign conditions and deliver real value. On the other hand, security research repeatedly shows that robustness collapses under targeted manipulation. This is not a contradiction. It is the normal pattern of complex systems whose decision logic was not designed for adversarial environments (Biggio 2018). AML emphasizes that average-case performance is not a security guarantee because attackers intentionally seek the worst-case.

2.3   Retrieval & Privacy Risks in RAG Systems

A second systems-level risk emerges from retrieval augmentation. RAG improves factual grounding by injecting external evidence but it also creates a new privacy surface: the retrieval database. The model may leak snippets of retrieved documents, reveal what documents exist, or allow an adversary to infer membership in the retrieval corpus. Empirical studies of privacy in RAG show that the retrieval mechanism can both increase and shift privacy risks, making the retrieval database a target in its own right (Zeng 2024). From a red teaming perspective, this means that evaluation must include not only whether the model leaks training data but whether the retrieval layer leaks data, and whether leakage can occur indirectly through inference rather than explicit quotation.

The emerging research on membership inference against RAG strengthens this concern by showing that even when a system is designed to be grounded, attackers may still infer whether certain content exists in the retrieval store. This is concerning because existence can itself be sensitive such as whether a merger document exists, whether a vulnerability report exists, or whether a user belongs to a confidential project (Wang 2025). What matters for evaluation is therefore not only disclosure of content but also disclosure of metadata and structure, including existence, frequency, and relationship signals that can be inferred from responses and tool behavior.

Privacy attacks such as membership inference also provide a conceptual bridge between classical ML risks and agentic systems. Even when the system does not print sensitive content verbatim, it may respond in ways that allow an adversary to infer whether a particular record is present, whether a particular policy exists, or whether a user belongs to a privileged group. Surveys of membership inference attacks show that inference can be subtle and that defenses must consider both the model and the surrounding system because observables include confidence patterns, refusal behavior, and other side channels (Hu 2022). For AI agents, the analogous concern is that tool selection, error codes, latency, or refusal phrasing can serve as an oracle about the presence or absence of sensitive resources.

2.4   Persistence & Guardrail Limitations

A third systems-level risk is persistence through memory and caching. Many AI agents store conversational summaries, user preferences, or task artifacts to improve personalization and reduce friction. Yet persistence can create delayed vulnerabilities: a poisoned memory entry can influence future actions; a cached tool result can be replayed out of context; and a mis-scoped memory can leak information across users (Chhabra 2026). The agentic AI security literature treats memory and autonomy as distinctive risk amplifiers because they enable influence to persist across time and across tasks, turning one successful manipulation into a long-lived foothold in the agent’s decision loop.

Systematic reviews of LLM use in cybersecurity are valuable here because they show how quickly deployment patterns are expanding. When LLMs are applied to vulnerability detection, incident triage, threat intelligence summarization, and automation of operational workflows, they increasingly handle sensitive data and become deeply coupled to security-critical decisions. That coupling raises the stakes for agentic failures because the agent’s output may be acted upon by humans who assume it is reliable or by automation that assumes it is authorized (Zhang 2025b). A security posture that is adequate for a productivity assistant can be inadequate for a SOC automation agent because the latter operates closer to the control plane.

A systems framing also clarifies why guardrails often disappoint. Guardrails that operate only on the final text output can miss tool-call attempts and can be bypassed when downstream systems interpret outputs as commands. Guardrails that operate only by adding warnings into the system prompt can be undermined when the model is exposed to conflicting instructions in retrieved data. Guardrails that operate only by blocking specific strings can fail when the model produces semantically equivalent but syntactically different outputs. None of these limitations implies that guardrails are useless. Rather, they imply that effective guardrails must be positioned at enforcement points where the system can be made deterministic such as middleware validators, authorization checks, and tool capability restrictions (OWASP 2024).

2.5   Organizational Responsibility & Design Implications

Finally, agent risk is exacerbated by the distribution of responsibility across components and teams. In many deployments, one team owns the model choice, another owns the orchestration, another owns the retrieval index, and another owns the tool integrations with security reviews occurring late. This division is operationally reasonable but security-dangerous if no one owns the boundary invariants end to end. A systems threat model therefore needs to be a living artifact that identifies which component enforces which invariant, and red teaming needs to be designed to test the invariants at the boundary, where they matter rather than at the layer where they are easiest to observe.

The consequence of this systems view is that the most important security questions for AI agents are architectural (NIST 2024). Which inputs are treated as instruction and which as data, and how is that separation enforced? Which tools exist, what permissions do they carry, and where is authorization validated? How does retrieval label provenance and trust, and how does it prevent untrusted content from becoming control signals? What telemetry exists to reconstruct tool decisions and state changes, and how is that telemetry balanced against privacy requirements? These questions motivate a T&E methodology that begins with authority profiling, treats failure as boundary violation, and evaluates the system through campaigns that emulate realistic adversarial adaptation while remaining safe and reproducible.

A red teaming approach starts by deciding what counts as failure. In traditional software security, a vulnerability is a violation of an intended property such as memory safety, authorization, or confidentiality, and a test is valuable only when it can reliably detect that violation. The analog for AI agents is that failures must be defined in terms of observable boundary outcomes rather than in terms of persuasive text. The most useful failure definitions for enterprise settings are invariants that should hold across all tasks, including invariants about who can access what data, which actions require approval, and what data may appear in outputs. The reason to insist on invariants is that they transform an amorphous safety concern into a property that can be monitored, tested, and improved through regression (NIST 2024).

To make this concrete, consider an agent that can retrieve documents and create tickets. A superficial evaluation might ask whether it refuses to reveal confidential information but a formal evaluation asks whether any sequence of user messages and retrieved content can cause it to retrieve documents outside the user’s authorization, to include sensitive snippets in its outputs, or to create tickets that embed secrets or that trigger downstream automation. In this view, the same user prompt can be safe or unsafe depending on what the agent is allowed to do, which is why red teaming must begin not with clever prompts but with authority profiling: an explicit inventory of tools, permissions, parameter domains, retrieval connectors, memory policies, and middleware checks (Manès 2021). Authority profiling is the analog of threat modeling in fuzzing: before generating inputs, you must understand the input surface and the failure oracle.

3.1 Authority Profiling & Campaign-Based Testing

Authority profiling should be operational rather than aspirational. It is not enough to document what the agent is supposed to do. The profile must reflect what it can do in the deployed environment. That includes the identity under which tools run, the scopes of tokens, the breadth of API permissions, and the existence of any hidden capabilities such as file system access or administrative endpoints. In practice, this profiling often reveals a mismatch between intention and reality such as a tool token that has wide read permissions because it was easiest to provision. From a security standpoint, the mismatch matters because adversaries exploit capability, not documentation. Red teaming that begins after this profiling can focus on the most dangerous boundaries rather than dispersing effort across low-impact behaviors.

Once invariants and authority are defined, the methodology proceeds by organizing tests as campaigns rather than as single prompts. The core reason is that many failures require multi-turn persuasion, gradual goal shifting, tool chaining, or time-separated memory effects. Campaigns also reflect how real adversaries behave. They probe for what is possible, observe partial refusals and error messages, and adapt. In prompt injection research, benchmark-driven evaluation makes this adaptation measurable by defining multi-stage tasks and success conditions across a matrix of models and defenses (Liu 2024). Red teaming should borrow this logic and treat each campaign as an experimental protocol with controlled variables, logged evidence, and clear endpoints.

The notion of a campaign is not merely more prompts. It is a structured sequence with a hypothesis about the boundary under test, a plan for how an adversary might attempt to violate that boundary, and a definition of what evidence would confirm or refute the hypothesis. For example, a campaign testing cross-user isolation should not only ask for another user’s data. It should also test whether indirect artifacts can change the agent’s interpretation of identity, whether the agent can be induced to use a higher-privileged tool token, and whether tool errors leak clues about the existence of the other user’s resources (Chhabra 2026). This is precisely why campaigns benefit from diverse personas, including an external attacker who can influence public content, a low-privileged insider who can access internal documents, and a compromised tool or connector that can return adversarial outputs.

In practical environments, it is helpful to treat campaigns as experimental protocols with a fixed structure: a setup phase that establishes the baseline task, an influence phase that introduces adversarial content through a defined channel, an action phase where the agent is induced to call tools or access resources, and a verification phase where the evaluator checks logs and state diffs against invariants. This protocol lens helps teams avoid a common red teaming failure mode in which tests become a series of unrelated prompts and the findings become impossible to aggregate. It also supports automation, because a protocol can be encoded in a harness that executes and scores runs repeatedly.

3.2   Manipulation Pathways: Direct, Indirect, Tool, Memory

Campaign design benefits from a taxonomy of manipulation pathways that correspond to where untrusted content enters the loop; see Figure 3 below. Direct manipulation includes user messages that attempt to override policy, to request restricted information, or to request out-of-scope actions. Indirect manipulation includes malicious instructions embedded in retrieved documents, emails, or tool outputs that the agent later consumes. Tool manipulation includes adversarial responses from tools such as error messages that contain instruction-like content, or data fields that embed prompts. Memory manipulation includes attempts to poison persistent memory or to cause the agent to store and later act on unsafe directives. These pathways are not mutually exclusive. Sophisticated failures often chain them, for example by using indirect injection to provoke a tool call whose output then becomes a new influence (Zhan 2024). What matters for T&E is that the campaign explicitly states which pathway is being exercised and what boundary is being tested so that results can be interpreted and reproduced.

Figure 3:   Taxonomy of AI agent manipulation – 4 critical attack pathways, including direct prompting, indirect retrieval-based injection, tool-response manipulation, and memory poisoning

A distinctive challenge in red teaming is safety. The evaluator must not cause real harm while testing for harm. The solution is to use sandboxes with synthetic but realistic corpora and to instrument tools so that high-impact actions are simulated or require explicit approvals. This mirrors how security researchers test web vulnerabilities in staging environments rather than production. The literature emphasizes the value of controlled benchmarks and safe test harnesses, because they enable systematic exploration without unacceptable operational risk (Chhabra 2026). In practice, red teams can implement a shadow execution mode in which tool calls are logged and validated but not executed, or executed against a synthetic twin of production data, allowing the evaluator to observe attempted boundary violations without causing damage.

The design of the sandbox is itself part of the security program because it determines what evidence can be observed. A strong evaluation environment logs the entire decision loop, including the assembled context, the retrieved artifacts, the tool calls, the validator decisions, and any downstream state diffs. It also makes it easy to replay and minimize failures because reproducibility is the prerequisite for remediation.

T&E also requires careful construction of failure oracles because agents can fail in ways that are not visible in their text. The oracle for a sensitive-data invariant must include all output channels, including tool call arguments, logs that might be exposed to users, and any files or tickets the agent creates. The oracle for an authorization invariant must include attempted access, not only successful access, because an attempted out-of-scope tool call indicates a control weakness even if middleware blocks it. The oracle for an integrity invariant must include state diffs in downstream systems. A crash is the easiest oracle, but the most important vulnerabilities are logic flaws that require richer oracles such as invariant checks and semantic assertions. AI agent testing similarly needs semantic oracles that interpret traces and state changes.

3.3   Automation & Scaling of Campaigns

Because AI agents rely on language, a common question is how much of red teaming can be automated. The short answer is that automation can help generate and mutate test cases but human judgment remains important in selecting realistic attacker goals and in interpreting ambiguous boundary outcomes. Prompt injection benchmarks illustrate a useful hybrid: they define tasks and then evaluate many attacks and defenses automatically, allowing systematic comparison without relying on anecdotal examples (Liu 2024). Organizations can adopt a similar approach by defining a library of campaign templates that are instantiated across tools, connectors, and policy contexts while still allowing human red teamers to refine hypotheses and triage findings.

A further insight from recent research is that some defense strategies can themselves be evaluated as inverted attacks. Work on defending against prompt injection by leveraging attack techniques highlights that the boundary between attack and defense is often a matter of intent: both aim to steer the model toward one instruction and away from another (Chen 2025). This suggests that red teams should not only test whether a defense reduces attack success but also whether the defense introduces new failure modes such as over-refusal, brittle behavior under paraphrase, or leakage through alternative channels.

At scale, the challenge is not merely to run campaigns once but to keep running them as the system evolves. This requires that the organization treats the campaign library as code and invests in maintenance. When a tool’s schema changes, tests must be updated. When a connector is added, new campaigns must be instantiated. When a retrieval corpus evolves, corpora snapshots must be versioned so that regressions can be distinguished from distribution shift. The process resembles how organizations maintain security regression tests for critical web workflows: the work is continuous, and the payoff is stability and confidence.

The campaign library should also include negative controls that verify the harness itself. For example, a campaign that is known to be safe should remain safe. If it begins to trigger violations, the harness may be mis-scoring or the system may have regressed. Conversely, a campaign known to be unsafe in a baseline configuration should become safe when mitigations are enabled. If it does not, the mitigation may be ineffective (Biggio 2018). This experimental discipline is borrowed from robust evaluation practice in AML, where defenses must be validated against adaptive attack variants to avoid false confidence.

3.4   Reproducibility, Triage, Regression Harness

Because stochasticity complicates reproducibility, the methodology must include practices that make tests stable enough for regression. This includes pinning model versions, prompts, tool schemas, and corpora snapshots; controlling randomness when possible; and running tests multiple times to estimate variance. The goal is not perfect determinism but stable detection of boundary violations with acceptable confidence. In AML, evaluation that ignores variance can mislead by underestimating risk because rare failures can be catastrophic (Biggio 2018). Therefore, we should treat repeated trials and confidence intervals as part of the evidence, especially for high-impact boundaries where a one-in-a-thousand failure rate may still be unacceptable at scale.

A further practical element is triage and minimization. When a campaign produces a failure, the evaluator needs a way to reduce the interaction to a minimal reproducer that isolates the cause. This again parallels fuzzing, where minimization reduces a long input into a small crashing sample that developers can debug. For agents, minimization may involve reducing the number of turns, isolating which retrieved document triggered behavior, or narrowing which tool parameter caused an unsafe state change. Minimization is essential because without it, findings become anecdotes that are hard to fix and harder to regress (Manès 2021). A mature red teaming pipeline should therefore include tooling to replay traces, swap components, and systematically ablate context.

The output of this approach is not merely a report; it is a suite. That suite includes an authority profile, a set of invariants, a corpus of test artifacts (including adversarial documents and tool responses), a collection of campaigns with expected outcomes, and a set of evidence templates that specify what logs and state diffs must be collected. Over time, the suite becomes a regression harness that is run automatically for each release, making agent security a continuous property rather than a periodic audit. This is aligned with lifecycle risk management guidance that treats evaluation as an ongoing process because systems evolve rapidly and because threat landscapes shift (ISO/IEC 2023).

The most common measurement mistake in LLM testing is to treat attack success as a binary property of the model’s text response. This is understandable because text is visible and easy to score but it misrepresents the risk in agentic systems, where tool calls and downstream state changes are the real hazard. A system may refuse in text while still attempting an out-of-scope tool call, or it may produce a seemingly harmless summary that embeds a harmful instruction consumed by downstream automation. Therefore, evaluation should treat “what executed” and “what changed” as first-class observables, and it should score outcomes at multiple layers (Yao 2024).

4.1   Boundary Metrics Trio: Violations, Pressure, Coverage

A practical metric framework begins by distinguishing three categories: boundary violations, boundary pressure, and boundary coverage; see Figure 4 below. Boundary violations are the outcomes that should never occur such as unauthorized retrieval, cross-user leakage, unsafe writes, or creation of artifacts that expose secrets. Boundary pressure captures attempted or near-miss events such as out-of-scope tool call attempts blocked by middleware, or partial leakage such as revealing existence of a confidential document without revealing its content. Boundary coverage measures how thoroughly tests exercised the authority surface, including the diversity of tools invoked, the range of parameter values explored, and the variety of trust-boundary transitions attempted. This tripartite view matters because a system that rarely violates boundaries but frequently applies pressure may be one change away from catastrophe, while a system that shows low pressure may simply be under-tested (Manès 2021). The structure borrows from the intuition of security testing in complex systems: exploitability, not just exploitation, is a leading indicator of risk.

Figure 4:  The boundary metrics trio as a comprehensive framework for testing and measurement of AI agent security – realized violations, near-miss pressure, and coverage over the authority surface

Boundary violation metrics should be aligned to impact, which depends on what the agent can do (ISO/IEC 2023). For a read-only assistant, leakage is the primary risk; for a write-capable agent, integrity and availability become equally central because actions can damage systems. A severity index should therefore incorporate impact magnitude, scope, exploitability, and detectability. Impact magnitude includes the cost of a leaked secret, the operational consequences of an unauthorized write, and the reputational and compliance implications of exposure. Scope includes how many systems and users are affected, and whether the violation crosses tenants or organizations. Exploitability includes how much adversarial access is needed and how reliably the failure reproduces. Detectability includes whether monitoring would likely detect the violation before harm occurs. This framing aligns with broader risk-management guidance that demands measurable, comparable risk statements rather than purely qualitative labels.

Coverage deserves special emphasis because agentic systems have large, uneven attack surfaces. One reason red teaming can appear shallow is that it exercises only the chat interface and ignores tool surfaces, retrieval connectors, and memory. Coverage metrics should therefore be derived from the authority profile. If the agent has 15 tools, the test suite should report which tools were exercised. If a tool has a rich schema, the suite should report which parameter classes were explored. If the agent retrieves from multiple repositories, the suite should report which connectors were used and whether provenance labeling and access control were tested for each. In fuzzing, coverage-guided input generation has been transformative because it turns exploration into a measurable objective (Manès 2021). For AI agents, explicit boundary coverage can play a similar role by preventing “safe because untested” conclusions.

4.2   Evidence Collection & Privacy-Aware Logging

Evidence collection is the bridge between measurement and remediation. A finding that cannot be reproduced, explained, or traced to a component will not be fixed reliably, especially in large organizations. Evidence standards therefore need to be defined in advance. For each invariant, the standard should specify which logs must be captured, including model inputs and outputs, retrieved documents and provenance, tool call requests and responses, middleware validation decisions, and downstream state diffs. It should also specify what constitutes a minimal reproducer such as a compact conversation trace, a specific retrieved artifact, and a tool call sequence that produces the violation. This evidence orientation follows the engineering discipline that makes fuzzing and vulnerability triage effective: developers need concrete artifacts, not narratives.

Because systems often have privacy and compliance constraints, evidence collection must also be privacy-aware. Logging everything can violate privacy; logging too little can destroy auditability. Research on privacy risks in language models highlights that exposure can occur through many channels and that protection often involves trade-offs between utility, observability, and privacy guarantees. Differential privacy research illustrates this trade-off formally: noise can reduce leakage but also reduce signal for monitoring and debugging. In red teaming, the practical implication is that auditability must be designed as a security feature, with mechanisms such as selective logging, privacy-preserving aggregation, and protected audit stores rather than treated as an afterthought (Wei 2020). The red team should explicitly test whether incident response remains feasible under the organization’s privacy posture.

4.3   Measurement to Action: Tail Risk, Operational Insight

The evaluation should also account for model drift and system evolution. Even if the base model is stable, prompts change, retrieval corpora evolve, connectors are added, and tool schemas are modified. Therefore, measurement must be versioned. Every test run should record the model identifier, prompt hash, tool schema version, retrieval index snapshot, and middleware policy version. This allows comparisons across releases and supports regression when a previously fixed boundary violation reappears. A related measurement challenge is stochasticity and long-tail risk. Rare failures matter in enterprise settings because scale amplifies probability. If an agent handles a million tasks per day, a one-in-a-million vulnerability is a daily event. Security evaluation therefore needs to report not only mean performance but tail behavior under adversarial pressure. In AML, robustness evaluation often reveals that models behave well on average but fail catastrophically on a small subset of inputs crafted to target weaknesses (Biggio 2018). A red team should mirror this by including stress campaigns that search for tail failures, and by reporting confidence bounds that reflect repeated trials and variability.

The privacy literature also motivates metrics that go beyond explicit leakage. Membership inference surveys emphasize that inference risk can be quantified by adversary advantage, and that defenses must be evaluated against realistic threat models rather than assumed benign query patterns. Translating this to AI agents suggests measuring not only whether confidential text appears but whether an adversary can reliably infer sensitive facts such as membership in a retrieval store or access to a particular system (Hu 2022). When combined with the evidence orientation above, such metrics help identify vulnerabilities that are otherwise dismissed as “not a leak” because no sensitive sentence was printed.

In addition to quantitative metrics, T&E programs need narrative synthesis that explains why a failure occurred and what control would prevent it. The key is to keep narrative tied to evidence. A persuasive write-up traces the event from the initial influence channel, through context assembly, through model decision, through tool call attempt, to the enforcement point where the boundary was crossed or where it was successfully blocked. This trace-based narrative is more useful than a transcript of the conversation because it points directly at the component that should be changed (Zhang 2025b). Systematic reviews of LLMs in cybersecurity repeatedly emphasize reproducibility and methodological rigor as prerequisites for claims about effectiveness, which supports the idea that evaluation outputs must be engineered artifacts rather than ad hoc anecdotes.

Lastly, actionable measurement requires that metrics connect to controls. If a boundary violation is reduced by adding a middleware validator, the metric should show decreased violation and decreased pressure. If a provenance labeling control is added to retrieval, the metric should show reduced susceptibility to indirect injection campaigns. This measure-before-and-after discipline is what turns security work into engineering rather than rhetoric. It is also how red teaming earns trust from leadership. It demonstrates that investments reduce measurable risk.

The central operational lesson of AI agent security is that the most reliable controls live outside the model. Prompts can help steer behavior but they are not enforceable boundaries. A manipulated model can still attempt dangerous actions, and a model can still be fooled by indirect context. Therefore, defense in depth for AI agents should prioritize hard boundary enforcement at the tool and retrieval layers, supported by monitoring and governance that make failures detectable and improvable. This is not a rejection of model alignment. It is an acknowledgment that security requires controls that do not depend on the model’s goodwill.

5.1   Tool-layer Controls: Authorization, Validation, Capability Design

At the tool layer, the highest leverage control is server-side authorization with least privilege. Tools should execute under identities whose permissions match the minimal requirements of the task, and authorization should be validated by the tool service itself, not delegated to the model. Parameter validation should enforce safe domains, preventing the model from smuggling unsafe operations through creative strings. High-impact actions should be gated by explicit approvals or by multi-party authorization, especially for write operations that affect production systems. These practices are familiar in traditional security engineering but agents reintroduce their importance because a tool call is essentially untrusted input from a probabilistic controller.

A second tool-layer control is capability shaping through schemas and contracts. Tool descriptions should communicate constraints clearly but more importantly, the orchestration layer should enforce those constraints by rejecting out-of-policy tool calls. This includes rejecting calls that request disallowed scopes, that contain suspicious parameter patterns, or that attempt to chain tools in ways that violate policy. Research on tool-integrated agent vulnerabilities shows that tool calling introduces new attack surfaces, including manipulation of tool selection and injection through tool metadata (Zhang 2025b). This reinforces the need to treat tool schemas as security interfaces. The tool catalog is a privileged API surface and must be subject to the same review discipline as any production API.

Tool contracts also need to acknowledge that models can be wrong in predictable ways. Models may fabricate required parameters, swap identifiers, or overgeneralize from partial context. A robust contract therefore includes defensive defaults, idempotency where possible, and explicit refusal states at the API boundary. This mirrors lessons from safety engineering in distributed systems: components must be designed so that failures are contained rather than amplified. While these lessons predate LLMs, they become newly salient because the LLM is a high-variance component placed at the center of orchestration.

5.2   Retrieval-layer Controls: Provenance, Access, Privacy

At the retrieval layer, the core controls are provenance, access control, and instruction-data separation. Provenance means that retrieved content carries labels about its source, trust level, and authorization scope. Access control means that retrieval respects user identity and context, preventing cross-user or cross-tenant retrieval. Instruction–data separation means that retrieved content is treated as untrusted data and is framed in the context window as such, with explicit delimiters and policies that instruct the model not to treat it as instruction. While delimiters are not perfect, they are more effective when combined with additional mechanisms such as content filtering, policy-based chunk selection, and refusal to retrieve untrusted sources for high-stakes tasks (Zhan 2024). Benchmarks for indirect prompt injection in tool-integrated agents show that external content is a key manipulation channel, which is why retrieval-layer security must be a first-class design concern.

Because retrieval also raises privacy risks, controls must include safeguards against retrieval database leakage. Studies of privacy in RAG demonstrate that attackers can target the retrieval corpus and infer its contents, while also showing that RAG can shift leakage patterns compared to pure parametric models. This suggests a dual strategy: limit exposure by enforcing strict authorization on retrieval and by minimizing what is returned to the model, and reduce inference by controlling responses, rate limiting, and auditing suspicious query patterns. The broader literature on privacy risks in language models emphasizes that privacy protection is not a single technique but a system of choices that spans data collection, model training, retrieval, and output handling (Chen 2025).

5.3   Runtime Isolation & Monitoring

A third layer of controls is runtime isolation and segmentation. In security practice, segmentation limits blast radius by separating networks, identities, and privileges. For agents, segmentation translates into separating tools by risk class, isolating high-impact tools behind approval gates, and preventing the model from using arbitrary connectors. It also includes isolating untrusted content sources from privileged instruction channels, which can be implemented through context compartmentalization and strict mediation of what tokens can influence what decisions (Chhabra 2026). It is emphasized that autonomy is a risk amplifier, which makes segmentation especially important for agents that can plan and act repeatedly without human oversight.

Monitoring and detection are the fourth pillar of operationalization. In agentic systems, monitoring must extend beyond conventional logs to include tool-call traces, policy decisions, and retrieval provenance. It must also detect boundary pressure events such as repeated attempts to access restricted resources, because these are often precursors to successful exploitation. However, monitoring can conflict with privacy and with the desire to limit retention of sensitive data. Effective governance reconciles this by defining what must be logged for accountability, what must be redacted or protected, and how long evidence is retained. Risk management standards emphasize the importance of accountability and traceability, and agent governance should treat these as design requirements, not operational afterthoughts.

5.4   Governance, Response, Continuous Improvement

Governance becomes concrete when it is embedded into engineering workflows. A useful pattern is to treat every agent release as a change to a safety-critical system that must satisfy a small set of non-negotiable invariants while allowing rapid iteration in lower-risk areas. This implies release gates tied to the most consequential invariants, combined with an exception process that documents accepted risk and requires explicit ownership. It also implies that regression suites are treated like unit tests: they run continuously, and failures block release. This approach resonates with how mature organizations govern other high-risk systems, and it is consistent with lifecycle-oriented risk management approaches.

Operational maturity also requires education and process alignment. Prompt injection is not a model bug but a boundary design problem, and that effective defense requires engineering controls. We should recognize that agents blur the line between application security, data governance, and privacy, requiring cross-functional collaboration. Successful adoption depends on process and governance as much as on model capability. In practical terms, this means integrating agent security checks into architecture reviews, threat modeling sessions, and incident response planning, rather than relegating them to late-stage testing (Zhang 2025a).

A practical governance program must also plan for incident response and recovery. Because agentic failures can produce real state changes, organizations need procedures to detect suspicious tool usage, to roll back unsafe changes, and to rotate credentials if an agent is suspected of having been manipulated. This is where auditability, least privilege, and approval gates become mutually reinforcing. They reduce the chance of catastrophic actions and they increase the chance that when something goes wrong it is detected quickly and can be contained (Chhabra 2026). Evaluation and governance are treated as complementary since defenses are not durable without continuous adversarial evaluation and evaluation is not meaningful without clear operational controls.

Lastly, mitigation must be validated experimentally. A control that sounds plausible but does not reduce boundary violations under realistic campaigns is not a control; it is a hypothesis. The T&E methodology therefore requires re-running the same campaign suite before and after mitigations, comparing boundary violation rates, pressure metrics, and detectability. This is particularly important for controls that may create new failure modes such as filters that block benign content or that can be bypassed by adversarial formatting (Biggio 2018). In AML, defenses that are not evaluated adaptively often fail when attackers adjust, which is why mitigation validation should include adaptive testing that mirrors how adversaries respond to controls. Operationalizing security means moving from model-centric guardrails to system-centric boundary enforcement, using red teaming not as a one-time exercise but as a continuous measurement and improvement loop.

Here we provided a framework for red teaming agents that starts by translating policy into explicit invariants and profiling the agent’s real authority so that testing targets what the system can actually do; see Figure 2. It then evaluates the system through multi-turn campaigns that probe direct and indirect manipulation pathways and treats failure as an observable violation at the output, tool-call, or state-change layer, prioritizing execution traces and state diffs over persuasive narrative responses; see Figures 1 and 3. This approach is grounded in lessons from software security testing and adversarial evaluation: systematic discovery requires clear failure oracles, coverage, reproducibility, and triage that turns failures into fixable defects (Manès 2021).

This framing matters because AI agents change the risk profile of LLM deployment: they connect language generation to operational authority; see Figure 2. The consequential failures are therefore not primarily unsafe sentences but boundary violations in information flow and action execution (unauthorized retrieval, cross-user leakage, unsafe tool invocation, and persistent manipulation via memory). Thus, treating the problem as “prompting” alone misses where the risk actually lives. The right framing is systems security. The agent is a control plane, and the security objective is to enforce boundaries independently of model behavior.

Figures 1 through 4 provide a staged view of this work. In Figure 2, we locate risk in the agent’s system architecture by identifying the trust boundaries across input, instruction, retrieval, tools, and state. What constitutes failure across that architecture is defined in Figure 1: violations may appear in the agent’s narrative output, in attempted or executed tool invocations, or in downstream state changes. With the architecture and failure layers defined in Figures 1 and 2, Figure 3 organizes red-team campaigns by the principal manipulation pathways through which adversarial influence reaches those boundaries. In summary, Figure 2 specifies where influence enters and propagates, Figure 1 specifies what counts as failure when it does, and Figure 3 specifies how campaigns are constructed to induce and test those failures. Figure 4 then describes how those pathways are operationalized in testing through metrics for realized violations, near-miss pressure, and boundary coverage. Together, the figures move from where risk exists, to what failure means, to how failure is induced, to how it is measured.

The operational lesson is that measurement and controls determine whether findings reduce risk. Headline jailbreak rates are insufficient for AI agents. Boundary-aware metrics, pressure indicators, coverage reporting, and privacy-aware evidence standards are needed to make results actionable and auditable with severity grounded in organization impact rather than rhetoric (Chhabra 2026). The highest-leverage mitigations enforce boundaries outside the model (least-privilege tool design, server-side authorization, middleware validation, retrieval provenance and access control, and approval gates) so red teaming becomes a continuous engineering discipline when mitigations are validated experimentally and the campaign suite becomes a release-blocking regression harness. It should be also emphasized that a mature T&E program cannot rely on a single framework document or a single checklist because agentic risk emerges from the interaction of software security, machine learning robustness, privacy, and operational governance. The T&E program succeeds when it makes those interactions explicit and engineerable, turning vague fears about unsafe AI into concrete boundary properties that can be tested, monitored, and enforced.

Biggio, B., and Roli, F. (2018). “Wild patterns: 10 years after the rise of adversarial machine learning,” Pattern Recognition, 84: 317–331.

Chen, K., Zhou, X., Lin, Y., Feng, S., Shen, L., and Wu, P. (2025). “A survey on privacy risks and protection in large language models,” Journal of King Saud University Computer and Information Sciences, 37: e163.

Chen, Y., Li, H., Zheng, Z., Wu, D., Song, Y., and Hooi, B. (2025). “Defense against prompt injection attack by leveraging attack techniques,” Proceedings of the Annual Meeting of the Association for Computational Linguistics, 18331–18347.

Chhabra, A., Datta, S., Nahin, S.K., and Mohapatra, P. (2026). “Agentic AI security: threats, defenses, evaluation, and open challenges,” IEEE Access, 14: 49455-49482.

Cloud Security Alliance (2025). Agentic AI Red Teaming Guide (https://cloudsecurityalliance.org/artifacts/agentic-ai-red-teaming-guide)

Hu, H., Salcic, Z., Sun, L., Dobbie, G., Yu, P.S., and Zhang, X. (2022). “Membership inference attacks on machine learning: a survey,” ACM Computing Surveys, 54: e235.

ISO/IEC (2023). “Information technology – AI – guidance on risk management,” International Organization for Standardization (https://www.iso.org/standard/77304.html)

Liu, Y., Jia, Y., Geng, R., Jia, J., and Gong, N.Z. (2024). “Formalizing and benchmarking prompt injection attacks and defenses,” Proceedings of the USENIX Security Symposium, 1831–1847.

Manès, V.J.M., Han, H., Han, C., Cha, S.K., Egele, M., Schwartz, E.J., and Woo, M. (2021). “The art, science, and engineering of fuzzing: a survey,” IEEE Transactions on Software Engineering, 47: 2312–2331.

NIST (2024). “Artificial intelligence risk management framework: generative AI profile,” National Institute of Standards and Technology (https://doi.org/10.6028/NIST.AI.600-1)

NIST (2025). “Adversarial machine learning: a taxonomy and terminology of attacks and mitigations,” National Institute of Standards and Technology (https://doi.org/10.6028/NIST.AI.100-2e2025)

OWASP (2024). “OWASP top 10 for LLM applications,” Open Worldwide Application Security Project Foundation (https://owasp.org/www-project-top-10-for-large-language-model-applications/assets/PDF/OWASP-Top-10-for-LLMs-v2025)

Wang, G., He, J., Li, H., Zhang, M., and Feng, D. (2025). “RAG-leaks: difficulty-calibrated membership inference attacks on retrieval-augmented generation,” Science China Information Sciences, 68: e160102.

Wei, K., Li, J., Ding, M., Ma, C., Yang, H.H., Farokhi, F., Jin, S., Quek, T.Q.S., and Poor, H.V. (2020). “Federated learning with differential privacy: algorithms and performance analysis,” IEEE Transactions on Information Forensics and Security, 15: 3454–3469.

Yao, Y., Duan, J., Xu, K., Cai, Y., Sun, Z., and Zhang, Y. (2024). “A survey on large language model security and privacy: the good, the bad, and the ugly,” High-Confidence Computing, 4: e100211.

Zeng, S., Zhang, J., He, P., Liu, Y., Xing, Y., Xu, H., Ren, J., Chang, Y., Wang, S., Yin, D., and Tang, J. (2024). “The good and the bad: exploring privacy issues in retrieval-augmented generation (RAG),” Findings of the Association for Computational Linguistics, 4505–4524.

Zhan, Q., Liang, Z., Ying, Z., and Kang, D. (2024). “InjecAgent: benchmarking indirect prompt injections in tool-integrated LLM agents,” arXiv ( https://arxiv.org/pdf/2403.02691)

Zhang, J., Bu, H., Wen, H., Liu, Y., Fei, H., Xi, R., Li, L., Yang, Y., Zhu, H., and Meng, D. (2025a). “When LLMs meet cybersecurity: a systematic literature review,” Cybersecurity, 8: e55.

Zhang, R., Wang, H., Wang, J., Li, M., Huang, Y., Wang, D., and Wang, Q. (2025b). “From allies to adversaries: manipulating LLM tool-calling through adversarial injection,” Proceedings of the Association for Computational Linguistics: Human Language Technologies, 2009–2028.