A Successful Workflow Is Not Yet a Security Proof

Picture a mid-sized B2B SaaS company that sells account-management software to other businesses. Its engineering team ships an AI assistant embedded in the account view. An account manager types:

"Summarize the recent activity for Acme and tell me whether there are unresolved support issues."

The assistant identifies the account, pulls recent CRM activity, searches the support ticket system, references a few lines from the company's internal playbooks, and produces a clean, useful summary. It correctly names open tickets. It correctly flags an overdue renewal. It reads naturally, the account manager stops checking three separate tabs, and the feature is declared a win in the sprint review.

Before it shipped, QA ran the workflow end to end. They confirmed the button triggers the request, the prompt reaches the backend, the response renders without layout bugs, the correct customer's data appears, error states display sensibly when the CRM API times out, and the feature behaves consistently across Chrome, Safari, and Edge. Every test passed. By any conventional functional definition, the feature works.

Now a different set of questions arrives — not from QA, but from whoever eventually asks them, whether that's an internal security review, a customer's procurement questionnaire, or an incident.

Can a user logged into a different company's workspace retrieve Acme's information through this same assistant? Can a support ticket or an uploaded document that contains unusual phrasing change which tools the assistant decides to call? Does the model receive contract terms, internal notes, or billing metadata it never needed to answer the question that was actually asked? If the assistant's output is later inserted into an email, a workflow automation, or a rendered HTML view, can that text carry more authority than a customer-facing summary should have? Does anything from one user's conversation persist into another user's session? Does the CRM's own API enforce authorization independently of what the assistant's interface happens to display, or does it simply trust whatever the orchestration layer sends it? Does the AI execute with the account manager's actual permissions, or with a broader service identity that can do more than the account manager is allowed to do? Could any of the customer data involved end up sitting in a log file, an observability dashboard, or a third-party model provider's request history?

Functional testing did not answer any of those questions. It was never designed to. It confirmed that an authorized user, following the intended path, in a clean staging environment, gets a correct and well-formatted answer. That is a real and necessary result. It is also a narrow one.

This is the idea the rest of this article is built around:

A Successful Workflow Is Not Yet a Security Proof

Nothing here should be read as a claim that the hypothetical assistant above is broken, or that AI features are inherently unsafe. The questions above are exactly that — questions. Some SaaS teams already have solid answers to all of them, because they built authorization, retrieval scoping, and tool permissions correctly from day one. Others have partial answers. Many have never been asked the questions at all, because the AI feature was tested the way every other feature is tested: does it work for the user it's supposed to work for.

The distinction that matters is this: functional correctness proves that an AI feature can perform its intended job for an authorized user, under expected conditions. Security testing has to prove something structurally different — that the feature cannot cross boundaries it was never supposed to cross, even when the context changes, the user changes, the retrieved content changes, the permissions turn out to be broader than assumed, or a downstream system trusts the model's output more than it should.

A feature can be functionally correct in every scenario a QA script covers and still be architecturally insecure, because functional correctness is evaluated along the happy path, and security failures overwhelmingly live off it — in the tenant that wasn't tested, the document that wasn't expected, the field that was never meant to leave the database, or the tool call that technically succeeded but shouldn't have been allowed to.

The rest of this article walks the actual architecture behind an AI-powered SaaS feature, boundary by boundary, and lays out ten concrete tests — really, ten proof obligations — that a team can use to move from "the feature works" to "we know where the feature's authority stops."

---

Draw the Security Map Before Writing Security Tests

The first mistake many teams make is treating "the AI" as a single component to test — as if there were one black box that either behaves safely or doesn't. In a production SaaS assistant, the model is one part of a longer chain, and a meaningful share of the security risk lives outside it entirely.

A realistic AI-powered feature touches most or all of the following:

• the frontend that renders the conversation and displays results;
• authentication, which establishes who the user is;
• authorization, which governs what that user is allowed to see and do;
• a backend or application layer that receives the request;
• an AI orchestration layer that assembles context, decides what to retrieve, and coordinates tool calls;
• system instructions that define the assistant's role and constraints;
• a retrieval mechanism, often backed by a vector database, that pulls relevant documents or passages;
• one or more application databases holding structured account data;
• internal or third-party APIs the assistant can call;
• tools — discrete, named actions the model can invoke, such as "create ticket" or "look up customer";
• a third-party model provider that actually generates the completion;
• an output parser or renderer that takes the model's text and decides what to do with it;
• memory, whether short-lived conversational state or longer-lived personalization;
• logs, traces, and analytics that capture what happened for debugging and observability.

A vulnerability can exist in any layer of that chain, and it is a mistake to assume the model is where the risk concentrates. An authorization bug in the CRM API is exactly as dangerous whether the request that triggers it comes from a person clicking a button or a model generating a tool call — arguably more dangerous in the AI case, because the model can generate many differently-shaped requests quickly, in response to inputs a human tester would never think to try. A retrieval system that doesn't filter by tenant is a data breach waiting to happen regardless of how well-behaved the model itself is. A logging pipeline that captures full request payloads by default will happily record customer PII whether that payload originated from a REST call or an LLM's context window.

This is why "test the AI for security" is the wrong framing. The more useful framing is: test every boundary the request and the data cross on their way from the user to an answer and back.

The AI Trust Map

To make that concrete, this article introduces a working framework — the AI Trust Map. It is not an OWASP standard or an industry-recognized certification; it is an analytical tool for organizing security thinking around a specific architecture, and it is meant to be redrawn for whatever a given team has actually built.

[Visual concept: A layered architecture diagram showing the path from Browser → SaaS API → AI Orchestration Layer → RAG / CRM / Internal APIs → Documents/Vector Store, with a parallel path from Orchestration → Model Provider → Model Output → Application/Tool/User, and side surfaces for Memory, Logs, Telemetry, Third-Party Services, and Identity/Authorization. Each boundary crossing marked with a small lock icon and a question mark to represent an unproven trust assumption.]

The map identifies, for each boundary in the architecture, what is crossing it, under what identity, headed to what destination, and what should be true if the system is behaving correctly.

"Trust boundary" itself is a long-established application security concept, present in threat modeling long before generative AI existed. What is specific to this article is the particular set of ten boundaries above, mapped onto a realistic AI-augmented SaaS architecture, and the vocabulary used to reason about each one. A team adopting this approach should expect to redraw the table for their own system — the boundaries that matter will depend on which of retrieval, tools, memory, and external connectors are actually present.

It's worth being explicit about what this article deliberately avoids: turning into a walkthrough of the OWASP GenAI LLM Top 10 category by category. That list is genuinely useful — the 2026 edition (OWASP GenAI LLM Top 10 2026, published August 2026) reorders prompt injection, sensitive information disclosure, excessive agency, supply chain risk, data and model poisoning, unbounded consumption, misinformation, hidden context exposure, vector and embedding weaknesses, and improper output handling based partly on a corpus of real incident data for the first time. The companion OWASP Top 10 for Agentic Applications (ASI01–ASI10, published December 2025) covers what happens once a model stops being a component that returns text and starts being an actor that plans, holds memory, and calls tools with delegated authority. Both are referenced throughout this article where they are directly relevant, and a SaaS team building an AI feature should have both open on a second monitor.

But neither list is organized the way a SaaS engineering team actually needs to think about their own system. Those lists answer "what are the recognized categories of GenAI risk." This article tries to answer a narrower and more operational question: what should a SaaS team actually go test, given the specific architecture they built — which boundary, which identity, which data, which proof.

[Internal link opportunity: security testing]

---

Security Test 1 — Can One User Reach Another User's AI Context?

What is AI security testing? In its most concrete form, it is the discipline of proving — with tests, not assumptions — that an AI-powered feature cannot be made to cross a trust boundary it was designed to respect. The first and most consequential boundary in almost any B2B SaaS product is the tenant boundary, and it is the natural place to start.

Consider the same assistant from the opening scenario, now deployed across two customer workspaces: Workspace A (a company called Northline Logistics) and Workspace B (a company called Acme Fulfillment). Both workspaces use the AI assistant. Both have account managers who ask it questions about their own customers, their own tickets, their own documents.

The functional test for this feature confirms that a Northline user asking about a Northline account gets a correct, well-formed answer. It does not, by itself, tell you anything about what happens when a Northline user's request — through an identifier, a phrasing, a cached reference, or a retrieval query — ends up touching Acme's data.

This is not a hypothetical style of bug. It is a direct extension of one of the longest-standing and most common failure modes in web application security: broken object-level authorization, ranked API1 in the current OWASP API Security Top 10 (2023 edition, still the operative version as of 2026). BOLA occurs when a system correctly authenticates a user — it knows who they are — but fails to verify that they are authorized to access the specific object they are requesting. In a conventional REST API, this shows up as changing a numeric ID in a URL and getting back someone else's record. In an AI-augmented feature, the mechanism is less visible but the underlying failure is the same: the AI orchestration layer, the retrieval system, or a tool call fails to independently verify that the requesting user's tenant matches the tenant of the data being returned.

The reason this deserves to be the first test in this article, rather than a footnote under "access control," is that AI features change how the request for an object gets generated. In a traditional application, a developer writes the query. In an AI-augmented feature, the model — or the orchestration code around it — constructs the retrieval query, the CRM lookup, or the tool call dynamically, often based on natural-language references rather than explicit, validated identifiers. "Summarize activity for Acme" is not a parameterized, pre-validated request the way GET /accounts/482 is. It has to be resolved into one. That resolution step — turning a natural-language reference into a scoped, authorized data request — is a new place for tenant isolation to quietly fail, and it did not exist in the pre-AI version of the product.

Where isolation actually needs to hold

There are at least four places in the architecture where tenant scoping has to be enforced independently, because a failure in any one of them can produce a cross-tenant leak even if the other three are correct:

Object and tenant identifiers. Whatever the orchestration layer uses to resolve "Acme" into an actual account record needs to be constrained to the requesting user's own tenant before any lookup happens — not filtered afterward, and not left to the model to "remember" which company it's supposed to be talking about.

Retrieval scoping. If the assistant searches a knowledge base, support tickets, or documents via a vector index, the retrieval query itself needs a tenant filter applied at the database or index level, not merely a hope that semantically similar content from other tenants won't surface. Vector similarity search has no inherent concept of tenancy; if the underlying index mixes documents across tenants without a hard filter, a query about "unresolved support issues" can retrieve tickets that were never supposed to be visible to the requesting account manager, regardless of how the results are later formatted for display.

Backend and API authorization. This is the most important of the four, and the one teams most often skip, because it feels redundant once the orchestration layer "already checked." The CRM API, the ticketing API, and any internal service the assistant calls should enforce authorization on their own, independent of whatever the AI layer believes about the user's permissions. If the CRM API trusts every request that arrives from the orchestration service without re-verifying tenant and object ownership, then a bug anywhere upstream — in prompt construction, in a tool schema, in how the model resolves an ambiguous reference — becomes a full data exposure instead of a contained one.

Cached and conversational context. If the assistant retains recent context to make follow-up questions work — "what about their last three tickets?" — that cached context needs to expire, scope-check, or otherwise refuse to leak across a tenant switch, a logout, or a different user picking up what looks like the same conversation thread.

The distinction a CTO needs to internalize

The single most important idea in this section is one line: UI isolation is not authorization. Hiding another tenant's records from the search results a user sees is a usability feature, not a security control. It proves nothing about whether the backend would actually refuse a request for that data if one arrived through a slightly different path — a different phrasing, a direct API call, a tool invocation the UI doesn't normally trigger but the orchestration layer technically permits. Security testing has to test the boundary the UI is hiding, not just the UI.

Identity Continuity

This leads to the article's first named framework. Call it Identity Continuity — again, an analytical term introduced here, not an established industry standard, but a useful way to describe the property that actually needs to be true:

The authenticated user's identity and authorization context must remain intact, unmodified, and enforceable at every step as a request passes through the AI orchestration stack — from the frontend, through the backend, through the orchestration layer, through any tool call, to the data source, and back.

[Visual concept: A horizontal chain diagram — User → App → AI Orchestration Layer → Tool → Data — with a continuous colored line running underneath representing "identity," and a callout at each of the four transition points asking "Did authorization survive this hop?"]

The test is simple to state and genuinely useful to run: pick a request, trace the user's tenant ID and permission set as it moves through every hop in the chain, and confirm it is still present, still correct, and still being checked at the final data access point — not merely assumed to be correct because it was correct three hops earlier.

What evidence actually looks like

A defensible answer to "have we tested tenant isolation for the AI assistant" is not "we tried it and it seemed fine." It looks more like a small, repeatable authorization matrix:

• An authorized request for a user's own account succeeds and returns the expected data.
• An equivalent request, submitted under a different tenant's authenticated session but referencing the first tenant's account (by name, by inferred ID, or by any other resolvable reference the assistant might construct), is rejected — not filtered from view, rejected.
• The retrieval index, queried directly and independently of the assistant's UI, returns only documents belonging to the querying tenant.
• Every tool the assistant can call is exercised once under a session from the "wrong" tenant, with parameters that would, if unscoped, touch another tenant's data, and each call is confirmed to fail authorization at the tool or API level.
• Any cached or session-level context is confirmed to reset or refuse cross-tenant reuse after a session change, logout, or workspace switch.

None of this requires exotic tooling. It requires treating the AI assistant the same way a security-conscious team already treats every other multi-tenant feature — with paired test accounts across tenants and a habit of asking "what if the other request comes in through here instead."

[Internal link opportunity: API testing]

---

Security Test 2 — Does the Model Receive More Data Than It Needs?

The second test moves from "who can reach what" to a quieter and easier-to-miss problem: exposure that happens before the model generates a single word of output, simply because of what was placed in its context window.

Return to the same request: "What's the renewal date for Acme?" A well-designed response contains exactly that — a date. But consider what commonly happens on the way there. The account manager's question triggers a lookup that returns the full customer record, because it was simpler to fetch the whole object than to write a narrower query. That record includes internal account notes, billing metadata, support history, and possibly personal information about individual contacts. All of it gets assembled into the context sent to the model, because the orchestration code that builds the prompt was written for convenience, not for minimization. The model, doing exactly what it was asked, ignores the irrelevant fields and returns the renewal date.

Functionally, this is a complete success — the user got a correct, concise answer, and nothing looked wrong in the UI. Architecturally, something happened that a screenshot of the chat window will never reveal: a broad, sensitive dataset was assembled and transmitted to a component — the orchestration layer, and often an external model API — that had no need to see most of it. If that transmission crosses into a third-party model provider, it has left the SaaS company's own infrastructure with a footprint far larger than the visible answer would suggest. If a future bug in the same code path causes the model's response to be less selective — a plausible failure mode, since models are not deterministic filters — the reader learns everything that was quietly sitting in context. The information disclosure risk is not hypothetical because a specific bug will occur; it is present because unnecessary sensitive data has already crossed a trust boundary the moment it entered context, independent of whether the model's output ever surfaces it.

This concern connects directly to LLM02:2026, Sensitive Information Disclosure — the one entry in the 2026 OWASP GenAI LLM Top 10 where the community's vote and the incident data fully agreed on its severity, and one of the two risks that has held its position at or near the top of the list across every edition of the document. Disclosure doesn't require an attacker. It only requires broad context assembly and a downstream step — a rendering bug, a debugging session, a log capture, a future prompt change — that exposes what was already there.

Where the exposure actually happens

It is worth being precise about where in the pipeline this test needs to look, because the model's final text output is the least useful place to inspect:

• Retrieval payload. What does the query to the CRM, the document store, or the vector index actually return? Is it the full record, or a field-scoped subset relevant to the request type?
• Context assembly. What does the orchestration layer place into the prompt that will be sent to the model? This is often the single highest-leverage place to intervene, because it is code the SaaS team fully controls, independent of the model provider.
• Tool responses. If a tool call returns structured data to the model as an intermediate step — for instance, a get_customer tool returning a JSON object — does that object contain fields the current task has no use for?
• The outbound model request itself. For teams using a third-party model API, this is the literal payload leaving company infrastructure. It deserves the same scrutiny a security review would give to any outbound integration carrying customer data.
• Logs and traces. If the orchestration layer logs prompts for debugging (a common and reasonable practice), the logged payload is exactly as sensitive as the original context — arguably more so, because logs often have broader internal access than the live request path and longer retention.

Context Least Privilege

This article's second framework: Context Least Privilege.

The model — and every intermediate system that assembles its input — should receive the minimum information reasonably required to complete the specific task at hand, not the most convenient superset of data that happens to be available.

[Visual concept: Two side-by-side panels. Left panel labeled "Convenient context assembly" showing a wide funnel — full customer record, internal notes, payment metadata, support history — narrowing to a single visible answer. Right panel labeled "Context Least Privilege" showing a narrow, purpose-built query returning only the renewal date field, feeding directly into the same visible answer.]

This is a direct extension of the principle of least privilege — one of the oldest ideas in security engineering — applied to a place it doesn't traditionally get applied: information exposure to a component, rather than action authority granted to a component. Least privilege is usually discussed in terms of what a service account can do. Context Least Privilege asks the same question about what a service — including the model itself — is allowed to see, even transiently, even without ever acting on it.

Practically, this means treating context assembly as a designed interface, not an implementation detail. A few concrete disciplines make the difference:

Field-level filtering. Retrieval and lookup functions used to build model context should return purpose-specific subsets — a get_renewal_summary query, not a get_full_customer_record query reused everywhere for convenience — rather than the complete underlying object, filtered only at render time.

Purpose-specific retrieval. A document search triggered by "summarize support issues" has a narrower legitimate scope than a general knowledge-base search, and the retrieval call should reflect that scope rather than pulling the broadest set of "probably relevant" content and trusting the model to select the right pieces.

Data classification. Not all fields in a customer record carry the same sensitivity. Payment metadata, personal contact information, and internal-only notes deserve to be flagged and excluded from AI context by default, with a deliberate, reviewed exception process for the rare feature that genuinely needs them.

Redaction where appropriate. For fields that are useful in aggregate but sensitive in raw form — support ticket content that may contain a customer's own confidential business information, for instance — redaction or summarization before the data reaches the model's context is a reasonable mitigation, applied without straying into legal or compliance guidance a QA or engineering article shouldn't attempt to give.

None of this is exotic. It is the same discipline a careful engineering team already applies to API response shapes, database views, and analytics pipelines. The reason it needs explicit attention in an AI feature is that context assembly code tends to get written fast, iterated on quickly, and rarely reviewed with the same rigor as a public API contract — even though, functionally, it is a data export interface, just one whose consumer happens to be a model instead of a client application.

---

Security Test 3 — Can Untrusted Content Change the Agent's Behavior?

The third test addresses the risk most associated with generative AI specifically, and the one that has held the top position — LLM01 — across every edition of the OWASP GenAI LLM Top 10, including the 2026 release: prompt injection, and specifically its indirect form.

The mechanism is worth describing precisely, without describing a workable attack, because the difference between understanding the risk and being handed a recipe matters here. In direct prompt injection, a user types something into the chat interface intended to make the model ignore its instructions. That variant is comparatively well understood, and most teams have already put some thought into it. Indirect prompt injection is subtler and, per the current OWASP guidance, is the dominant real-world variant precisely because it doesn't require an attacker to interact with the system at all.

Here is the shape of it, without a payload: the AI assistant retrieves content it was designed to retrieve — a support ticket, an uploaded document, a CRM note, a page fetched from the web. That's normal, intended behavior; retrieval exists precisely so the model can incorporate outside information. The problem arises when that retrieved content contains text that reads, structurally, like an instruction rather than like data — and the model, which has no reliable, built-in way to distinguish "information I should summarize" from "a command I should follow," treats it as the latter.

The core security question this test needs to answer is not "can we stop the model from ever being fooled by this" — the 2026 OWASP guidance is explicit that this cannot be reliably guaranteed at the model level, describing prompt injection as present "everywhere a model reads untrusted input, which is to say everywhere." The right question is narrower and answerable: does retrieved data remain data, or can it become authority? Can a phrase embedded in a support ticket cause the assistant to call a tool it wasn't asked to call, disclose information it wasn't asked to disclose, or treat a later, less privileged instruction as if it came from the system itself?

Instruction Authority Boundary

This article's third framework helps make that question testable. Call it the Instruction Authority Boundary — a way of classifying every source of instruction-like text an AI system encounters, by how much authority it should be granted:

[Visual concept: A four-tier pyramid or nested-boundary diagram, with "System / Policy" at the top and narrowest, "Untrusted Data" as the widest base layer, and arrows showing that authority should only flow downward — data can inform an answer, but nothing in the untrusted tier should be able to reach upward and act as a system instruction.]

The core principle worth stating plainly: untrusted content should not be able to grant itself higher authority merely because the model is capable of reading and parsing it. A document being legible to the model is not the same as that document being authorized to redirect the model's behavior. This distinction sounds obvious stated abstractly and is genuinely difficult to enforce in practice, because most model architectures do not, by default, carry a strong structural separation between "instructions" and "data" once everything has been concatenated into a single context window. That is precisely why this needs to be a tested property of the system architecture, not an assumed one.

What the test actually verifies

Security testing for this boundary is defensive testing, and it should be described and conducted at the level of the property being verified, not at the level of specific adversarial phrasing that could function as a working attack. The categories of evidence a team should be gathering:

Policy remains intact. After the assistant processes a document containing unusual, instruction-like phrasing, does the assistant's actual behavior — the tools it calls, the data classes it accesses — still match the constraints defined by the system-level policy? Or did the retrieved content manage to shift what the assistant considered itself permitted to do?

Tool permissions do not expand. If the assistant has access to both a read-only tool and a state-changing tool, does content encountered during a read-only workflow ever cause the state-changing tool to be invoked when it otherwise wouldn't have been?

Sensitive data remains protected. Does exposure to a retrieved document ever cause the assistant to surface information — from a different customer's record, from a different part of the context, from earlier in the conversation — that the current request had no legitimate reason to touch?

Unexpected instructions in retrieved content do not become privileged directives. This is the most direct test of the boundary itself: when a document contains text that reads as an instruction, does the system's behavior change in a way that reflects following that instruction, as opposed to simply reporting or summarizing that the document contains unusual text?

Architecturally, the mitigations that make this testable and defensible line up with the same system-level hardening the 2026 OWASP guidance recommends: treating all retrieved content as data by default rather than as instructions; constraining which tools are even reachable during a given task, rather than relying on the model to decline to use them; requiring independent, server-side authorization checks on any tool call regardless of what prompted it, so that even a successfully manipulated model cannot act outside its already-granted authority; and, where the consequence of a tool action is significant, requiring a confirmation step that isn't itself mediated entirely by the same model that could have been influenced.

This is also where the boundary between the LLM Top 10 and the Agentic Top 10 becomes practically relevant rather than academic. The 2026 LLM list treats prompt injection as a risk to a model that receives input and produces output. The moment that output is allowed to trigger a tool call with real-world consequences, the risk has moved into ASI01 — Agent Goal Hijack — territory as defined in the OWASP Top 10 for Agentic Applications, and the mitigations shift from "be careful what the model reads" to "constrain what the agent can do regardless of what it was told." A SaaS team whose assistant only summarizes text is exposed mainly to the first framing. A SaaS team whose assistant can create tickets, send emails, or modify records is exposed to both, and needs both.

---

Security Test 4 — Can the AI Call a Tool With More Authority Than the User?

The fourth test is where AI security testing starts to look less like a new discipline and more like an old one — identity and access management — applied to a system that makes its own decisions about which actions to attempt.

Give the assistant a small set of tools: read_customer, create_ticket, update_customer, issue_credit, export_report. A support agent using the assistant has a defined role: they can read customer records and create tickets, but they cannot issue credits or export bulk reports — those actions require a different role in the underlying application.

The functional test confirms that when a support agent uses the assistant, it correctly reads customer data and correctly creates tickets. It likely never attempts to trigger issue_credit, because nothing in the intended workflow asks it to. The security question is different and rarely covered by the same test suite: when the AI executes a tool call, does it execute with the effective authority of the current user, or with the authority of the service account or integration credential that happens to sit behind the AI system?

This distinction is genuinely easy for a technical founder to miss, because it is invisible in normal operation. The AI service, as a piece of infrastructure, frequently needs broad backend access to function at all across many users and roles — it has to be able to serve a request from an admin as easily as from a support agent. If that broad backend credential is what actually executes every tool call, regardless of who initiated the request, then the AI system's effective authority is the maximum authority of anyone it serves, applied uniformly, rather than the specific, narrower authority of whoever is actually sitting at the keyboard.

Delegated Authority Gap

This article's fourth framework names the failure mode directly:

A Delegated Authority Gap exists when the AI-facing execution identity — the credential or service account that actually performs an action — can perform more privileged operations than the human context it is acting on behalf of should permit.

[Visual concept: A comparison diagram. Left side: "Correct delegation" — a support agent icon connected through a narrow, role-scoped channel to only the tools their role permits. Right side: "Delegated Authority Gap" — the same support agent icon connected through a wide service-account channel that has access to every tool, including issue_credit and export_report, with a highlighted crack where the user's actual role should have narrowed the channel but didn't.]

This is not a claim that every AI feature has this gap, and it should not be described that way to a reader. It is a specific, testable architectural question: does the system re-derive and enforce the initiating user's actual permission set at the moment of tool execution, or does it rely on the assumption that the model simply won't ask for anything the current user shouldn't have?

That assumption is fragile for reasons independent of any adversarial behavior. A model can misinterpret an ambiguous instruction. A prompt can be constructed in a way that makes an out-of-scope action seem, from the model's perspective, like a reasonable next step in fulfilling a legitimate request. A bug in how the orchestration layer decides which tools are "available" for a given conversation can expose the wrong tool to the wrong role. None of these require malicious intent from anyone; they only require the execution layer to trust the model's judgment about scope rather than independently enforcing it.

What good architecture looks like, and what the test verifies

The mitigation pattern here is a direct import from standard access-control engineering, applied at the tool-call boundary specifically:

Least privilege at the credential level. Wherever feasible, the AI system should execute tool calls using scoped, delegated credentials tied to the initiating user's actual role — not a single broad service account used uniformly for every request.

Server-side checks independent of the model. Every tool implementation should independently verify, at execution time, that the initiating user's role permits the requested action — the same check that would exist if the action were triggered by a button click rather than a generated tool call. This check should not live inside the prompt, inside the model's instructions, or inside orchestration logic that trusts the model's own stated intentions.

Resource-level checks, not just function-level checks. It is not enough to confirm the user's role can call update_customer in general; the check needs to confirm the user's role can update this specific customer record — the object-level authorization concern from Test 1, now applied to a write path rather than a read path, where the consequences of failure are typically worse.

Approval requirements scaled to consequence. For higher-impact tools — anything resembling issue_credit — a human confirmation step, gated by the same role-based authorization, adds a layer that doesn't depend entirely on the correctness of the automated check.

A concrete test plan follows directly from this: authenticate as a support agent, and attempt — through the assistant's normal conversational interface, phrased as a plausible, if out-of-scope, request — to trigger each of the higher-privilege tools. The expected, correct outcome is a clean rejection at the tool-execution layer, not merely the model politely declining because it judged the request inappropriate. The model declining is a nice-to-have; the backend refusing regardless of what the model decided is the actual security property, because it is the one that holds even when the model's judgment fails.

[Internal link opportunity: penetration testing]

---

Security Test 5 — What Happens When Model Output Becomes an Input to Another System?

The fifth test flips the direction of scrutiny. The first four tests examine what flows into the model — data, instructions, authority. This one examines what flows out, and it rests on a principle that should be uncontroversial once stated but is frequently absent from how AI features get built: model output is untrusted input to whatever system receives it next.

This is not a novel security idea. It is the same discipline applied to user-submitted form data for decades — never trust client input, validate and encode it before use. The reason it needs restating for AI features is that generated text has a way of feeling trustworthy that raw user input never quite does. It reads fluently, it's produced by "the system," and it's easy to forget that its content is, from a security perspective, exactly as unverified as anything a user could have typed directly, because in the indirect-injection scenario from Test 3, a user effectively can have influenced it, several steps removed.

Model output ends up in a wide range of destinations in a typical SaaS assistant: rendered as HTML or Markdown in the chat UI; inserted into an outgoing customer email; used as a search query against an internal system; passed as a parameter to a tool call; used to construct a filename; inserted into a structured JSON payload consumed by another service; or fed into an internal automation workflow that takes further action based on its content.

The risk is not uniform across these destinations, and treating it as uniform is itself a mistake — either through overreacting to low-risk cases or, more commonly, underreacting to high-risk ones because the team's mental model of "the AI just writes text" doesn't update as the destinations grow more consequential.

Output Authority Escalation

The article's fifth framework captures this gradient directly:

Output Authority Escalation describes how identical model-generated text carries different security consequences depending on how much authority the receiving system grants to it. The text itself doesn't change. What changes is what happens if that text turns out to contain something unexpected.

[Visual concept: A rising staircase diagram. Each step is labeled with a destination — "Customer-facing description text," "Search query parameter," "Structured tool-call parameter," "Internal automation trigger" — with the height of each step representing increasing downstream authority, and a warning icon appearing starting at the third step to indicate where independent validation becomes non-negotiable rather than optional.]

The practical guidance that follows is not exotic, but it is easy to skip under deadline pressure precisely because it feels redundant when "the AI already checked." It didn't check in the sense that matters here — the model isn't a validation layer, and expecting it to behave like one confuses a probabilistic text generator with a deterministic guard. The receiving system — the renderer, the tool executor, the workflow engine — needs to apply the same encoding, schema validation, and authorization discipline it would apply to any other untrusted source, because that is precisely what model output is, regardless of how confident or well-formatted it looks.

This section connects most directly to LLM10:2026, Improper Output Handling, in the current OWASP GenAI LLM Top 10. Notably, that entry's rank moved down in the 2026 reordering even as its scope grew — the 2026 edition explicitly expanded it to cover terminal and IDE rendering sinks that interpret escape sequences, and client renderers that automatically fetch external resources referenced in model output, both of which function as exfiltration channels if left unvalidated. The rank drop reflects other risks becoming more urgent in the 2026 incident data, not that improper output handling itself became less dangerous — a distinction worth remembering before treating list position as a priority signal on its own.

---

Security Test 6 — Can Memory Cross a User, Session, or Tenant Boundary?

The sixth test addresses a surface that is easy to overlook because it often isn't visible in the product at all: memory, in the broad sense of anything the system retains from one interaction that influences a later one.

Consider the lifecycle events a SaaS product routinely handles without a second thought: a user logs out and logs back in; a user switches from one customer workspace to another, if their account has access to multiple; an admin changes a user's role mid-session; a permission gets revoked; a record referenced earlier in a conversation gets deleted. In a conventional application, these events are handled by well-understood session and authorization mechanisms. In an AI-augmented feature with any form of persistent context — conversation history, cached retrieval results, personalization data, summarized long-term memory — each of these events raises a question that doesn't have an obvious default answer: does the AI system's memory actually respect the lifecycle event, or does it just keep going?

Context Residue

This article's sixth framework, built to describe this specific failure mode without reusing established agent-security vocabulary from elsewhere:

Context Residue is information created during one authorized interaction that remains available — and capable of influencing behavior or being surfaced — in a later context where it should no longer be visible or relevant, because the authorization, session, or lifecycle conditions that justified its presence have changed.

The critical discipline here is one stated plainly earlier in this article and worth repeating precisely because it is so easy to assume otherwise: the fact that the UI starts a fresh-looking conversation does not prove the backend actually isolated the underlying context. A "New Chat" button is a UI affordance. Whether it corresponds to an actual reset of server-side state — cached retrieval results, session-scoped embeddings, summarized memory records, personalization inputs — is an architectural question that has to be verified independently of what the interface implies.

Several distinct memory mechanisms deserve separate attention, because they fail differently:

Session memory. Conversational history retained for the duration of an active session. The test: does this genuinely clear on logout, or does session state persist longer than the authenticated session that created it, in a way that could let a subsequent user of the same browser or a subsequent authenticated session inherit it?

Persistent personalization. Longer-lived preferences or summarized facts the system retains across sessions to make future interactions more useful. The test: if a user's role changes, or their access to a particular account is revoked, does previously-stored personalization still reflect the old, now-invalid permission state, and can that stale state influence a future response in a way that reveals or acts on information the user should no longer be able to reach?

Cached retrieval. Results from an earlier retrieval operation, cached for performance. The test: is the cache keyed in a way that respects tenant and user scope, or is it possible for a cache entry populated by one user's request to be served to a different user whose query happens to match?

Server-side state tied to a specific conversation object. If a conversation is represented as a persistent object in the backend — common in products that let users revisit past AI interactions — the test: does access to that conversation object independently re-verify the requesting user's current permissions, or does it assume that whoever created the conversation is still authorized to see everything in it, even after a permission change?

The unifying test design across all four: force a lifecycle transition — logout/login, workspace switch, role change, permission revocation — and then attempt to elicit information or behavior that depended on the pre-transition state. A correct system either has no memory of the prior state at all, or re-validates current authorization before letting that memory influence anything. An incorrect system quietly carries the old context forward, and because the failure produces a plausible-looking, well-formatted answer rather than an error, it is exactly the kind of bug that passes functional testing indefinitely.

[Internal link opportunity: regression testing]

---

Security Test 7 — Can Poisoned or Low-Trust Data Enter the Knowledge Path?

The seventh test moves upstream, to the question of what gets into the system's knowledge sources in the first place, rather than what happens once it's there.

Every AI assistant with retrieval capability has a knowledge path — a set of documents, tickets, notes, or indexed content it treats as ground truth when answering questions. The security question this test asks is deceptively simple: who is allowed to add information to the AI's trusted knowledge sources, and does the system know, for any given retrieved fact, where it actually came from?

In most SaaS products, the knowledge path has several distinct sources, each with a different trust profile: internal documentation written and reviewed by employees; customer-submitted documents uploaded through the product; support tickets, which may contain content written by the support team, the customer, or both; content pulled from external websites, if the assistant has any web-retrieval capability; and automated feeds from vendor or partner systems.

Treating all of these as equally trustworthy once they're indexed is the core failure mode this test is designed to catch. A support ticket written by a customer is not the same trust class as an internal playbook written and reviewed by the company's own team, even though both might end up in the same vector index and both might get retrieved and summarized with equal confidence by the assistant. If an unreviewed customer submission — a ticket, an uploaded file, a free-text field — can end up indexed as if it carried the same authority as reviewed internal documentation, the system has effectively let low-trust content masquerade as high-trust content, without any single obvious "bug" being responsible.

This is connected to, but broader than, the model and data poisoning risk described in LLM05:2026 of the current OWASP GenAI LLM Top 10. That entry is framed mostly around training-time and fine-tuning-time data integrity. The concern here is application-level and operates continuously, in production, every time new content is ingested into a retrieval index — a distinction worth being precise about, since the mitigations differ. This is not about someone tampering with model training data; it's about ordinary product usage — uploads, tickets, notes — continuously feeding a knowledge base whose access controls and provenance tracking may not have kept pace with how much the assistant now relies on it.

Knowledge Provenance Chain

This article's seventh framework gives the test a concrete shape:

The Knowledge Provenance Chain is the set of facts a system should ideally be able to establish for any piece of retrieved content that materially influenced an AI-generated answer: where the information came from, who had permission to add it, which tenant owns it, when it was indexed, whether it is still considered authoritative, and whether the requesting user should be able to retrieve it at all.

[Visual concept: A chain-link diagram, each link labeled with one provenance question — Source? Uploader permission? Tenant ownership? Index timestamp? Still authoritative? Requester's retrieval right? — with a broken-link icon shown at whichever point a system commonly fails to track this metadata, illustrating that provenance usually degrades at ingestion rather than at retrieval.]

Testing this boundary without drifting into full data-governance territory — which is a different discipline, and not the focus of a security testing article — comes down to a handful of concrete, answerable questions a team can actually verify against their own system:

Source control at ingestion. Does the ingestion pipeline record where each piece of content came from and who submitted it, or does everything get flattened into an undifferentiated index the moment it's added?

Access control that survives indexing. If a document had restricted visibility in its original system — a support ticket only visible to certain roles, a customer document scoped to a specific account — does that restriction carry through to the retrieval layer, or does indexing implicitly grant broader visibility than the source system ever intended?

Lifecycle handling. When a source document is deleted, corrected, or its access is revoked, does the retrieval index reflect that promptly, or can the assistant continue surfacing deleted or superseded content because the index wasn't refreshed?

Trust-tier awareness at retrieval time. Does the retrieval or ranking logic distinguish between reviewed, high-trust internal content and unreviewed, lower-trust submitted content, or are they interchangeable inputs to the same summarization step?

A useful, low-effort version of this test: pick a fact currently being surfaced by the assistant in a live or staging environment, and manually trace it backward through the provenance chain above. If the team cannot answer more than two or three of those questions for a fact the assistant is actively presenting as reliable, that is a concrete, actionable gap — not a theoretical one — regardless of whether any actual poisoning has occurred.

---

Security Test 8 — What If a Third-Party Model or API Sees Something You Never Intended to Send?

The eighth test turns outward, toward the dependencies a SaaS company doesn't fully control but whose behavior directly shapes what data leaves the company's own infrastructure.

A typical AI-augmented feature depends on a small constellation of third parties: the model provider itself; possibly a separate embedding provider used to generate vectors for retrieval; an observability or tracing vendor used to monitor the AI system's behavior in production; a managed vector database; and any external APIs or connectors the assistant is wired up to call, from a support platform to a calendar system to a document-signing tool.

None of this is inherently a problem — this is ordinary, necessary software supply chain, the same category of dependency every SaaS product has always had. What's different, and what this test is specifically designed to surface, is that AI features tend to route a broader slice of live customer data through these dependencies than a typical integration does, often continuously, and the decision to do so frequently gets made incrementally, integration by integration, without anyone stepping back to look at the aggregate picture of what leaves the building.

This maps to LLM04:2026, Supply Chain, in the current OWASP GenAI LLM Top 10 — an entry the 2026 edition explicitly expanded to include "artifact-trust failure," reflecting how much of a modern AI stack is composed of externally-sourced models, embeddings, plugins, and connectors rather than code the team wrote itself.

External Exposure Inventory

The eighth framework this article introduces is less a conceptual model and more a concrete artifact every team building AI features should actually maintain:

The External Exposure Inventory is a record, kept current, of every external provider that touches AI-related data, covering: what data can leave the company's infrastructure through this integration; why that data needs to leave; under which user or service context the transmission happens; which credentials authorize it; whether the receiving party stores the data, and for how long; and how the system behaves if that provider is unavailable or fails.

[Visual concept: A hub-and-spoke diagram with the SaaS application at the center and each third-party dependency as a labeled spoke, with a small data-flow arrow on each spoke annotated with "what leaves" — deliberately showing some spokes as thick (heavy data flow) and some as thin (minimal), to make the point that not all integrations carry equal exposure.]

This is not a section that names or evaluates any specific vendor's security posture — that would be both inappropriate for this format and legally imprudent, and it isn't the point. The point is that the SaaS team's own responsibility doesn't end at "we chose a reputable model provider." It includes knowing, concretely, what data reaches that provider, under what configuration, and what the fallback behavior looks like if that provider fails, changes its retention policy, or gets swapped for a different model as part of routine engineering iteration — a genuinely common event, since model routing and provider selection tend to change far more often than most other infrastructure decisions in an AI-augmented product.

A useful discipline for keeping this current, rather than a one-time audit that goes stale within a quarter: every time a new external connector, model provider, or data-processing vendor is added to the AI pipeline, adding a row to this inventory becomes a checklist item in the same review process that would already cover a new dependency in any other part of the stack — no different in principle from how a mature engineering team already tracks third-party library and API dependencies for conventional supply-chain risk.

[Internal link opportunity: Quality Engineering]

---

Security Test 9 — Can the AI Reveal Security-Sensitive Internal Information Through Logs, Errors, or Debugging?

The ninth test looks past the chatbot's visible response entirely, toward the layer of the system built specifically to help engineers understand what happened — which is exactly why it deserves scrutiny, not despite that purpose but because of it.

A production AI feature typically generates several parallel streams of recorded information beyond the user-facing response: application logs; AI-specific execution traces, often capturing full prompts and completions for debugging; internal debug dashboards used by engineering and support; product analytics; support tooling that lets an agent see a customer's session history; error messages, which can be more revealing than intended when something fails partway through a multi-step tool-calling sequence; archived prompts, kept for quality review or fine-tuning purposes; captured model requests, kept by observability tooling; and general telemetry.

The security questions for each of these channels are consistent: who can actually access this data, what specifically gets recorded, and does the recorded content include secrets, credentials, or customer PII that the original request never intended to persist anywhere beyond its immediate use?

The Observability Exposure Paradox

This section's framework names a genuine tension that doesn't have a clean resolution, and shouldn't be presented as if it does:

The Observability Exposure Paradox: more telemetry improves a team's ability to diagnose what an AI system actually did, which is essential given how much of its behavior is non-deterministic and hard to reproduce. But the same telemetry, by capturing more of the system's internal state, increases the volume of sensitive context that ends up stored somewhere — and that storage location often has broader internal access, longer retention, and less rigorous access control than the original, live request path ever did.

[Visual concept: A balance scale. One side labeled "Diagnosability — able to understand what the AI actually did" with a growing stack of trace data. The other side labeled "Exposure — sensitive data now sitting in a secondary store" with a matching stack, illustrating that these two quantities rise together rather than trading off cleanly, and that the fix is deliberate design rather than simply choosing a point on the scale.]

The resolution this article recommends is not "log less" — reduced observability is its own serious risk, particularly for a system whose behavior is probabilistic and whose failures can be genuinely difficult to reproduce without a detailed trace. The resolution is deliberate telemetry: treating what gets captured, how it's redacted, who can access it, and how long it's retained as design decisions made with the same intent as any other data-handling choice in the product, rather than as a default the logging library happened to ship with.

Concrete practices that make this testable rather than aspirational:

Redaction before persistence. Sensitive fields — credentials, payment data, personal information — should be stripped or masked before a trace is written to a persistent store, not filtered only at display time in a debugging UI, which leaves the raw data sitting underneath the filter.

Access controls scoped to the telemetry's sensitivity. If a trace store contains the equivalent of full customer conversations and internal system prompts, it deserves access controls comparable to the production database it's effectively a shadow copy of — not the broader, more permissive access that debugging tools often default to for the sake of engineer convenience.

Structured logging over raw capture. Logging specific, named fields relevant to diagnosis — which tool was called, which record was accessed, what the authorization outcome was — produces far better diagnostic value per byte of stored sensitivity than capturing the entire raw prompt and response by default.

Retention limits. Debugging value from a trace decays quickly; the exposure risk from an old, forgotten trace does not. Retention policies for AI-specific telemetry deserve explicit review, separate from whatever default retention the logging platform ships with.

Audit access to the telemetry itself. Given that AI traces can contain some of the most sensitive data in the entire system — full context windows, including whatever was assembled per Test 2 — access to the trace store itself should be logged and auditable, the same way access to production customer data would be.

The test, concretely: sample a representative set of recent AI interaction logs and traces, across a range of feature paths, and check them against a short, specific list — do any contain unredacted credentials, unredacted PII, or internal system details (schema names, internal service URLs, tool definitions) that shouldn't be broadly visible to whoever has access to the logging platform. No detailed secret-extraction methodology is needed to run this test; it is closer to a data-loss-prevention scan than a penetration test, and most teams already have or can adapt tooling for exactly this kind of sampling.

---

Security Test 10 — Can You Prove What the AI Actually Did After an Incident?

The tenth and final test in this list is the one that determines whether everything above is actually verifiable after the fact, or only assumed to be true because nothing has visibly gone wrong yet.

Imagine a customer reports: "The AI assistant changed a record it shouldn't have touched." This is a deliberately plausible scenario, not an alarming one — it's the kind of report any support team handling an AI feature with write access should expect to receive eventually, whether the underlying cause turns out to be a genuine bug, a misunderstanding of the assistant's intended scope, or something more serious.

The question this test asks is whether the engineering team, faced with that report, can actually answer a specific sequence of factual questions: which user initiated the request that led to the change; what identity was actually used to execute it, per the Delegated Authority Gap concern from Test 4; which retrieved context or documents were present at the time, given the Instruction Authority Boundary concern from Test 3; which tool was selected and why, to whatever extent "why" is even recoverable from a probabilistic system; which specific object was targeted; whether the authorization check that should have gated the action actually ran, and what it returned; what the state of the record was before and after the change; what response the user was shown; and which model version, prompt version, and configuration were active at the exact time of the request.

The Security Reconstruction Test

The article's final framework turns this from a hypothetical worry into an actual, runnable exercise:

The Security Reconstruction Test: instead of waiting for a real incident to discover whether reconstruction is possible, deliberately create a controlled, low-stakes event in a staging or sandboxed environment that exercises a state-changing tool call. Then hand the available logs, traces, and audit records to an engineer who did not run the test and did not witness it happen, and ask them to reconstruct the full sequence using only that evidence.

[Visual concept: A timeline reconstruction diagram — a sequence of numbered dots representing "user request," "identity resolved," "context retrieved," "tool selected," "authorization checked," "state changed," "response returned" — with a second engineer's magnifying glass icon positioned below the timeline, trying to reconnect the dots using only recorded evidence, with a gap shown at whichever step commonly has no corresponding audit trail.]

If that engineer can produce an accurate, evidence-backed account of what happened, the team has real proof of auditability — not a belief that logging is "probably good enough," but a demonstrated capability. If they cannot, the gap they hit is exactly the gap that would exist during a real incident, discovered under far better conditions than a live customer escalation or a compliance inquiry.

This distinction is the practical core of the section, and worth stating as directly as the earlier ones: "we have logs" is a different claim from "we can reconstruct the security-relevant sequence." Plenty of systems produce enormous volumes of logs while still being unable to answer a specific question like "which authorization check governed this particular write, and what did it return" — because the logs were designed for general debugging, not for reconstructing a specific security-relevant decision chain.

Making a system pass this test generally requires a small number of structural decisions, most of which are inexpensive if made early and considerably more expensive if retrofitted after the fact: a stable, queryable identifier that ties every step of a request — from initial user action through every tool call and data access — together as a single traceable unit; explicit, recorded outcomes for authorization checks (not just "the action succeeded" but "the check ran, and here is what it evaluated"); versioning for prompts, model configuration, and tool definitions, so that "what was the system actually configured to do at 3:14pm on the day in question" has a real answer instead of "whatever the current version happens to be now"; and before/after state capture for any action with a meaningful write path, independent of whatever the AI's own conversational response claimed had happened.

---

The Attack Surface Is Not the Same as the Model

Having walked through all ten tests, it's worth stepping back and naming a pattern that should be obvious in retrospect but is easy to lose sight of while working through the details: a substantial share of the failure modes described above have nothing specifically to do with the model.

Broken tenant isolation is an authorization bug. Excessive context exposure is a data-minimization and API-design problem. Delegated authority gaps are an identity and access management problem. Improper output handling is an input-validation problem wearing a different hat. Memory leaking across sessions is a state-management and session-lifecycle problem. Third-party exposure is a supply-chain and vendor-management problem. Log exposure is a data-handling and observability-design problem. Every one of these is a category application security teams have been working on for years, well before generative AI entered the picture.

What generative AI genuinely adds — and this is worth being precise about, because overstating it is as unhelpful as ignoring it — is a specific, new set of behaviors layered on top of that familiar foundation: the system ingests untrusted natural-language instructions and cannot always structurally separate them from trusted ones; its decisions are probabilistic rather than deterministic, so the same input can occasionally produce different behavior, and comprehensive test coverage in the traditional sense is harder to achieve; context gets dynamically assembled from multiple live data sources rather than following a fixed, auditable code path; the system's output can itself function as an input to other systems, in ways a simple form submission never could; and in agentic configurations, the system chooses and sequences its own actions, rather than executing a fixed, developer-written sequence.

A useful way to hold both truths at once, without collapsing into either "AI security is just AppSec with extra steps" or "AI security is an entirely new discipline requiring a complete rebuild of security practice":

AppSec, API security, and AI security aren't three separate domains stacked side by side. AI security sits on top of the other two, inheriting every one of their failure modes while adding the layer of risk that comes specifically from probabilistic, instruction-following, tool-using behavior. A team that has strong AppSec and API security practice, and simply extends that discipline to cover the new AI-specific behaviors, is in a fundamentally better starting position than a team treating AI security as an entirely separate, bolted-on program.

---

Functional QA and Security QA Ask Different Questions

The distinction underlying this entire article is easiest to internalize as a direct, side-by-side comparison, applied to the same handful of feature behaviors most AI-augmented SaaS products share.

This table is deliberately built for a founder to read without any AI-security background, because the underlying idea doesn't require one: for every behavior an AI feature exhibits, there is a functional version of "did it work" and a separate, independent security version of "was it allowed to." A test suite covering only the first column can pass completely — every summary correct, every retrieval accurate, every tool call successful, every memory helpful, every response well-rendered — while every question in the second column remains untested and unanswered.

---

The AI Feature Security Proof File

Ten individual tests are useful, but without a structure to hold their results, they tend to degrade into a one-time exercise — something run once before a launch, filed away, and never revisited as the feature evolves. This article's central operational framework is meant to prevent that.

The AI Feature Security Proof File is an operational model for organizing security evidence per AI feature — not a formal security certification, compliance framework, or audit standard, but a structure a team can maintain internally to keep security testing connected to the actual, current state of the architecture.

For each production AI feature, the Proof File organizes evidence under ten areas, mapped directly to the trust boundaries and tests covered above:

[Visual concept: A ten-row checklist or dossier-style layout, each row showing the area name, a small icon, a "last verified" date field, and a status indicator — deliberately styled to look like a living document rather than a static certificate, to reinforce that this is meant to be revisited, not filed away.]

The purpose of organizing evidence this way is narrow and practical: to prevent a security review from collapsing into a single vague claim — "we tested prompt injection once" — standing in for genuine, current assurance across every boundary the feature actually touches. A team that can point to specific, dated, owned evidence for each of the ten areas above has something meaningfully stronger than a team that ran one adversarial red-team session before launch and never revisited the question.

[Internal link opportunity: Quality Engineering]

---

High-Risk vs. Low-Risk AI Features: Consequence Radius

Not every AI feature deserves the same depth of security scrutiny, and treating all of them identically is itself a design mistake — it either underinvests in the features that carry real risk or wastes engineering time gold-plating the ones that don't.

Consider five features a SaaS product might plausibly ship, in rough order of increasing exposure:

Feature A: rewrite marketing text. The user supplies text; the model rewrites it; the result is displayed back to the same user for their own use. No retrieval, no other users' data, no tools, no persistence.

Feature B: answer questions using internal documentation. The model has read access to a knowledge base. Retrieval now matters, but the data is internal documentation, not per-customer sensitive data, and there are no write actions.

Feature C: query customer CRM records. The model can read live, per-tenant customer data. Tenant isolation (Test 1) and context minimization (Test 2) become directly relevant, because real customer data is now in play.

Feature D: change CRM records. The model can write, not just read. Delegated authority (Test 4), output handling as it feeds into a write path (Test 5), and auditability (Test 10) all become materially more important, because a mistake here doesn't just disclose information — it changes it.

Feature E: issue refunds or modify financial state. The model's actions have direct financial consequence and may be difficult or impossible to fully reverse. Every one of the ten tests matters here, and the bar for evidence — not just "we tested it" but "we can reconstruct exactly what happened, every time" — is highest.

Consequence Radius

Consequence Radius describes how far a security failure in a given AI feature could plausibly propagate, considered across several independent dimensions rather than collapsed into a single risk score.

Deliberately, this framework resists being reduced to a single number. A feature can score low on some dimensions and high on others, and averaging that into one figure hides exactly the information a team needs to prioritize correctly.

[Visual concept: A radar/spider chart with six axes — data scope, user scope, tenant scope, action authority, external communication, recoverability — with three overlaid shapes representing Feature A, Feature C, and Feature E, visually showing how the "radius" expands dramatically from the text rewriter to the refund-issuing agent, without reducing any of it to a single composite score.]

The practical use of Consequence Radius is prioritization, not gatekeeping. A team building Feature A genuinely does not need the full Security Proof File — applying it would be disproportionate effort for a feature whose failure mode, worst case, is a poorly-written sentence the user themselves supplied and reviews before using. A team building Feature E needs every one of the ten tests, current, owned, and re-verified on a defined cadence, because the Consequence Radius is wide across nearly every dimension at once.

---

How Security Testing Changes as an AI Feature Gains Capabilities

This has practical value for a specific, common decision point: whether to add "just one more integration" to an existing AI feature. That decision rarely gets evaluated for its security-testing implications, because it's usually framed as a product or engineering-effort question rather than a trust-boundary question.

Stage 1: Text generation only. The model receives text and returns text, with no access to other data sources, no tools, and no persistence. The relevant tests are limited — mainly output handling (Test 5), since even pure text generation eventually feeds into some rendering or downstream context.

Stage 2: Internal document retrieval. The model can search a knowledge base. Retrieval scoping and knowledge provenance (Test 7) now matter, along with basic context minimization (Test 2).

Stage 3: Customer-specific data access. The model can read live, tenant-specific data. Tenant isolation (Test 1) becomes a first-order concern, along with data minimization (Test 2) and, if any web or external retrieval is involved, indirect injection (Test 3).

Stage 4: Tool calls. The model can invoke defined actions against internal or external systems. Delegated authority (Test 4) and output-as-tool-input handling (Test 5) become directly relevant, and the boundary with the OWASP Agentic Top 10 becomes practically important rather than academic.

Stage 5: State-changing actions. The model can write, not just read — creating records, sending communications, modifying financial or account state. Every test in this article applies, auditability (Test 10) becomes non-negotiable rather than optional, and Consequence Radius is typically at its widest.

[Visual concept: A five-stage horizontal progression diagram, each stage shown as a widening funnel, with small icons of the relevant security tests appearing and accumulating at each stage — Test 5 appearing at Stage 1, Tests 1/2/7 joining by Stage 3, Tests 3/4 joining at Stage 4, and the full set of ten present by Stage 5 — visually reinforcing that the required testing depth grows with capability rather than staying fixed.]

The point this progression is meant to make concrete for a founder or engineering leader evaluating a roadmap decision: adding a single new tool to an existing AI feature is rarely "just one more integration" from a security-testing standpoint. It can move the feature from Stage 3 to Stage 4, which brings an entire additional category of tests — delegated authority, output-as-action-input — into scope that wasn't previously relevant. Recognizing that shift at planning time, rather than discovering it after the integration ships, is the difference between security testing that keeps pace with the product and security testing that is perpetually one release behind it.

---

Security Testing Needs Negative Expectations

There's a structural reason AI security testing often lags behind functional testing even on teams that genuinely care about it, and it has to do with what a test is fundamentally designed to check.

A traditional functional test asserts that something happens: an authorized user retrieves their account, and the test confirms the account data appears, correctly formatted, in the response. Security testing for an AI feature frequently needs to assert the opposite: that something does not happen, under conditions where a naive design might let it. An unauthorized user does not retrieve another tenant's account. Retrieved content does not expand the assistant's tool access. A read-only workflow does not trigger a write. An invalid tool request does not execute. A secret does not appear in the user-facing response.

This is a genuinely different testing posture, and it deserves to be named as its own category, because "nothing happened" is an unusual thing for a QA process built around "does the feature work" to treat as a successful, meaningful test outcome.

Forbidden Outcome Assertions

A Forbidden Outcome Assertion is a test that explicitly defines a state that must never occur, and verifies — under conditions specifically designed to make that state plausible — that it does not, in fact, occur.

Concrete examples, kept deliberately defensive and free of anything resembling an attack recipe:

• No record belonging to a tenant other than the currently authenticated one appears in any response, retrieval result, or tool output, under any tested input.
• No privileged, state-changing action occurs without the authorization check that should gate it actually running and returning a positive result.
• No credential, API key, or other secret appears in user-facing output, logs, or traces, under any tested condition.
• No write operation occurs as a side effect of a request the user intended and initiated as read-only.
• No external provider receives a data field outside its approved class, as defined in the External Exposure Inventory.

Writing tests this way requires a deliberate shift in how a team designs its test suite: instead of only asking "what should the correct output be for this input," also asking "what specific bad outcome would we most want to catch here, and how do we construct a test scenario plausible enough to actually exercise the boundary that's supposed to prevent it." This is a genuinely different design discipline from writing conventional functional assertions, and it tends to require security-minded thinking from whoever writes the test — which is exactly why pairing QA/QE expertise with application security expertise, rather than expecting either group to cover this alone, tends to produce a materially stronger test suite than either working in isolation.

[Internal link opportunity: test automation]

---

Red Teaming Is Not the Whole Program

It's worth addressing directly a confusion that shows up often once a team starts taking AI security seriously: the assumption that hiring someone to red-team the model, or running a structured adversarial evaluation, constitutes the security program.

AI red teaming — deliberately probing a model or system with adversarial inputs to see how it responds — is a genuinely valuable practice, and this article is not arguing against it. It's particularly good at surfacing the kind of emergent, hard-to-predict behavior that comes from a probabilistic system: the unexpected way a model handles an ambiguous instruction, the surprising path it takes through a multi-step tool-calling sequence, the specific phrasing that gets past a guardrail a deterministic test never would have thought to try.

What red teaming is not particularly good at, and was never designed to be, is verifying the deterministic controls this article has spent most of its length describing. Whether the CRM API independently enforces tenant-level authorization is not really an adversarial-prompting question — it's a question you answer by making two API calls with different tenant tokens and checking the response, the same way you'd test it if there were no AI layer involved at all. Whether a tool call correctly rejects an out-of-scope request is a server-side authorization test, not a model behavior test. Whether logs are redacted is a data-handling audit, not a red-team exercise.

A security program built entirely around red teaming leaves exactly the deterministic, architectural controls described throughout this article unverified — the ones most likely to produce a clean, unambiguous, and severe failure if they're wrong, precisely because they don't depend on the model's judgment at all.

A genuinely strong program combines several distinct practices, each doing something the others don't:

• Traditional AppSec practices — authorization testing, API security review, secret management audits — applied to every trust boundary in the AI Trust Map, exactly as they would be applied to any other feature.
• Automated security regression tests — the Forbidden Outcome Assertions described above, run continuously in CI, catching architectural regressions the moment they're introduced rather than discovering them in a periodic manual review.
• AI-specific adversarial evaluation — red teaming and structured adversarial testing, targeted specifically at the behaviors that are genuinely probabilistic and model-dependent, where deterministic tests can't reach.
• Manual architecture and code review — a security-literate engineer walking the actual data flow, independent of any automated tooling, asking the "what is trusted here, what shouldn't be" questions this article opened with.
• Controls built into the architecture itself — least privilege, scoped credentials, independent server-side authorization — which reduce how much any of the above testing has to catch in the first place, because the system is structurally harder to get wrong.

None of these substitutes for the others, and a team that has only invested in one — most commonly, red teaming, because it's the most visibly "AI-specific" activity and the easiest to procure as a discrete engagement — should treat that as a partial program, not a complete one.

---

Where Automation Helps

A fair amount of what this article describes can be automated, and doing so is what makes ongoing verification — rather than a one-time pre-launch exercise — actually sustainable for a team shipping AI features on a normal release cadence.

Strong candidates for automation, because they test deterministic properties with clear pass/fail outcomes:

• Authorization matrices run automatically across every role and tenant combination relevant to a feature, re-executed on every relevant code change rather than only before a major release.
• Cross-tenant isolation checks, exercising retrieval, tool calls, and API access under paired test accounts from different tenants.
• Tool permission tests, confirming that every defined tool independently rejects calls from roles that shouldn't be able to invoke it.
• Data-field exposure checks, inspecting assembled model context against an approved field allowlist for a given feature or request type.
• Log and trace scanning, sampling recorded telemetry against redaction rules and flagging unredacted sensitive fields.
• Security regression suites, encoding the Forbidden Outcome Assertions from earlier as a persistent, continuously-run test set rather than a one-time manual exercise.
• Configuration checks, verifying that provider settings, retention configuration, and credential scoping match the documented External Exposure Inventory.

AI-specific adversarial evaluation can also be partially automated — running a defined, evolving set of adversarial scenarios against the system on a regular cadence, rather than relying entirely on a manual, point-in-time engagement. But this deserves a caveat that shouldn't be softened: model behavior carries genuine variability, and a single pass/fail run of an adversarial evaluation suite is a weaker signal than the same result would be for a deterministic test. Repeated runs, evaluated for consistency and trend rather than a single binary outcome, produce a materially more trustworthy signal than treating one clean run as proof the underlying behavior is reliably safe.

[Internal link opportunity: test automation]

---

A Worked Security Review

To make the framework concrete rather than abstract, walk through a single, detailed fictional example from end to end.

The company is a mid-market B2B customer-success SaaS platform. Its AI assistant, embedded in the customer-facing (internally facing, at their customers) success workflow, can: retrieve customer account information; search internal playbooks describing recommended customer-success actions for various account situations; read support tickets associated with a given account; draft a follow-up email for the customer-success manager to review and send; create a task in the CRM to track a follow-up action. It explicitly cannot change billing information — that capability was deliberately excluded from the tool set at design time.

Three roles use the product: Account Executives, who own the commercial relationship with a set of customers; Support Agents, who handle technical issues; and Workspace Admins, who have broader account and configuration access within their own company's workspace.

Drawing the AI Trust Map for this feature

Applying the framework from earlier in this article, the boundaries that matter for this specific architecture are:

Browser → SaaS API. The Account Executive's session token establishes their identity, role, and tenant. Nothing unusual here relative to any other feature in the product — this boundary is well-understood, existing infrastructure.

SaaS API → AI orchestration. The application backend passes the authenticated user's ID, role, and tenant ID to the orchestration layer alongside the request. The security assumption: this identity travels intact and gets independently checked at every subsequent step, not merely referenced once and trusted thereafter.

Orchestration → CRM. The orchestration layer queries the CRM for account data. Given this is a read at this stage, the immediate risk is tenant-scoped data exposure (Test 1) and over-broad field retrieval (Test 2), rather than an authority-escalation concern, since no write is happening yet.

Orchestration → playbook retrieval. The internal playbooks are company-authored, reviewed content — a comparatively high-trust knowledge source. The main test here is less about tenant isolation, since playbooks are shared across the company's own use of the product rather than being customer-specific, and more about the Instruction Authority Boundary (Test 3): does anything in a playbook, however unlikely, function as an instruction that could expand the assistant's behavior beyond what a given user's role should permit.

Orchestration → support ticket retrieval. Tickets are lower-trust than playbooks by construction — they can contain content written by the customer, not just internal staff — and are tenant-specific. Both Test 1 (tenant isolation) and Test 3 (instruction authority) apply directly here, and Test 7 (knowledge provenance) matters if tickets are indexed into the same retrieval system as playbooks without a clear trust-tier distinction.

Orchestration → draft email tool. Drafting an email is a generation task, not an execution task — the draft is presented to the Account Executive for review before anything is sent. Test 5 (output handling) matters here in a specific, narrow sense: does the drafting step ever pull in content, formatting, or embedded references from a retrieved ticket in a way that could make the draft misleading or could carry hidden content the reviewing human wouldn't notice.

Orchestration → create-task tool. This is a write action. Test 4 (delegated authority) is directly relevant: does task creation execute with the Account Executive's actual permission scope, and would it correctly fail if, say, a role without CRM-write access somehow reached this workflow. Test 10 (auditability) matters here specifically because it's the one clear write path in the whole feature — if anything is going to be disputed later, it's most likely to be a task that was or wasn't created correctly.

The explicit non-capability: billing. Because billing changes were deliberately excluded from the tool set, the relevant test isn't "does the assistant handle billing safely" — it's a negative one, a Forbidden Outcome Assertion: no phrasing of a user request, however constructed, should be able to cause a billing-related tool call, because no such tool should be reachable from this orchestration context at all. This is worth testing explicitly rather than assumed, because a future refactor that shares tool definitions across features could inadvertently make a billing tool technically reachable even though product intent never meant it to be.

Selected evidence the team should gather

Given limited review time, prioritizing by Consequence Radius points clearly toward three areas: tenant isolation across the read paths (widest data exposure if wrong), delegated authority on the one write path (task creation), and the negative assertion around billing (low probability, high consequence if ever violated).

For tenant isolation: paired Account Executive accounts across two different customer tenants, each attempting to retrieve the other's account summary, support tickets, and CRM data through the assistant — confirming rejection at the CRM API level specifically, not merely absence from the UI.

For delegated authority on task creation: a Support Agent account, whose role in this hypothetical does not include CRM-write access, attempting — through a plausible, in-scope-sounding request to the assistant — to trigger task creation, confirming the tool-level authorization check rejects it independent of how the request was phrased.

For the billing negative assertion: a direct review of the tool definitions available to the orchestration layer in this specific feature's context, confirming no billing-capable tool is technically reachable, plus a runtime test attempting to elicit billing-related tool use through varied, plausible phrasing, confirming the assistant has no path to attempt it even if it wanted to.

This worked example is deliberately narrower than a full Security Proof File — it's meant to illustrate how the trust map and the ten tests translate into an actual, prioritized review for one real (if fictional) feature, not to serve as a template that covers every case. A production review would extend this to cover memory (Test 6, if the assistant retains any cross-session context about a given account), third-party exposure (Test 8, for whichever model provider and vector database are in use), and observability (Test 9, for whatever logging the orchestration layer produces) — each following the same pattern of tracing the specific architecture rather than applying a generic checklist.

---

A Contrasting Mini-Case: The Low-Surface AI Feature

It's worth closing the practical section with a deliberately simple counter-example, because the framework above can otherwise read as if every AI feature demands this level of scrutiny, and that would be both inaccurate and, for a smaller team, discouraging in a way that undermines the article's actual point.

Consider an AI writing assistant embedded in the same product, whose only job is to rewrite text a user has already written — tightening a paragraph, adjusting tone, fixing grammar. It has no retrieval capability. It has no tools. It maintains no persistent memory between requests; each rewrite is a self-contained request-response pair.

Walking this feature through the same trust map produces a strikingly short list. There is no cross-tenant data exposure risk, because the only data involved is text the requesting user themselves supplied and will see the result of. There is no context-minimization concern, because there's no broader record being fetched — the input is exactly and only what the user wrote. There is essentially no indirect injection surface, because there's no retrieved external content for a hidden instruction to hide inside. There is no delegated authority concern, because there are no tools to call. Memory and knowledge provenance don't apply. Third-party exposure is limited to whatever text the user submitted reaching the model provider — a real consideration, but a narrow, well-bounded one relative to a feature pulling in full customer records. Auditability matters far less, because there's no state-changing action to reconstruct.

The one test that still applies with real weight is output handling (Test 5) — if the rewritten text is later inserted somewhere with elevated authority, such as directly into an outgoing email without further review, the same principle from earlier still holds: model output is untrusted input to whatever receives it next, regardless of how narrow the feature's other risks are.

The point of this contrast is not that this simpler feature deserves no security attention. It's that the depth of security testing should track the actual Consequence Radius of the specific feature, not a fixed checklist applied uniformly regardless of what the feature actually does. A team that spends equal security-review effort on the text rewriter and the customer-data-querying assistant from the worked example above is misallocating effort in both directions — under-scrutinizing the feature that touches real customer data across tenants, and over-scrutinizing the one that fundamentally cannot, by its own architecture, cross most of the boundaries this article is concerned with.

AI security controls, in other words, should follow architecture, not follow hype. A feature's actual capabilities — what it can read, what it can do, what it remembers, what it's connected to — determine how much scrutiny it needs, not the fact that it happens to involve a language model.

---

The 60-Minute AI Security Boundary Review

Given everything above, a natural next question from an engineering leader is practical: how does a team actually start, without committing to a multi-week security program before shipping anything.

This article proposes a specific, time-boxed format — the 60-Minute AI Security Boundary Review — designed to be run for a single feature, with the right people in the room, producing not a certification but a concrete artifact: a filled-in trust map and a prioritized testing backlog.

Participants: a representative from engineering (ideally whoever built the orchestration layer), QA or Quality Engineering, security (or whoever plays that role on a smaller team), product, and the AI feature's owner, if distinct from engineering.

Minutes 0–10: Draw the AI Trust Map. As a group, sketch the actual architecture for this specific feature — not a generic AI architecture diagram, but this feature's real data flow: what it retrieves, what it calls, what it can write, what it remembers. This step alone frequently surfaces disagreement among the people in the room about what the system actually does, which is itself valuable information.

Minutes 10–20: Identify data classes and identity transitions. For each boundary on the map, name what data crosses it and what identity is present. This is where the group should be explicitly asking, for every hop: whose identity is this, and does it survive to the next hop unchanged.

Minutes 20–30: Identify tools and authorities. List every tool the assistant can call, and for each one, name the minimum role or permission that should be required to invoke it, and the maximum authority the AI system's actual execution credential has, independent of what any given user should be allowed.

Minutes 30–40: Identify external providers and persistent state. Walk through the External Exposure Inventory for this feature specifically — which third parties see data, and what — and separately identify every place the feature persists anything: session memory, cached retrieval, long-term personalization.

Minutes 40–50: Define forbidden outcomes. As a group, write down the specific Forbidden Outcome Assertions that matter most for this feature — the handful of things that must never happen, prioritized by Consequence Radius rather than attempting to be exhaustive in one sitting.

Minutes 50–60: Assign security proof tests. For each forbidden outcome and each open question from the trust map, assign an owner and a rough test design — not the test itself, but who is responsible for building it and by when.

[Visual concept: A clock-face or timeline graphic divided into six ten-minute segments, each labeled with its corresponding activity, styled like a meeting agenda card that a team could plausibly print out and use directly.]

It's worth being explicit, the way the brief for this article insists, about what this meeting is and isn't. This meeting produces the map and the testing backlog — not a security certification. Sixty minutes is enough time to identify where the risk actually lives in a specific feature's architecture and to leave with clear ownership of the next steps. It is not enough time to actually run the ten tests, gather the evidence for a Security Proof File, or make any defensible claim that the feature has been secured. Treating the output of this meeting as if it were the security work itself, rather than the plan for the security work, is exactly the kind of shortcut this article is trying to help teams avoid.

---

Ten Questions a CTO Should Be Able to Answer

Pulling the ten tests and the frameworks above into a single executive-level checklist — deliberately not a numeric maturity score, because a single number tends to obscure exactly the kind of dimension-specific detail (per the Consequence Radius discussion) that actually matters for prioritization:

Which data can our AI model actually receive, and is that scope reviewed against a data classification policy rather than left to whatever context-assembly code happened to be convenient to write?

Does every retrieval operation our AI feature performs retain and enforce tenant identity, independently verified rather than assumed from upstream?

Which tools can our AI system call, and do we have a current, accurate list — not a list from when the feature launched, but one that reflects every tool added since?

Do our tools independently enforce authorization at execution time, or do they trust that the AI orchestration layer already checked?

What can retrieved, untrusted content actually influence in our system — does it stay data, or can it, under some condition we haven't tested, become an instruction?

Where in our architecture is AI-generated output treated as untrusted input requiring validation, and where does it currently get trusted by default because it's "the system's own output"?

What persists after a conversation with our AI feature technically ends — and have we actually verified that against the backend, rather than trusting that a "new conversation" button in the UI means what it appears to mean?

What data leaves our infrastructure through our AI stack, to which providers, and do we have a current answer rather than one from whenever the integration was first built?

What security-sensitive information — credentials, PII, internal system details — currently reaches our logs, traces, and observability tooling, and who has access to that store?

If a customer disputed a specific action our AI system took, could we actually reconstruct, with evidence, what happened — or would we be reconstructing it from memory and best guesses?

A CTO who can answer all ten with specific, current, evidence-backed responses is in a genuinely strong position. A CTO who realizes, reading through this list, that several answers are assumptions rather than verified facts has just identified exactly where to point the next sprint of security testing — which is the intended, practical use of the list.

---

What a Security Pass Actually Means

There's a phrase worth challenging directly, because it gets used casually and creates a false sense of permanence: "we security-tested the AI." Stated that way, it implies a completed, durable state — something checked off, filed, and no longer in question.

Security, for an AI feature or any other, isn't a state a system reaches and then holds. It's a claim that's true relative to a specific, described architecture at a specific point in time — and that architecture changes constantly, often faster for AI features than for most other parts of a SaaS product, because model versions, tool sets, and connectors tend to iterate quickly.

Specific changes that should be treated as automatically invalidating prior security evidence, not requiring a fresh judgment call each time about whether they "count":

• a new model version or model provider;
• a new tool added to the assistant's available set;
• a new connector or third-party integration;
• a new data source added to retrieval;
• any authorization or role-permission change in the underlying application;
• any addition to what the system remembers or persists;
• a new external provider anywhere in the pipeline;
• a new workflow the assistant is wired into;
• any change to the orchestration logic itself.

Security Evidence Expiry

Security Evidence Expiry describes the principle that a piece of security evidence is only as valid as the architectural assumption it was gathered against — and that when the underlying architecture changes, the evidence doesn't get worse, it simply stops being evidence for the new system, even if nobody has explicitly invalidated it.

[Visual concept: A dated stamp or expiry-label graphic, showing a "Security Evidence — Verified [date]" stamp with an architecture diagram underneath, and a second, slightly modified architecture diagram beside it — one new tool added — with the same stamp shown crossed out, illustrating that the change, not the passage of time, is what invalidates the evidence.]

A concrete illustration, directly from earlier in this article: a team runs a thorough security review of their AI assistant when it can only read CRM data — Test 1 and Test 2 pass cleanly, tenant isolation holds, context is appropriately scoped. The team reasonably considers this feature reviewed and moves on. Two months later, a new release adds an update_customer tool, unlocking a write path that didn't exist during the original review. The prior test results say nothing meaningful about whether Test 4 — delegated authority — now holds for this new capability, because that test was never run against an architecture that included a write path at all. The earlier "pass" wasn't wrong; it simply expired the moment the architecture it described stopped existing.

This is why Security Evidence Expiry connects directly and necessarily to ordinary product change management. The trigger for re-running relevant security tests shouldn't be a calendar reminder or a periodic audit cycle alone — it should be built into the same process that already tracks feature changes, so that "we added a tool" and "we need to re-verify delegated authority for this feature" become linked, automatic consequences of each other rather than two separate processes that have to be manually kept in sync by someone remembering to connect them.

---

Your AI Feature Needs to Prove What It Cannot Do

Before closing, it's worth restating plainly what this article has deliberately not claimed, because balance matters as much here as anywhere else in the piece. This is not an argument that AI applications are inherently insecure, that large language models cannot be secured, or that every AI feature demands an enormous, dedicated security program regardless of what it actually does. A text rewriter with no retrieval, no tools, and no memory is not the same security problem as an autonomous account-management agent with write access to financial state, and treating them identically wastes effort in both directions. Security depth should follow architecture and consequence — that has been the organizing idea throughout, made concrete through the Consequence Radius and the progressive-capability framework above.

Return to where this article began. A B2B SaaS company shipped an AI assistant. An account manager asked it to summarize an account and flag unresolved issues. Functional QA confirmed, thoroughly and correctly, that the assistant could perform exactly that workflow — retrieve the right data, produce a useful summary, render it cleanly, handle errors gracefully. That work was real, necessary, and well done.

Security testing has to prove something categorically different, and it's worth listing the claims explicitly, one more time, because together they define the actual scope of the work: that the assistant cannot retrieve customers or accounts outside the requesting user's authorized tenant, regardless of how the request is phrased or which internal component constructs the underlying query. That it cannot be maneuvered, through retrieved content it was never designed to treat as instructions, into expanding its own authority beyond what was explicitly granted. That it does not, as a matter of routine operation, expose more sensitive data to its own context — and by extension to logs, traces, and third-party providers — than the task genuinely requires. That it cannot execute an action the initiating user's actual role does not permit, independent of what the model itself judged appropriate. And that if any of the above nonetheless goes wrong, the team has enough recorded, structured evidence to reconstruct exactly what happened, rather than reconstructing it from memory, screenshots, and guesswork.

None of that is visible in a passing functional test suite. All of it is testable, with the right trust map, the right questions at each boundary, and evidence that's treated as something to maintain rather than something to file away.

The defining security question for an AI feature is not only whether it can do the job. It is whether the system can prove where that capability stops.

---

AI security testing is most useful when it's connected directly to the actual application architecture behind a feature — the identities, data flows, APIs, retrieval paths, tools, outputs, and third-party dependencies that make up the real system, rather than a generic checklist applied from the outside. QAtronic works with SaaS engineering teams to build this kind of AI-focused security testing alongside functional QA, API testing, test automation, and the broader Quality Engineering practice that already supports the rest of the product.

---

A note on evidence. AI security guidance is moving quickly, and the framework names and product-specific behaviors discussed in this article should be checked against current official sources before being treated as settled. The AI Trust Map, Identity Continuity, Context Least Privilege, Instruction Authority Boundary, Delegated Authority Gap, Output Authority Escalation, Context Residue, Knowledge Provenance Chain, External Exposure Inventory, Observability Exposure Paradox, Security Reconstruction Test, AI Feature Security Proof File, Consequence Radius, Forbidden Outcome Assertions, and Security Evidence Expiry are all analytical frameworks introduced in this article for the purpose of organizing security testing thinking — none of them are OWASP or NIST standards, and none should be cited as such. The B2B SaaS scenarios used throughout, including the customer-success platform in the worked review and the text-rewriting mini-case, are hypothetical illustrations built to demonstrate the framework, not descriptions of any real product or incident.

---

Sources and Further Reading

• OWASP GenAI LLM Top 10 2026 — OWASP GenAI Security Project
• OWASP Top 10 for Agentic Applications 2026 — OWASP GenAI Security Project, Agentic Security Initiative
• OWASP API Security Top 10 (2023) — OWASP Foundation
• OWASP API1:2023 Broken Object Level Authorization — OWASP Foundation
• NIST AI Risk Management Framework (AI RMF 1.0) — National Institute of Standards and Technology
• NIST AI 600-1, Generative Artificial Intelligence Profile — National Institute of Standards and Technology
• OWASP GenAI Security Project — Initiatives Overview — OWASP GenAI Security Project
• State of Agentic AI Security and Governance 2.01 — OWASP GenAI Security Project
• MITRE ATLAS (Adversarial Threat Landscape for Artificial-Intelligence Systems) — MITRE Corporation
• Cloud Security Alliance AI Controls Matrix (AICM) — Cloud Security Alliance