The Empty Prompt

Can we offer this customer a larger credit limit?

Here is a reasonable first attempt at getting an AI system to answer it.

SYSTEM:
You are a financial operations assistant.
Provide accurate, policy-compliant answers.

USER:
Can we offer this customer a larger credit limit?

The instruction is not badly written. It establishes a role, sets an expectation of accuracy, and references policy compliance. A prompt engineer could improve it further: add a required output schema, add few-shot examples of well-formed credit assessments, add explicit reasoning steps, add a refusal clause for insufficient information. All of that would help at the margins.

None of it addresses the actual problem, which is that the model has no idea who "this customer" is.

It does not know the account identifier. It does not know the current balance, the utilization ratio, the payment history, the number of delinquencies in the last twenty-four months, or whether a payment cleared this morning. It does not know which credit policy version is currently in force, whether that policy was superseded three weeks ago, or whether the customer sits in a jurisdiction where an automated limit increase requires an affordability check before it can be offered. It does not know whether the person asking is a support agent with read-only access to masked account data or a risk analyst authorized to see the full exposure position. It does not know whether this same question was already reviewed and declined last quarter, and if so, on what grounds.

The model can produce fluent text about credit limits in general. It cannot produce a decision about this customer, because nothing in the request describes this customer. The system will either refuse, ask for clarification, or generate something that reads like an answer and is not one.

This is the distinction worth holding onto for the rest of this article. Prompt engineering controls instructions: what the system should do, how it should behave, what shape the output should take, what tone and constraints apply. Context engineering controls the information available for reasoning: which facts, documents, records, tool results, and prior state actually reach the model at inference time, in what form, with what priority, and with what indication of trustworthiness.

Production quality depends on both. A perfectly assembled context handed to a model with vague or contradictory instructions produces inconsistent output. A beautifully written system prompt handed to a model with no access to the relevant facts produces confident emptiness. The two disciplines are not in competition, and the growth of one does not imply the decline of the other. What has changed is where the marginal effort now pays off. For most teams that have already written a decent system prompt, the next order-of-magnitude improvement is not in the wording of the instruction. It is in the construction of the information environment around it.

The credit limit question will reappear throughout this article. It is used here as a technical example, not a story. It is useful precisely because it is unremarkable: it is the kind of question enterprises ask thousands of times a day, it depends on a mixture of documents and database records and live service calls, it is governed by policy and regulation, it is subject to access control, and it has a correct answer that can be wrong in expensive ways.

Model Knowledge and Runtime Knowledge

There is a structural fact about language models that is easy to state and easy to forget in practice: a model knows two categories of things, and only one of them comes from the model.

Model knowledge is what was absorbed during training. It includes language, general reasoning patterns, public facts, common code idioms, widely documented standards, and an approximate picture of how the world worked up to some cutoff. It is broad, cheap to access, always present, and impossible to update without retraining or fine-tuning. It is also generic by construction: every organization using the same model has the same model knowledge.

Runtime knowledge is everything the application supplies at inference time. It includes the user's identity, the current state of a database row, the text of an internal policy, the output of a risk scoring service, a summary of the last three conversations with this customer, the list of tools the model is allowed to call, and the timestamp attached to each of those facts. It is narrow, expensive to assemble, present only if the application put it there, and updatable at the speed of the underlying systems.

Generic writing tasks lean almost entirely on model knowledge. Summarize this paragraph, rewrite this in a more formal register, explain what a B-tree is. The model already contains everything required. This is why the first generation of generative AI applications felt so productive so quickly, and why prompt technique appeared to be the dominant variable. When all the information is already inside the model, the only lever available is how you ask.

Enterprise reasoning inverts the ratio. Decision support, operations, workflow automation, and agentic systems depend overwhelmingly on facts that exist only inside the organization's own systems and that changed sometime after the model finished training. The model contributes reasoning, language, and structure. The organization contributes the facts. A credit decision is roughly ninety percent runtime knowledge and ten percent model knowledge, and the ten percent is the easy part.

This has an uncomfortable implication for how teams evaluate progress. Benchmarks that measure model capability measure the shared component, the part every competitor also has. They say very little about whether a specific enterprise system will answer a specific enterprise question correctly, because that depends on plumbing the benchmark never touches. Two organizations running the identical model on the identical question will get materially different answers if one of them retrieved the current policy and the other retrieved a draft from 2023.

The practical consequence is that the model becomes one component inside a larger information system rather than the system itself. Its job is to reason over a supplied working set. Someone still has to decide what belongs in that working set, and that someone is the application.

Eleven Kinds of Context, Not One

A common misreading of context engineering treats it as a capacity problem. The model has a two hundred thousand token window, or a million, so put more in. This confuses the container with the contents.

Context is not one substance measured in tokens. It is a set of distinct information types with different origins, lifetimes, trust levels, update frequencies, and security boundaries. Treating them as interchangeable text is the source of a surprising number of production failures.

Instruction context describes what the system should do. It includes the system prompt, developer-supplied behavioral rules, output schemas, and refusal conditions. It is authored deliberately, changes on a deployment cycle, and carries the highest trust level in the system because it originates from the people operating the application.

User context describes who is asking. Identity, role, department, entitlements, locale, seniority, and organizational position. For the credit question, this determines whether the answer should be a recommendation, an approval, or a refusal to proceed. It is supplied by the authentication layer, not by the user's typed text, and this distinction matters enormously: a user can claim to be a risk manager in a chat message, and the system must not believe it.

Conversation context is the current dialogue. Previous turns, clarifications, corrections, and the accumulated implicit state of what "this customer" refers to. It has a session lifetime, moderate trust, and grows monotonically unless actively managed.

Retrieved context is external knowledge pulled from indexed sources in response to the current request. Policy documents, procedure guides, past case notes, product terms. Its trust level varies by source and its relevance is probabilistic rather than guaranteed.

Structured business context is what the databases and APIs currently say. Balances, statuses, dates, classifications, counts. It is precise, typed, authoritative for the facts it holds, and completely absent unless the system explicitly queried for it.

Tool context is the set of actions available and the observations they have already produced. Tool descriptions, parameter schemas, prior call results, and error messages. It shapes not what the model knows but what it can do next.

Operational context describes the state of the system itself. Which services are degraded, which index was last rebuilt, which feature flags are active, what the current rate limit budget looks like. It rarely appears in the model context directly but often should influence what the system attempts.

Historical context covers prior decisions and their outcomes. Was this customer reviewed before? What was decided? Did the decision hold? This is distinct from conversation history because it spans sessions, users, and often years.

Memory is what the system has chosen to persist about a user, an entity, or a task, extracted and consolidated from prior interactions rather than stored verbatim. It has the longest lifetime and, as discussed later, the greatest capacity for silent corruption.

Authorization context determines what this user or agent is permitted to know and do. It is not a filter applied at the end. It is a constraint that should shape retrieval itself.

Temporal context establishes what is current. Effective dates, versions, supersession chains, event timestamps, index freshness, and cache age. Without it, the system cannot distinguish a policy that applies from a policy that used to apply.

Each of these has a different failure signature. Stale structured context produces confidently wrong numbers. Missing authorization context produces data leakage. Corrupted memory produces errors that recur across sessions and resist debugging. Bloated conversation context produces attention dilution and rising cost with no accuracy benefit. They do not fail in the same way, they cannot be fixed by the same intervention, and they should not be managed by the same code path.

Once you accept that context is heterogeneous, the engineering problem becomes visible. Someone has to decide which types are required for a given request, fetch each from a different subsystem with different latency and failure characteristics, reconcile them when they disagree, fit them into a shared budget, and present them to the model in a form that preserves their distinctions.

The Context Pipeline

Below is the shape that most mature enterprise AI systems converge on, whether or not they name it this way. Not every request traverses every stage, and several stages run in parallel in practice, but the logical ordering holds.

REQUEST
   ↓
Intent Analysis
   ↓
Identity & Authorization
   ↓
Context Requirements
   ↓
Source Selection
   ↓
Retrieval ──┬── Structured Queries
            ├── Memory Retrieval
            └── Tool State
   ↓
Filtering
   ↓
Ranking
   ↓
Deduplication
   ↓
Compression
   ↓
Context Assembly
   ↓
LLM
   ↓
Validation
   ↓
Response / Action
   ↓
Feedback
   ↓
Memory & Knowledge Update

Intent analysis classifies what kind of question this is. The credit limit question is a decision-support request about a specific customer entity, governed by policy, requiring current financial data. A question about how to reset a password is a procedural lookup requiring no customer-specific data at all. These two requests should not follow the same retrieval path, and determining that early prevents enormous waste. Intent analysis can be a small classifier, a rules table, a cheap model call, or a combination. It also performs entity extraction: which customer, which account, which product, which region. Without resolved entities, every subsequent stage is guessing.

Identity and authorization resolves who is asking and what they may see. Critically, this happens before retrieval rather than after. The set of documents and records the user is entitled to access becomes a constraint on the search space, not a filter applied to search results.

Context requirements translates intent into a specification. For a credit decision the requirement might be: current policy version, customer master record, twelve months of payment behavior, current exposure across all products, regional regulatory constraints, prior decisions on this account within twenty-four months, and a live risk score. This specification is a contract that later stages must satisfy or explicitly report as unmet. It is also the artifact that makes evaluation possible, because it defines what "sufficient context" means for this request type.

Source selection maps requirements to systems. Policy comes from the approved document repository, not from the general wiki index. Exposure comes from the core banking API, not from the nightly data warehouse extract, because the warehouse is up to twenty-four hours stale and exposure moves intraday. This mapping is a design decision with real consequences, and it is one of the highest-leverage places to encode organizational knowledge into the system.

Retrieval, structured queries, memory retrieval, and tool state execute in parallel where possible. Each has different latency, different failure modes, and different degradation behavior. A well-built pipeline knows which of these are mandatory and which are optional, so that a slow enrichment service does not block a request that could proceed without it.

Filtering removes results that fail hard constraints: wrong tenant, wrong entity, expired document version, insufficient permission, below a relevance floor. Filtering is boolean and should be conservative. Anything that survives filtering is a legitimate candidate.

Ranking orders the surviving candidates by expected usefulness. This is where reranking models, recency weighting, and source authority signals apply. Ranking is where the difference between a mediocre and a strong retrieval system usually lives.

Deduplication collapses near-identical content. Enterprise corpora are full of the same policy paragraph appearing in a PDF, an intranet page, a training deck, and three team-specific copies. Sending the model four versions of one fact wastes budget and, worse, creates a false impression of corroboration.

Compression reduces the selected material to fit the budget while preserving decision-relevant detail. This is the stage most likely to silently destroy correctness, and it deserves its own treatment later.

Context assembly arranges the surviving material into a structured package with labels, boundaries, provenance, and priority ordering. This is not string concatenation. The structure itself influences model behavior.

Validation checks the model output before it becomes an action. Schema conformance, citation verification against the supplied context, policy checks, numeric plausibility, and abstention detection. A credit recommendation that cites a policy clause not present in the assembled context should not reach a human, let alone an automated workflow.

Feedback and memory update close the loop. Was the recommendation accepted? Was it overturned? Did the user correct a fact? These signals feed retrieval tuning, memory writes, and knowledge base corrections.

Two things are worth emphasizing about this pipeline. First, it makes explicit that context engineering governs both what information enters and how it is transformed on the way in. Most teams focus on the first and neglect the second, then wonder why adding a better vector index did not improve accuracy. Second, every stage is a place where quality can be measured. A pipeline you cannot inspect stage by stage is a pipeline you cannot debug.

Context Has a Budget

Context is never free, even when the window is large enough to hold it.

The obvious cost is tokens. Input tokens are billed, and at enterprise request volumes the difference between a four thousand token context and a forty thousand token context is a line item that finance will eventually ask about. Caching changes the arithmetic but does not eliminate it.

The second cost is latency. Assembling a large context requires more retrieval, more reranking, more database round trips, and more tokens for the model to process before the first output token appears. Users notice. Interactive systems have a latency budget in the same way they have a token budget, and the two are often in tension.

The third cost is the one teams underestimate: additional context can reduce answer quality. Several mechanisms contribute. Attention dilution means that relevant signal competes with a larger volume of plausible-looking noise. Conflicting information forces the model to arbitrate between sources without necessarily having the metadata to do so correctly. Duplicated information creates an illusion of consensus that can override a single authoritative source stating the opposite. Outdated material that looks structurally identical to current material is indistinguishable to a model unless the system labels it. Retrieval noise, meaning documents that scored well on similarity but are topically irrelevant, occupies budget and invites tangents.

The empirical shape of this is familiar to anyone who has instrumented a retrieval system: accuracy rises steeply as you go from three retrieved chunks to eight, flattens somewhere in the low teens, and then declines. The exact numbers depend on the corpus, the chunking strategy, the reranker, and the model, which is precisely why this needs to be measured per system rather than assumed.

So the real problem is a selection problem. Given a corpus with hundreds of thousands of chunks, plus a dozen structured systems that could each return arbitrarily many rows, plus a memory store, plus tool outputs, which twenty to fifty pieces of information should reach the model for this particular request?

That is an information architecture question, not a capacity question. It requires a notion of relevance to the specific intent, a notion of sufficiency relative to the requirement specification, and a willingness to exclude material that is genuinely related but not decision-relevant.

Deliberate exclusion is the part that feels wrong to engineers trained on the idea that more data is safer. For the credit question, the customer's full three-year transaction history is related. It is also thirty thousand tokens of noise that will bury the four aggregate metrics the policy actually references. The correct move is to compute the aggregates in the data layer and send those, discarding the raw rows entirely. The model does not need to see the evidence for a derived fact if the derivation is deterministic and auditable elsewhere.

The framing that helps here is to treat the context as a brief prepared for a competent analyst who has thirty seconds to read it. You would not hand that analyst the entire filing cabinet. You would hand them the current policy section that applies, the four numbers that policy references, the two exceptions on file, and a note about which of those numbers was refreshed this morning versus last night. Completeness is not the objective. Sufficiency for the decision, with an explicit signal about what was omitted, is.

The Context Window as a Working Set

There is a useful comparison to operating systems here, and it is worth using carefully rather than stretching it.

The context window behaves less like storage and more like a working set: the portion of an address space actively required for the current computation. It is small relative to the total information available, it is populated on demand, it is discarded when the operation completes, and the cost of a poorly chosen working set is not that the program fails but that it performs badly.

The comparison is useful for a few specific reasons. It reframes the question from "how much can we fit" to "what does this computation need right now," which is the correct question. It makes clear that population strategy matters more than capacity, which is the central point. And it explains why a system with a small context window and excellent selection can outperform one with a huge window and indiscriminate loading.

The comparison should not be pushed further than that. There is no hardware paging mechanism, no fault handler, no locality of reference in the memory sense, and no eviction policy operating below the application. Every one of those is something the application must implement itself. So the analogy describes a shape, not a mechanism.

Within that shape, it helps to separate several categories of material by lifetime. Active information is what the current turn requires and nothing else. Retrieved information was fetched for this turn and will not survive it unless deliberately promoted. Temporary reasoning artifacts are intermediate outputs the model produced while working, such as a plan, a partial calculation, or a tool call trace. These have a strong tendency to accumulate in agentic systems and are usually the largest source of context bloat. Persistent knowledge lives in indexes and databases and is fetched when needed. Long-term memory is deliberately extracted and stored, spans sessions, and requires its own governance.

The failure mode this framing guards against is the one that appeared as soon as long-context models arrived: loading an entire knowledge base into the window and treating retrieval as an obsolete workaround for small windows. This is architecturally weak for reasons that have nothing to do with capacity.

It is expensive per request, and the expense recurs on every request rather than being amortized into an index. It is slow, both in time to first token and in end-to-end processing. It provides no mechanism for access control, because once material is in the window it is available to the generation regardless of who is asking. It provides no provenance, because a flat dump loses the metadata that distinguishes an approved policy from a superseded draft. It provides no freshness guarantee unless the entire dump is reconstructed on every call. And it degrades accuracy on precisely the tasks where precision matters most, because the signal-to-noise ratio in a whole-corpus dump is terrible.

Larger windows are genuinely valuable. They allow longer agent trajectories before summarization becomes mandatory, they permit whole-document reasoning where document structure matters, they reduce the pressure on aggressive compression, and they make certain multi-document comparison tasks tractable that previously were not. They expand the space of workable designs.

What they do not do is remove the need for retrieval, ranking, memory design, permission filtering, or context selection. Those exist because information must be chosen, attributed, authorized, and kept current, and none of those requirements is a function of window size.

Retrieval Is Context Construction

Introductory treatments of retrieval-augmented generation describe a two-step process: embed the query, fetch the nearest neighbors, prepend them to the prompt. Systems built that way work acceptably on clean, homogeneous corpora and degrade sharply on real enterprise content.

A production retrieval path has considerably more structure:

Query Understanding
   → Query Transformation
      → Candidate Generation (dense + sparse + structured)
         → Filtering (permissions, tenant, entity, version)
            → Reranking
               → Context Construction

Query understanding extracts what the user is actually asking for. "Can we offer this customer a larger credit limit?" contains an implicit entity reference, an implicit action, and an implicit policy dependency. The literal string is a poor search query for any of the three. It contains none of the terms that appear in the credit policy document, which will use language like "credit line adjustment," "exposure threshold," and "affordability assessment." Embedding the raw question and hoping for semantic overlap is leaving substantial recall on the table.

Query transformation addresses this. Techniques include rewriting the query into the vocabulary of the target corpus, decomposing a compound question into independent sub-queries, generating multiple query variants to broaden coverage, and generating a hypothetical answer document to embed instead of the question, on the theory that answers resemble answers more than questions resemble answers. Each of these adds latency and cost, and each has corpora where it helps and corpora where it does not. Multi-query retrieval in particular can triple retrieval cost for a modest recall gain, which is a reasonable trade for a high-stakes decision and a poor one for a chat assistant.

Candidate generation should usually be hybrid. Dense vector search captures semantic similarity and handles paraphrase well, but it is weak on exact identifiers, product codes, rare technical terms, acronyms, and version numbers. Sparse lexical search using BM25 or similar handles precisely those cases and fails on paraphrase. Enterprise corpora are full of both: policy documents written in formal prose and populated with clause identifiers, product codes, and regulatory references. Running both and fusing the result lists, commonly with reciprocal rank fusion, is close to a default. Metadata filtering runs alongside, constraining candidates by document type, effective date, owning department, jurisdiction, and access classification.

Reranking applies a cross-encoder or similar model to score each candidate against the query jointly rather than through independent embeddings. This is computationally heavier per candidate, which is why it runs on the top fifty or hundred rather than the whole corpus, and it typically produces the largest single quality improvement available in a retrieval stack. It is also where non-semantic signals get folded in: recency, source authority, document status, and prior usefulness for similar queries.

Context construction is the step most often skipped. Retrieved chunks are fragments torn out of documents. A chunk containing "the threshold in section 4.2 does not apply" is useless without knowing what section 4.2 says, which document this is, whether it is in force, and what it governs. Construction means restoring enough surrounding structure and metadata that each fragment is interpretable: document title, section path, effective date, version, owner, and where relevant the parent section text.

Several architectural patterns support this. Parent-child retrieval indexes small chunks for precise matching but returns the larger parent section for context, decoupling match granularity from delivery granularity. Hierarchical retrieval searches document-level summaries first to identify relevant documents, then searches within them, which improves precision on large corpora with many topically similar documents. Contextual chunk augmentation prepends a short generated description of where each chunk sits within its document before embedding, which measurably improves retrieval on documents where individual chunks are ambiguous in isolation.

The point worth taking from all of this is that retrieval quality cannot be assessed by whether the final answer sounds correct. A model can produce a correct-sounding answer from bad evidence, from no evidence, or from its own parametric knowledge while ignoring the evidence entirely. It can also produce a wrong answer from perfect evidence. End-to-end answer quality conflates two independent systems and tells you almost nothing about which one to fix.

Retrieval therefore needs its own evaluation, using measures borrowed from information retrieval. Recall@K asks whether the necessary evidence appears anywhere in the top K results, which is the ceiling on what the model can possibly do. If recall@10 is 0.6, then forty percent of requests are unanswerable regardless of model quality, and no amount of prompt refinement will fix them. Precision@K asks what fraction of the returned results are actually relevant, which bounds how much budget is being wasted and how much noise the model must ignore. Mean reciprocal rank captures how high the first relevant result appears, which matters because position within the context influences attention. NDCG extends this to graded relevance, distinguishing the authoritative current policy from a related but secondary document.

These measures require labeled data: for a set of representative questions, which documents or chunks constitute the necessary evidence. Building that labeled set is unglamorous work that teams routinely postpone, and it is the single highest-value investment available for a retrieval-dependent system. Without it, retrieval tuning is guesswork, and every change is evaluated by whether a handful of hand-checked answers look better.

Chunking Is Information Architecture

Chunking is usually treated as a preprocessing parameter. It is closer to a schema design decision, and getting it wrong imposes a ceiling on retrieval quality that no reranker can lift.

The default approach splits documents into fixed windows of some token count with some overlap. It is simple, uniform, and predictable. It is also indifferent to meaning, and on structured enterprise documents it destroys a great deal.

Consider what a fixed 512-token window does to real content. It splits a table so that the header row lands in one chunk and half the data rows in another, leaving both uninterpretable. It separates a defined term from its definition, so a chunk referencing "Eligible Balance" retrieves without the definition that gives it meaning. It cuts a numbered procedure between step four and step five, so a retrieved chunk describes a partial process with no indication that steps are missing. It severs a conditional from its condition, producing chunks that state rules without their qualifying clauses, which in a policy context is actively dangerous. It detaches headings from the content they govern, removing the strongest available signal about what a passage is about.

The alternatives form a rough spectrum of increasing structural awareness.

Sentence-aware chunking respects sentence boundaries, which prevents the worst mid-clause truncations at essentially no cost. This should be considered a floor rather than a strategy.

Paragraph-aware chunking groups complete paragraphs up to a size limit. It works well on prose-heavy content where the paragraph is the natural unit of argument.

Section-aware chunking uses the document's own structural markup, whether headings, XML elements, or a parsed outline, to align chunks with authored sections. This preserves the author's own judgment about what belongs together, which is usually better than any inference the pipeline could make.

Semantic chunking detects topic shifts by measuring embedding distance between consecutive sentences and splitting where the distance spikes. It is appealing in principle and inconsistent in practice: it works well on flowing narrative content and poorly on terse structured content where consecutive sentences are legitimately dissimilar without a topic change.

Hierarchical chunking builds multiple levels simultaneously: document summary, section summary, and leaf chunks, with parent links preserved. Retrieval can then match at any level and deliver at the appropriate one. It costs more to build and index and gives the most flexibility at query time.

The strategy should follow the knowledge type, and the differences are substantial.

Contracts and policies are hierarchical, heavily cross-referential, and semantically dependent on definitional sections that may be a hundred pages from the clause that uses them. Chunk by clause, preserve the full section path in metadata, and either inline resolved definitions or ensure the definitions section is retrievable alongside any clause that depends on it. Version and effective-date metadata is mandatory rather than optional, because a chunk from a superseded contract is indistinguishable in text from the current one.

Technical documentation has strong heading hierarchy and mixes prose with code blocks and parameter tables. Split on headings, keep code blocks intact regardless of length, and preserve the heading path so that a retrieved chunk about a configuration option carries the product, version, and component it belongs to.

Support tickets are conversational threads where the resolution is usually at the end and the problem statement at the beginning, with substantial noise in between. Chunking the raw thread is generally a mistake. A better approach is structured extraction into problem, environment, diagnosis, and resolution fields, indexing the extracted structure rather than the transcript.

Source code has syntactic boundaries that mean something. Chunk by function, class, or module using a parser rather than a token counter, and attach the file path, imports, and enclosing type as metadata. A function body retrieved without its signature and imports is much less useful than one retrieved with them.

Product catalogs are records, not documents. They should generally not be chunked at all. Serialize each product into a compact structured representation, index that as a unit, and let structured filters handle attribute constraints that vector similarity handles badly.

The general principle is that chunk boundaries should coincide with the boundaries of a complete, independently interpretable idea in the source material. Where the source has explicit structure, use it. Where it does not, invest in extracting some. The cost of chunking well is paid once during ingestion. The cost of chunking badly is paid on every query, forever.

When the Answer Lives in a Table, Not a Document

There is a bias built into the standard retrieval stack: it assumes enterprise knowledge is documents. Vector databases store text. Chunking operates on text. Reranking scores text against text. Everything in the tooling points toward a corpus of unstructured content.

Most of what an enterprise actually knows is not in documents. It is in relational databases, CRM records, ERP modules, ticketing systems, analytics warehouses, event streams, service APIs, configuration management systems, and identity directories. These systems hold the current state of the business, and they hold it in a form that is precise, typed, transactional, and queryable with exact semantics.

The credit limit question makes the split obvious. Documents explain the rules: what the credit policy requires, which thresholds apply to which customer segments, what documentation an affordability assessment demands, which regional variations exist. Structured systems supply the facts the rules operate on: current balance, utilization ratio, account age in months, count of late payments in the trailing twelve, aggregate exposure across all products, internal risk grade, and current account status.

Neither half answers the question alone. The policy without the facts is a general description of credit criteria. The facts without the policy are numbers with no interpretive frame. A retrieval architecture that only handles one of them produces systems that can quote the rules but not apply them, or apply arithmetic with no governing authority.

Combining them requires the pipeline to run structured queries alongside document retrieval, which introduces a set of problems document retrieval does not have.

The first is entity resolution. A structured query needs an identifier. "This customer" must resolve to a customer ID before any query can execute. In conversation, entity references are ambiguous, elliptical, and often carried implicitly from earlier turns. Resolving them reliably requires explicit state tracking rather than hoping the model infers correctly, and it requires a disambiguation path when resolution is uncertain. Answering a credit question about the wrong customer is worse than refusing to answer.

The second is query generation. Natural language to SQL is the obvious approach and deserves more caution than it usually receives. It works well on small, clean, well-documented schemas with clear naming and few joins. It degrades quickly on real enterprise schemas with hundreds of tables, cryptic column names, denormalized history tables, soft deletes, effective-dated rows, and business logic encoded in conventions no schema comment records. A generated query that is syntactically valid and semantically wrong returns a number that looks authoritative and is not, which is the worst possible failure mode because nothing downstream can detect it.

Where generated queries are used, several controls are close to mandatory:

• Schema awareness with semantics, not just DDL. The model needs to know that acct_stat_cd = 'A' means active, that bal_amt is in minor currency units, and that the history table contains one row per statement cycle rather than per transaction. This is a curated semantic layer, and building it is real work.
• Permission propagation. The query must execute under the requesting user's authorization, not a service account with broad access. Row-level and column-level security must apply at the database, not in a post-processing filter.
• Query validation before execution. Parse the generated SQL, verify it is read-only, verify every referenced table is on an allowlist, verify required tenant predicates are present, and reject anything that fails.
• Resource limits. Statement timeouts, row limits, and cost estimation, because a generated query with a missing join predicate can produce a cross product against a fact table.
• Data masking. Fields the requesting user may not see should be masked at the query layer, so that unmasked values never enter the application process, let alone the model context.
• Auditability. Log the natural language request, the generated query, the execution plan, the row count, and the identity under which it ran. When a decision is later questioned, this is the evidence trail.

The safer architecture, where it fits, avoids generated SQL entirely. Instead of letting the model write queries, expose a set of parameterized, reviewed, permission-aware functions: get_customer_exposure(customer_id), get_payment_history_summary(customer_id, months), get_current_risk_grade(customer_id). The model selects among these and supplies parameters, which is a much smaller and much more verifiable decision than composing arbitrary SQL. The cost is flexibility: every new question shape may require a new function. For high-stakes domains that cost is usually worth paying, and a hybrid is common, with curated functions for regulated decisions and generated queries for exploratory analytics where a wrong answer is annoying rather than dangerous.

Context That Must Be Fetched Live

Some context cannot be indexed, because by the time it is indexed it is wrong.

Account balance changes with every transaction. Inventory changes with every order. Service status changes when something breaks. Shipment state changes as packages move. Subscription status changes when a payment fails. Infrastructure state changes continuously. For any of these, a vector index or a nightly warehouse extract is a snapshot of a past that no longer exists.

This is the boundary between retrieval and API access, and drawing it correctly is an architectural decision with a simple governing question: what is the cost of being wrong by the staleness interval? If the answer is "nothing much," index it. If the answer is "we quote a limit the customer cannot actually have," query the source of truth.

The practical split usually looks like this. Index content that is authored, revised deliberately, and stable between revisions: policies, procedures, product descriptions, contract templates, technical documentation, historical case notes. Query live for anything that reflects current operational state: balances, statuses, positions, availability, health. Index derived aggregates cautiously, with explicit freshness metadata, and prefer live computation when the aggregate feeds a decision with financial consequence.

Live context introduces engineering concerns that indexed retrieval does not.

Tool and API schemas become part of the context. The model must know what it can call, what parameters are required, what the response looks like, and what the call means. Schema quality directly determines call quality, which is discussed further in the section on tool context.

Response normalization matters because API responses are designed for programs, not for reasoning. A raw JSON payload with forty fields, nested envelopes, internal status codes, and epoch timestamps consumes budget and invites misinterpretation. Normalizing to a compact, labeled, human-readable representation with resolved enumerations and formatted dates improves both accuracy and cost. "account_status": "ACTIVE_RESTRICTED_02" should reach the model as something a reader could interpret, along with a note of what the restriction means.

Timeouts and partial failure need explicit policy. If the risk scoring service takes eight seconds and the interaction budget is three, the system must decide whether to proceed without it, degrade to a cached score with a staleness warning, or fail the request. All three are defensible depending on the decision's stakes. What is not defensible is silently proceeding as though the missing input were merely absent rather than unavailable. The context should state explicitly that the risk score could not be retrieved, so the model can abstain rather than reason around a gap it cannot see.

Authentication and delegation determine whose credentials the call runs under. Service-to-service authentication with a broad scope is convenient and turns the AI layer into a confused deputy. Where the platform supports it, calls should carry the requesting user's delegated authority.

Rate limits and cost constrain how many live calls a single request may make. Agentic systems in particular will happily make dozens of calls if nothing stops them. Per-request call budgets, deduplication of identical calls within a turn, and short-lived caching for repeated reads within a session are standard mitigations.

Freshness annotation should accompany every live fact. A balance retrieved at 09:14:02 should carry that timestamp into the context. When a downstream reviewer asks why the system recommended a limit that the account could not support, the answer is often that the balance moved between retrieval and decision, and only the timestamp makes that diagnosable.

The Temporal Dimension

A retrieved document can be highly relevant, from an authoritative source, correctly ranked, and still wrong, because it describes how things worked before they changed.

Enterprise content is full of temporal traps. Policies are revised and the old versions remain in the document store, often indistinguishable in text from the current one. Product terms change while marketing pages describing the old terms stay published. Procedures are superseded by new procedures that do not mention the old ones. Regional rules take effect on a date, so a document that is correct today was wrong last month and will be wrong again after an amendment.

Handling this requires the pipeline to distinguish several different kinds of time, which are frequently conflated:

• Creation time is when the content was authored. It is the weakest signal and the one most often used as a proxy for the others.
• Last modification time is when the file changed, which may reflect a formatting fix rather than a substantive revision.
• Effective date is when the described rule began to apply. This is the field that actually matters for policy content and the one most often missing.
• Expiry or supersession date is when it stopped applying, and by which document it was replaced.
• Event time is when the described thing happened, which for transactional records differs from when it was recorded.
• Processing time is when the system ingested or computed the value.
• Index freshness is how current the retrieval index is relative to the source.
• Cache age is how long ago a cached value was computed.

A system that only tracks modification time cannot answer "which policy was in force when this account was opened," which is a routine question in regulated domains and a common source of disputes.

The practical requirements follow directly. Version and effective-date metadata should be captured at ingestion and stored alongside every chunk, not inferred later. Superseded documents should either be excluded from the default retrieval scope or clearly marked, with an explicit path to retrieve them when historical reasoning is required. Supersession should be modeled as a link between versions rather than a flag, so the chain is traversable. Freshness should appear in the assembled context so the model can weigh it and, where the gap is material, decline to answer.

It is sometimes useful to think about context quality as the interaction of three factors:

context quality ≈ relevance × authority × freshness

This is a reasoning device, not a formula. There is no standardized definition of any of the three terms, no established weighting, and no reason to believe the relationship is multiplicative. Its value is that it makes the failure structure legible. A document that scores zero on any of the three is not partially useful, it is useless: a highly relevant, highly authoritative document that was superseded last month will produce a confidently wrong answer, and so will a current, authoritative document that is irrelevant to the question. Retrieval systems that optimize relevance alone are optimizing one factor of three and will hit a ceiling determined by the other two.

Authority, Ownership, Provenance

Enterprise systems accumulate multiple descriptions of the same thing, and they are not equally trustworthy.

For a question about credit limit policy, a search across an organization's content will plausibly surface: the approved credit policy document with a governance sign-off, a procedure guide written by the operations team summarizing that policy, an internal wiki page written two years ago by someone who has since left, a training deck prepared for onboarding, a support macro with a simplified rule of thumb, an archived email thread discussing a proposed change that was never adopted, and a message in a team channel where someone stated the threshold from memory.

All of these are topically relevant. Several will score well on semantic similarity, because they use the same vocabulary. Some of them are wrong. At least one of them is confidently wrong in a way that will read as authoritative to a model with no basis for distinguishing them.

The mitigation is provenance metadata carried from ingestion through to assembly. At minimum, each indexed unit should carry its source system, document identifier, owning team or individual, approval or governance status, version, effective date, and classification level. Where possible it should carry a lineage reference indicating what it derives from, so a procedure guide can be linked to the policy it summarizes.

From this metadata, an explicit source hierarchy becomes possible. A workable ordering for regulated content places formally approved and version-controlled documents at the top, followed by system-of-record data, followed by team-maintained procedural content with named owners, followed by unowned wiki content, followed by conversational and archival material which should generally be excluded from authoritative answers entirely and used only for discovery or historical reasoning.

This hierarchy has to be encoded somewhere the retrieval system can act on it, which usually means as a ranking signal in reranking and as a filter for certain question classes. A question classified as a policy interpretation should not be answerable from unowned wiki content at all, regardless of how well it matches.

Provenance also drives citation. A credit recommendation that cites "internal documentation" is not auditable. One that cites "Credit Policy v7.2, section 4.3, effective 2026-03-01, owner: Risk Governance" can be verified by a human in seconds and defended to a regulator. Citation correctness, meaning whether the cited source actually supports the claim, is a measurable property and one of the more useful automated checks available in the validation stage.

Underlying all of this is a principle worth stating plainly: the model should not treat every retrieved fragment as equally trustworthy, and it cannot make that distinction unless the system tells it. Trust is not inferable from text. It is metadata, and if the assembly stage strips it, the information is gone.

When Sources Disagree

Conflict is not an edge case in enterprise retrieval. It is the normal condition of any organization that has existed long enough to have documented the same thing more than once.

The conflict shapes vary. Two documents state different numeric thresholds. A policy states a rule and a procedure describes an exception that the policy does not mention. A structured system reports a status that contradicts a document describing the account. A regional supplement modifies a global policy in a way that is only apparent if both are present. Memory holds a customer preference that the CRM record contradicts. A cached value disagrees with a live one.

The default behavior of an unprepared system is arbitrary. The model will typically favor whichever statement is more prominent, more recent in the context ordering, more emphatically phrased, or more frequently repeated across retrieved chunks. None of those correlates with correctness. Repetition in particular is actively misleading, because duplicated content in enterprise corpora usually reflects copying rather than independent corroboration.

Several strategies help, and they compose.

Detection before resolution. Many systems never notice the conflict. Detecting it requires either a comparison step that checks retrieved claims against each other, which can be a cheap model call over the assembled candidates, or structural rules that flag when two chunks from different documents both match a known claim pattern such as a threshold value. Detection is imperfect but far better than nothing, and even a coarse detector that catches numeric disagreements is worth having.

Authority ranking. Where sources are ranked, the conflict can often be resolved by rule: the approved policy wins over the procedure guide, the system of record wins over any document describing it, the regional supplement wins over the global policy within its jurisdiction. These rules should be explicit and reviewable, not implicit in a ranking score.

Freshness and supersession. Where authority is equal, the current effective version wins, provided supersession is modeled rather than inferred from timestamps.

Ownership escalation. Where authority and freshness do not resolve it, the conflict is a genuine content defect. The correct output is not a confident answer. It is a flagged discrepancy routed to the owning team, plus a response to the user that states what was found and why it is unresolved.

Abstention. This deserves emphasis because it runs against the grain of how these systems are usually built and evaluated. For a class of enterprise decisions, a system that answers ninety percent of questions correctly and abstains on ten percent is more valuable than one that answers one hundred percent with a five percent error rate, because the errors are silent and the abstentions are visible. Abstention has to be designed in: the context must carry enough metadata for the model to recognize insufficiency, the instructions must authorize refusal, the output schema must have a place to express it, and the evaluation must reward it rather than scoring it as a failure.

Building an AI system that says "the current policy and the regional supplement give different thresholds for this case, and I cannot determine which applies" is harder than building one that picks a number. It is also the version that can be deployed against decisions that matter.

Memory Is Not Chat History

Memory is the term applied most loosely in this field. It commonly means "we append previous turns to the prompt," which is not memory in any meaningful sense. Several genuinely different mechanisms are being conflated, and they have different storage, different retrieval, different lifetimes, and different ways of going wrong.

Conversation history is the verbatim sequence of turns in the current session. It provides referential continuity, so that "this customer" and "the second option" resolve correctly. It grows linearly, has no value after the session ends, and is the cheapest thing to discard.

Session state is the structured working state of the current interaction: the resolved customer identifier, the current task, the values collected so far, the step of a multi-step process. This should be held as typed application state rather than inferred from the transcript. Systems that reconstruct state by re-reading the conversation each turn are fragile in exactly the way you would expect, because the model must re-derive facts it already established and will occasionally derive them differently.

User memory persists facts about the person interacting with the system: their role, their team, their stated preferences about verbosity or format, their recurring areas of responsibility. It spans sessions and is usually small.

Task memory persists the state of work in progress across sessions. A credit review begun on Tuesday and resumed on Thursday should not restart from nothing. This is the memory type most often missing and most often noticed by users when absent.

Semantic memory holds general facts the system has learned, independent of when it learned them. That a particular product line is not offered in a given region, that a specific approval requires two signatures, that a customer segment has an unusual billing cycle. It is closer to a knowledge base than to a log.

Episodic memory holds specific past events with their circumstances. That a limit increase for this account was reviewed in Q1 and declined for a stated reason. This is the memory type most relevant to decision consistency, because it is how an organization avoids contradicting itself.

Organizational memory spans users. A decision made by one analyst should be visible to another reviewing the same account. This introduces access control questions immediately, since organizational memory can carry information one user is entitled to and another is not.

What should persist is a design decision, not a default. The useful discipline is to store facts that are stable, reusable, and expensive to re-derive, and to discard everything else. A user stating "I manage the EMEA portfolio" is a durable fact worth storing. A user asking "what was the balance again?" is not.

Persistence therefore requires an extraction step rather than wholesale storage. Extraction identifies candidate facts from an interaction, typically through a model call over the completed exchange with a schema constraining what kinds of fact may be extracted. Candidates then need validation: does this contradict something already stored, is it a stable fact or a transient one, is it attributable to a reliable turn of the conversation, does the user have authority to assert it. A user correcting the system about their own role is authoritative. A user asserting a policy threshold is not, and storing that assertion as semantic memory is how systems learn to be wrong.

Consolidation merges related memories and resolves redundancy, so that fifteen observations of a preference become one durable record with a confidence and a last-confirmed date rather than fifteen near-duplicates that will all retrieve together and consume budget.

Retrieval of memory is a retrieval problem with its own ranking, not a dump. A user with two years of interaction history has far more stored memory than belongs in any single context. Memory retrieval should be query-conditioned and budget-constrained like any other retrieval path.

Expiration and correction close the system. Memories should carry a last-confirmed timestamp and, where appropriate, a time-to-live. Memories that contradict newly observed facts should be updated rather than accumulated alongside. Memories that have not been reinforced in a long period should decay in ranking weight even if not deleted outright.

How Memory Poisons Itself

Retrieval errors are transient. A bad retrieval affects one answer, and the next request starts fresh. Memory errors are not transient. A false fact written to memory will be retrieved again, and again, and each retrieval reinforces the impression that it is established.

The mechanisms are worth naming because each requires a different control.

False facts enter when extraction misinterprets a statement, most commonly by capturing a hypothetical, a question, or a quotation as an assertion. A user asking "does the policy require two approvals?" can be extracted as "the policy requires two approvals" by an extractor that is not attending to sentence mood.

Outdated preferences accumulate when a stored fact was true and stopped being true, and nothing in the system is watching for the change. A user who moved teams eight months ago is still described by their old team.

Incorrect entity associations are the most damaging class, because they cross record boundaries. A fact about customer A written under customer B's identifier will surface in every future interaction about B, and it will look plausible. This happens most often when entity resolution during a multi-customer conversation is ambiguous.

Stale task state leaves a completed workflow marked as in progress, so the system repeatedly tries to resume work that finished.

Duplicated memories arise when consolidation is absent, so the same fact stored six times crowds out other material during retrieval and creates the appearance of strong evidence.

Conflicting memories are duplicates that disagree, which produces nondeterministic behavior depending on which version wins the ranking on a given request.

What makes these expensive is that they are difficult to attribute. A user reports that the system keeps getting something wrong. The prompt looks correct. Retrieval on the current query looks correct. The model behaves correctly given its input. The defect is a record written three weeks earlier that nobody is looking at.

The controls follow a straightforward lifecycle: observe the interaction, extract candidate facts under a schema, validate against existing memory and authority rules, store with provenance and timestamps, retrieve under query-conditioned ranking, reinforce or correct based on subsequent evidence, and expire on decay or explicit invalidation. This is a lifecycle rather than a maturity progression: every one of these steps is required from the first day memory is enabled, not adopted incrementally as the system matures.

Two operational practices matter more than any of the individual steps. First, memory must be inspectable. A support engineer should be able to view everything stored about a user or entity, with provenance showing which interaction produced each record. Second, memory must be correctable, both by users about themselves and by operators about entities, with the correction propagating rather than being appended alongside the error.

Relationships as Context

A significant share of enterprise questions are not answerable by finding the right passage. They are answerable by traversing the right connections.

The credit question depends on a chain: the customer owns accounts, the accounts are governed by a contract, the contract references product terms, the product is subject to a policy, the customer is located in a region, and the region is subject to a regulation. No single document contains that chain. Each link lives in a different system, and the answer requires assembling the path.

Vector retrieval handles this poorly, because similarity does not traverse. A query about a customer will retrieve documents that mention the customer. It will not reliably retrieve the regulation that applies because of where the customer is located, since that regulation contains no mention of the customer and shares little vocabulary with the query.

Where relationships determine the answer, a graph representation of entities and their connections provides something text retrieval cannot: the ability to start from a resolved entity and walk outward along typed edges to collect exactly the connected facts that matter, bounded by traversal depth and edge type rather than by similarity score.

For context construction specifically, this changes what enters the window. Instead of ten text chunks that each mention credit limits, the context can contain a compact structured subgraph: this customer, these three accounts with their statuses, the governing contract with its effective terms, the applicable policy version, the region, and the two regulatory constraints that attach to it. That is far fewer tokens carrying far more decision-relevant information, and it is verifiable, because each edge came from a system of record rather than from a similarity score.

Two practical requirements dominate. Entity resolution must be reliable, because the entire traversal depends on starting at the correct node, and enterprise systems are full of duplicate and near-duplicate entity records across CRM, billing, and support systems. And traversal must be bounded, because unconstrained expansion from a well-connected node will return a large fraction of the graph.

Graph and vector retrieval are complementary rather than competing. The common pattern uses the graph to establish the entity scope and the relationship structure, then uses vector retrieval within that scope to find the relevant explanatory text. The graph answers "which policy applies," and retrieval answers "what does it say."

Assembly: Turning Fragments Into a Context Package

Suppose the pipeline has produced, for the credit question, five policy fragments, twelve customer facts drawn from structured systems, three previous decisions, two support incidents, one regulatory constraint, and four tool outputs. Twenty-seven items of heterogeneous type, origin, and trust level.

Concatenating them in retrieval-score order is the default and it is a poor one. Assembly is a design step, and several of its choices measurably affect behavior.

Grouping by role, not by source. Facts should be organized by what they contribute to the decision rather than by which system produced them. The model reasoning about credit exposure benefits from all exposure-related facts appearing together, whether they came from the core banking API, the CRM, or a derived aggregate.

Explicit boundaries and labels. Each block should be delimited and named. Structured delimiters, whether XML-style tags, headed markdown sections, or JSON objects, serve two purposes: they let the model distinguish an instruction from a document from a tool result, and they let the system programmatically verify what was sent, which matters for tracing.

Ordering with position in mind. Attention is not uniform across position. Material at the beginning and end of a long context tends to be attended to more reliably than material in the middle. The most decision-critical items, typically the governing policy and the hard constraints, should not be buried at the midpoint of a long block of supporting material.

Metadata attached to each item. Source, version, effective date, last-updated timestamp, and authority level travel with the content rather than being stripped during formatting. This is what enables the model to weigh conflicting items and to cite correctly.

Explicit absence. Where a required input could not be retrieved, the context should say so rather than omitting it silently. A model that cannot see that the risk score is missing will reason as though the absence is informative.

Token allocation by category. Rather than one global budget consumed first-come, allocate per category: a ceiling for policy, a ceiling for customer state, a ceiling for history. This prevents a verbose category from crowding out a terse but essential one, which is the usual outcome of a single shared budget filled in rank order.

A conceptual package for the credit question:

<identity>
  requester: analyst_id=4471, role=credit_risk_analyst
  entitlements: view_full_exposure, view_payment_history
  tenant: eu_retail
</identity>

<task>
  Assess eligibility for a credit line increase on account ACC-88214.
  Return: decision, governing_clause, limiting_factor, confidence.
  If any required input is missing or sources conflict, abstain and state why.
</task>

<customer_state as_of="2026-08-08T09:14:02Z" source="core_banking_api">
  account_age_months: 41
  current_balance: 3,240.00 EUR
  current_limit: 5,000.00 EUR
  utilization: 0.648
  late_payments_12m: 1 (severity: 1-29 days)
  aggregate_exposure_all_products: 11,780.00 EUR
  account_status: active
</customer_state>

<policy source="Credit Policy v7.2" effective="2026-03-01"
        authority="approved" owner="risk_governance">
  ...clause text with section path...
</policy>

<constraints jurisdiction="EU" source="Regional Supplement EU-4"
             effective="2026-01-15">
  ...affordability assessment requirement...
</constraints>

<history>
  2026-02-11 limit_increase_request declined
    reason: utilization above threshold at review date
    decided_by: analyst_id=2210
</history>

<tool_results>
  risk_score: unavailable (timeout after 3000ms) — treat as missing input
</tool_results>

<available_tools>
  recalculate_affordability(account_id, proposed_limit)
  fetch_risk_score(customer_id)  [currently degraded]
</available_tools>

Several properties of this package deserve notice. Every factual block carries a source and a time. The missing risk score is stated rather than omitted. The instruction block explicitly authorizes abstention and names the output fields, which gives the model a place to express insufficiency. The identity block is supplied by the application, not by the user's message, so a user cannot escalate their own entitlements by asserting a role in chat.

Structure of this kind is not cosmetic. Models behave differently when the boundary between an instruction and a retrieved document is explicit, and the difference is largest precisely in the cases that matter most: when retrieved content contains something that resembles an instruction.

Compression and What It Destroys

When the selected material exceeds the budget, something must give. The available techniques trade fidelity for size in different ways, and choosing badly removes the exact detail the decision required.

Extractive compression selects a subset of the original sentences or passages verbatim. It is lossy at the passage level and lossless at the sentence level, which means it preserves exact figures and clause wording. It is the safest option for regulated or numeric content.

Abstractive summarization rewrites content in condensed form. It achieves much higher compression and introduces paraphrase risk, which for policy text is a genuine hazard: a summary that renders "may be increased by up to twenty percent subject to affordability assessment" as "may be increased" has removed both the bound and the condition.

Hierarchical summarization maintains summaries at multiple granularities, allowing the system to include a summary by default and drill into the original where the query touches it. This is more useful than a flat summary and costs more to maintain.

Fact extraction converts prose into structured assertions with source references. It compresses aggressively and works well for numeric and categorical content, poorly for content whose meaning depends on qualifying language.

Query-focused summarization condenses relative to the current question rather than in general, which preserves more of what matters for this request and less of what does not. It cannot be cached across queries, which is its main cost.

The dividing question is whether a transformation is reversible in practice. If the system retains a reference to the original and can retrieve it on demand, compression is recoverable. If the original is discarded or unreachable at inference time, it is not, and any detail lost is lost for the decision.

The practical rule for high-stakes content is to keep the operative text verbatim and compress everything around it. For the credit question, the specific clause that determines the threshold should reach the model in its original words. The eight paragraphs of surrounding context explaining the policy's rationale can be summarized to a sentence or dropped entirely. Compression should be applied to the supporting material, not to the load-bearing material, and distinguishing between the two is a domain judgment that has to be encoded deliberately.

Caching the Expensive Parts

Some context is reconstructed identically on every request, and reconstructing it is not free.

Long system instructions, tool and function schemas, database schema descriptions with semantic annotations, stable product documentation, and conversation summaries all fall into this category. Several distinct caching mechanisms apply to them, and conflating them causes problems.

Prompt caching at the inference provider level caches the processed prefix of a context so that repeated identical prefixes are billed and processed at reduced cost. It requires stable prefixes, which has a design consequence: the invariant material must be placed at the front of the context and the variable material behind it. Systems that interleave stable instructions with per-request data forfeit this entirely.

Retrieval caching stores the result of a query against the index. It is safe for identical queries against an unchanged index and unsafe the moment either changes, which means invalidation must be tied to index generation.

Semantic caching returns a stored result for a query that is similar rather than identical to a previous one. It offers the largest savings and carries the most risk, because "similar" is a threshold judgment and two questions that embed closely can have different correct answers. Semantic caching of answers in a system where the underlying data changes is generally a mistake. Semantic caching of retrieval results, with a freshness bound, is more defensible.

Application caching holds normalized API responses and computed aggregates for a short window within a session, preventing the same balance from being fetched six times during one interaction.

The distinction that matters operationally is between a cached answer and a cached context. A cached answer is a claim about the world that may have expired. A cached context is a set of inputs that may have expired, and the model will still reason correctly over them, arriving at a correct conclusion from stale premises. The second failure is harder to detect because everything downstream looks healthy.

Every cached element should therefore carry its own age and invalidation basis, and any cached fact that enters the model context should carry its timestamp into the context so that staleness is visible rather than assumed away.

Tools Need Context Too

Once a system can act rather than only answer, a new category of context becomes decisive: the description of what actions exist and what they do.

Tool context includes the tool's name, description, parameter schema with types and constraints, return shape, side effects, cost or latency characteristics, current availability, required permissions, and the results and errors of calls already made in this session.

Poor tool context produces a recognizable set of failures. Wrong tool selection happens when two tools have overlapping descriptions and nothing distinguishes their intended use, so the model picks by surface similarity. Incorrect parameters happen when a schema declares a type but not a meaning, so a field named amount receives a value in the wrong unit, or a date field receives a formatted string the API rejects. Unnecessary calls happen when the model cannot tell that it already has the information, which occurs when prior results are buried in a long trajectory rather than surfaced as current state. Unsafe actions happen when a tool's side effects are not described, so a call that appeared to be a read turns out to write.

The correction is to treat tool interfaces as an artifact designed for model consumption rather than a mechanical export of an existing API surface. That implies several things in practice. Descriptions should state when to use the tool and when not to, since the negative case is what disambiguates overlapping tools. Parameter descriptions should include units, formats, and valid ranges. Enumerations should be enumerated rather than described. Errors should be returned as actionable text explaining what to do differently rather than as status codes. Read and write operations should be clearly separated, and destructive operations should require an explicit confirmation parameter that the model must set deliberately.

There is also a budget dimension. A system exposing eighty tools has spent a large amount of context on schema definitions before any request-specific information arrives, and selection accuracy typically degrades as the tool count grows. Filtering the exposed toolset by intent, so that a credit assessment sees the seven relevant tools rather than all eighty, improves both cost and accuracy.

Agents and the Trajectory Problem

Multi-step agents make context management the dominant engineering concern rather than a supporting one, because the context is no longer assembled once. It evolves across every step, and the naive evolution is unbounded growth.

An agent operating over a task must track its goal, its plan, its current position in that plan, the observations it has accumulated, the actions it has already taken, the tools available, the constraints it operates under, relevant memory, and the errors it has encountered. The straightforward implementation appends every action and every observation to a growing transcript and resubmits the whole thing each step.

This degrades in three ways simultaneously. Cost grows quadratically in the number of steps, since each step reprocesses everything before it. Latency grows with it. And accuracy declines, because the early steps of a trajectory contain a large volume of material that is no longer relevant, including failed attempts, superseded plans, and raw tool outputs whose useful content was a single field.

The alternatives all involve maintaining structured state separately from the transcript.

Explicit agent state holds the goal, plan, completed steps with their outcomes, and accumulated findings as typed data that the application owns. Each step's context is then constructed from this state rather than from a transcript, which means the context size is bounded by the state size rather than by the step count.

Observation distillation extracts the decision-relevant content from each tool result immediately and discards the raw payload. A query returning four hundred rows becomes a summary of the three that mattered.

Trajectory summarization compresses older steps into a condensed record while keeping recent steps verbatim, on the reasoning that recent detail matters more for the next action.

Checkpointing persists state at step boundaries so that a long-running task can be suspended, resumed, inspected, or rolled back without replaying the trajectory.

Event logs retain the full history outside the context for audit and debugging, so that discarding material from the working context does not mean losing it.

The design consequence is that an agent's memory architecture governs how long a task it can complete and how reliably. Two systems using the same model, given the same task, will differ enormously in outcome based on whether step forty has access to a coherent state summary or to thirty-nine steps of accumulated transcript.

What Each Agent Should Not Know

Multi-agent designs are often justified by specialization: a retrieval agent, an analysis agent, a validation agent, each better at its narrow task than a generalist would be.

That justification depends entirely on the context each agent receives. Specialization is a property of the context, not of the label. Three agents given identical full context and different system prompts are three prompts, not three specialists, and they will exhibit correlated failures because they are reasoning over the same material.

Genuine specialization means each agent operates over a deliberately restricted view. A retrieval agent needs the query, the source catalog, and the retrieval tools. It does not need the customer's financial position. An analysis agent needs the assembled evidence and the policy. It does not need the raw search results or the retrieval scores. A validation agent needs the proposed output, the evidence it claims to rest on, and the rules it must satisfy, and it should specifically not have access to the reasoning that produced the output, because seeing that reasoning biases it toward accepting it.

This raises the coordination questions that determine whether a multi-agent system is an architecture or an expense. What does each agent know, what is shared, what stays private, how are findings transferred between agents, and how is conflicting state resolved when two agents reach incompatible conclusions.

The common coordination patterns each make a different trade. Shared memory, where agents read and write a common store, is simple and makes isolation difficult. Message passing, where agents exchange explicit structured results, preserves isolation and requires defining the interfaces. Blackboard architectures, where agents post findings to a shared structured space and subscribe to relevant portions, sit between the two. Event-driven coordination decouples agents entirely and complicates reasoning about global state.

Whichever is chosen, the transfer format matters more than the topology. An agent that passes its full reasoning transcript to the next agent has transferred noise. An agent that passes a structured result with findings, confidence, sources, and unresolved questions has transferred information. The second is smaller, more useful, and far easier to validate.

Retrieval Must Not Become a Privilege Escalation Path

There is one principle in this entire area that admits no trade-off: retrieval must never provide access that the application would otherwise deny.

The failure is easy to build accidentally. An organization indexes its document repositories into a vector store. The vector store holds embeddings and text with minimal metadata. The AI application queries it with a service account. A user asks a question, retrieval returns the most similar chunks, and the model summarizes them. If one of those chunks came from a document the user cannot open in the source system, the application has just become a mechanism for reading it. No access control was bypassed in any conventional sense. The document was simply copied into a system where access control was not modeled.

Preventing this requires access decisions to constrain retrieval rather than to filter its output.

Pre-filtering applies the user's entitlements as a constraint on the search itself, so the candidate set only ever contains material the user may see. This requires permission metadata on every indexed unit and a query path that can express the constraint efficiently, which is a real engineering cost, particularly for permission models with group inheritance and many principals.

Post-filtering retrieves first and removes unauthorized results afterward. It is easier to build and has two defects. It leaks information through result counts, latency, and relevance behavior, which is a genuine side channel in some threat models. More importantly, it is fragile: any code path that forgets the filter, any caching layer that stores pre-filter results, any debugging endpoint that returns raw retrieval output becomes a disclosure.

The stronger position is that sensitive information should not enter the model context at all unless the requester is entitled to it, and the enforcement point should be as close to the data as possible.

Several other controls belong in the same layer. Document-level permissions must be captured at ingestion and refreshed when they change in the source system, which means the index needs a permission synchronization path, not a one-time copy. Row-level and column-level security must be enforced by the database under the user's identity rather than reimplemented in application code. Field-level masking should apply before values leave the data layer. Personally identifiable information and secrets should be detected and redacted during ingestion, since a credential pasted into a support ticket that gets indexed is now retrievable by anyone who can query the index.

The general shape is that context architecture becomes part of security architecture. Every new source connected to the retrieval layer is a new path by which information can reach a user, and it needs to be reviewed as such.

Instructions Arriving as Data

Retrieved content is data. Models do not reliably treat it that way.

Indirect prompt injection exploits this. Text placed in any source the system might retrieve can contain instructions, and a model processing that text may follow them. The vectors are numerous: a web page the system fetches, a document uploaded by a third party, an email in a shared mailbox, a support ticket submitted by a customer, a field in a CRM record, a code comment, the response body of an external API, or the output of a tool that itself consumed untrusted input.

The consequences scale with the system's capabilities. In a read-only question answering system the worst case is a manipulated answer. In an agentic system with tool access, injected instructions can cause data exfiltration through outbound tool calls, unauthorized writes, or manipulation of a decision with financial consequences. A credit assessment system that retrieves a support ticket containing text designed to influence the assessment is exposed to exactly this.

There is no complete defense. Anyone claiming otherwise is describing a mitigation. What is available is a set of controls that reduce the attack surface and limit the blast radius.

Explicit trust levels in the context. System instructions, developer configuration, authenticated user requests, retrieved documents, and tool results are five different trust categories, and the context should mark them as such rather than presenting them as undifferentiated text. Structured delimiters with explicit labels, combined with instructions stating that content inside data blocks is never to be treated as directive, measurably reduces susceptibility without eliminating it.

Capability restriction rather than content filtering. The more reliable control is to limit what a compromised reasoning step can do. Read and write tools separated, write operations requiring confirmation, outbound network calls restricted to an allowlist, per-request action budgets, and human approval for consequential actions. If injected text cannot cause an action, it can only cause a wrong answer.

Output validation independent of the reasoning. Checking that a proposed action is consistent with the original request, that cited sources actually contain the cited claims, and that the output conforms to the expected schema catches a meaningful fraction of manipulated outputs without needing to detect the injection itself.

Provenance on every claim. If a surprising conclusion can be traced to a specific retrieved chunk, the injection becomes visible in the trace even when it was not prevented.

Sanitization, with modest expectations. Stripping known instruction patterns from retrieved content catches naive attempts and fails against anything adaptive, since the space of ways to express an instruction in natural language is not enumerable. It is worth doing as one layer, and it should not be relied upon.

Tenant Boundaries Inside the Context

Multi-tenant AI systems add a failure mode with unusually high severity: context from one customer reaching another.

The paths are more numerous than they first appear. The vector index may hold vectors from multiple tenants with tenant identity as a filter rather than a partition, which means one missing predicate causes a cross-tenant retrieval. Memory stores may key on user identity without tenant scoping, which fails when a user identifier is not globally unique. Caches are a frequent culprit, since a cache key that omits tenant identity will serve one tenant's retrieval results to another. Conversation summaries and derived artifacts may be stored without inheriting the tenant scope of their source material. Tool credentials may be shared across tenants, so a tool call made in one tenant's session reads another tenant's data.

The architectural preference is separation rather than filtering wherever the cost allows. Separate indexes or namespaces per tenant remove the class of failure where a filter is omitted, at the cost of more index management overhead and worse resource utilization for very large numbers of small tenants. Where shared indexes are necessary, the tenant predicate should be enforced by the retrieval client rather than supplied by callers, so that it cannot be forgotten.

Every cache key, every memory key, every derived artifact identifier, and every tool authorization should carry tenant identity as a structural component rather than as an attribute. And the isolation should be tested rather than assumed, with automated cases that deliberately query for another tenant's known content and assert that nothing is returned.

The Context Lifecycle

Context engineering extends considerably further than the code path that runs during an inference call. Information passes through a sequence of stages, most of which occur long before any request arrives.

Create. Information enters the organization: a policy is approved, a record is written, a ticket is filed, a contract is signed. Decisions made here, particularly about structure, metadata, and ownership, determine what is possible downstream. A policy published as a scanned PDF with no version field constrains everything that follows.

Ingest. The system discovers and acquires content. This requires connectors, change detection, and permission capture. Incremental ingestion is significantly harder than initial ingestion and is where most pipelines accumulate defects, because deletions and permission changes in the source are easy to miss.

Normalize. Formats are converted, structure is extracted, metadata is standardized, entities are resolved, and duplicates are identified. This stage does the most to determine retrieval quality and receives the least attention.

Index. Content becomes retrievable through embeddings, lexical indexes, structured tables, or graph nodes and edges. Index generation and rebuild strategy belongs here, along with the freshness guarantees the rest of the system can rely on.

Select. At request time, relevant material is identified under the constraints established earlier.

Assemble. The runtime context is constructed.

Consume. The model reasons over it.

Observe. The system records what was retrieved, what was included, what the model produced, and what happened next.

Update. Knowledge bases are corrected, memory is written, retrieval tuning is adjusted based on observed outcomes.

Expire. Superseded content is marked, stale memory decays, orphaned records are removed, and permission revocations propagate.

The stages that fail most often in practice are normalize, update, and expire, and they fail quietly. A pipeline with excellent retrieval over content that was never normalized and is never expired will slowly degrade as the corpus fills with superseded material that continues to rank well.

Reconstructing What the Model Saw

Standard LLM observability records the prompt, the response, latency, token counts, and cost. For a context-engineered system this is insufficient, because the prompt as recorded is the end of a long process and reveals almost nothing about how it was produced.

Diagnosing a wrong answer requires reconstructing the information environment that produced it. That means capturing, per request: the classified intent and extracted entities, the resolved identity and entitlements, the context requirements derived from the intent, the sources selected, every query issued to every source with its parameters, the candidate sets returned with their scores, the filtering decisions and their reasons, the reranking scores, the deduplication actions, the compression applied and what it removed, the final assembled context with its composition by category and token count, the memory records retrieved, the tool calls made with their results and errors, and the validation checks with their outcomes.

This is a substantial volume of telemetry, and it needs sampling policies and retention limits to be affordable. It also needs care, because these traces contain the retrieved content itself, which means they inherit its sensitivity classification. A debugging store full of retrieved customer records is a data protection concern in its own right.

The payoff is that failure attribution becomes possible. When a system produces a wrong credit recommendation, the question "was this a retrieval failure, a freshness failure, a permission failure, a memory failure, an assembly failure, or a model failure" has an answer that can be looked up rather than guessed at. Without that, teams tune prompts in response to problems that prompts did not cause.

A Context Trace

A minimal trace for the running example:

request_id: 9f2c-441a
user: analyst_4471  role: credit_risk_analyst  tenant: eu_retail
intent: credit_limit_assessment   entities: {account: ACC-88214}

authorization:
  entitlements resolved: view_full_exposure, view_payment_history
  retrieval scope: eu_retail ∩ policy_approved ∪ analyst_readable

retrieval:
  policy_index      q="credit line increase threshold"   14 candidates
  policy_index      filtered → 9  (5 superseded versions removed)
  policy_index      reranked → 3  (top: Credit Policy v7.2 §4.3, 0.91)
  regional_index    q="EU affordability assessment"      2 candidates → 1
  case_history      account=ACC-88214, window=24m         1 record

structured:
  core_banking.get_account_state(ACC-88214)   42ms   as_of 09:14:02Z
  core_banking.get_exposure(CUST-11902)       61ms   as_of 09:14:02Z

memory:
  episodic: prior_review 2026-02-11 declined (utilization)

tools:
  risk_api.fetch_score(CUST-11902)  TIMEOUT 3000ms  → marked unavailable

assembly:
  blocks: identity, task, customer_state, policy, constraints,
          history, tool_results, available_tools
  tokens: 8,420  (policy 3,110 / customer_state 640 / constraints 890 /
                  history 210 / instructions 3,570)
  omitted: 6 candidates below relevance floor, 5 superseded versions

model: response in 2.4s
validation:
  schema: pass
  citation check: 2/2 cited clauses present in context
  abstention: triggered (missing required input: risk_score)
outcome: escalated to human review

Note that the interesting content is not the model call. It is the timeout, the five superseded policy versions that were correctly excluded, and the abstention that followed from a missing input being explicitly represented rather than silently dropped. A trace that recorded only the prompt and response would show a system that declined to answer, with no indication of why.

A Taxonomy of Context Failures

The value of naming failure modes is that each has a different detection method and a different fix, and teams that lack the vocabulary tend to treat all of them as "the model got it wrong."

Missing context is the most common and the most consequential. The required evidence was never retrieved, so the model reasons from a gap. It appears in production as a plausible answer that omits a critical qualifier, or as an unnecessary refusal. It is detected by evaluating retrieval against a labeled set of required evidence, not by reading answers.

Irrelevant context fills the budget with topically adjacent material that does not bear on the question. It shows up as answers that drift toward the wrong aspect of a topic and as rising costs without accuracy gains. Precision measures against a labeled set reveal it.

Stale context delivers material that was correct at some past time. It is the hardest failure to see, because the answer is well-formed, well-cited, and internally consistent. Detection requires comparing the effective dates of retrieved material against the request time, which is only possible if those dates were captured at ingestion.

Conflicting context delivers sources that disagree without the metadata to resolve them. It manifests as inconsistency: the same question answered differently across runs, which users report as unreliability rather than as a data defect.

Unauthorized context delivers material the requester should not see. It rarely produces a complaint, because the recipient has no way to know it was unauthorized. It is found by deliberate testing rather than by observation.

Overloaded context exceeds the point of diminishing returns, diluting attention across too much material. The signature is accuracy that fails to improve or declines as retrieval depth increases, which is only visible if depth is varied experimentally.

Duplicated context presents the same fact several times from copies of one source. It wastes budget and creates false corroboration that can override a single contradicting authority. Near-duplicate detection at ingestion and at assembly addresses it.

Misordered context places critical material where it is least likely to be attended to. It produces intermittent failures that look like model unreliability and respond to reordering experiments.

Low-authority context allows an unofficial source to answer a question that requires an official one. The answer may even be correct, which makes it worse, because the system has demonstrated a behavior that will eventually be wrong in a way nobody is watching for.

Incorrect memory produces errors that recur across sessions and resist prompt-level fixes. The distinguishing signature is persistence: the same wrong belief appearing in unrelated conversations.

Retrieval failure covers infrastructure-level problems, including index staleness after a failed rebuild, embedding model version mismatch between index and query time, and silent connector failures that stop ingesting a source without alerting. These produce broad degradation rather than specific wrong answers, and they are caught by monitoring index freshness and document counts rather than by evaluating outputs.

Tool-state failure occurs when tool results are missing, stale, or misrepresented in the context, most often when a failed call is dropped rather than reported. The fix is representing failure explicitly, as in the trace above.

Context injection is instruction-bearing content arriving through a data channel, discussed earlier. It appears as behavior inconsistent with the request, and it is found by tracing surprising outputs back to their source material.

Compression loss removes the operative detail while preserving the surrounding explanation. It is detected by evaluating whether the compressed representation still supports the correct answer on a labeled set, which is a different test from whether the summary reads well.

Entity resolution failure attaches the right analysis to the wrong entity. It is the most dangerous failure in this list, because the output is entirely well-formed and entirely about somebody else. It is guarded against by requiring explicit confirmation of resolved entities in the output and validating them against the request.

Unreliable Output as an Information Supply Problem

It would be convenient to claim that good context engineering eliminates hallucination. It does not, and the claim confuses two separable causes.

Some unreliable output is an information supply problem. The model lacked a fact and filled the gap. The model was given a wrong fact and reasoned correctly from it. The model was given two contradictory facts with no basis for choosing. The model was given an unofficial source and treated it as official. In each of these cases the model behaved reasonably given its input, and the defect is upstream. These failures respond to retrieval improvements, provenance metadata, freshness controls, and conflict handling.

Other unreliable output is model behavior. A model can invent a citation to a document that is not in its context. It can assert a conclusion the supplied evidence does not support. It can silently substitute its parametric knowledge for the provided material, which happens most often when the provided material contradicts what the model learned during training. It can fail to abstain when the context explicitly states an input is missing. None of these are fixed by better retrieval, because the evidence was already there.

The practical consequence is that model reliability and context reliability require separate evaluation and separate remedies. A team that responds to every error by expanding retrieval will make no progress on the second category, and a team that responds to every error by refining the prompt will make no progress on the first. Distinguishing them requires the failure attribution that tracing provides.

Evaluating the Layers Separately

End-to-end answer quality is the metric everyone starts with and the one that provides the least actionable information, because it conflates independent systems.

Two observations make the case. A system can produce the correct answer from poor retrieval, either by chance or because the model already knew the answer from training. This looks like success and will collapse the moment the question moves to material the model never saw. And a system can retrieve exactly the right evidence and still produce a bad answer, because the model ignored it, misread it, or over-generalized from it. This looks like a retrieval problem and is not.

Three evaluation layers are therefore needed, each with its own dataset and its own pass criteria.

Context evaluation asks whether the assembled information environment was adequate, independent of what the model did with it. Did retrieval surface the required evidence? Was irrelevant material excluded? Was the retrieved material current and non-superseded? Was the authoritative source selected over the secondary one? Were permissions correctly applied, both in what was included and in what was excluded? Were entities resolved correctly? Was the token budget allocated sensibly across categories? This layer is scored against labeled expectations, and it can be evaluated without running the model at all, which makes it fast and cheap enough to run on every change to the retrieval stack.

Model evaluation holds the context fixed and asks what the model did with it. Did it use the supplied evidence rather than substituting its own knowledge? Did it follow the output schema and the behavioral instructions? Did it cite correctly, meaning that cited claims are actually present in the supplied context? Did it abstain when the context was deliberately made insufficient? Did it correctly prefer the authoritative source when the context contained a conflict? This layer requires fixed context fixtures, including deliberately degraded ones, which is the only way to test abstention and conflict handling reliably.

System evaluation asks whether the outcome was useful. Was the answer correct and actionable? Was it produced within the latency budget? What did it cost? Was it safe? Was the escalation behavior appropriate? Did the human accept the recommendation, and if it was overturned, why?

Running all three is more work than running one. The compensation is that when a metric moves, you know which subsystem to look at.

Testing the Information Pipeline

Traditional software testing asserts that given an input, a system produces an expected output. AI systems resist this because outputs vary. The information pipeline, however, is largely deterministic and testable in the conventional sense, which makes it the most tractable part of the system to put under regression control.

A useful regression case for a context-engineered system stores more than an input and an expected answer:

id: credit_limit_eu_utilization_high
input: "Can we offer this customer a larger credit limit?"
user: {role: credit_risk_analyst, tenant: eu_retail}
entities: {account: ACC-88214}
required_evidence:
  - {doc: credit_policy, version: "7.2", section: "4.3"}
  - {source: core_banking, field: utilization}
  - {doc: regional_supplement_eu4, section: "2.1"}
forbidden_evidence:
  - {doc: credit_policy, version: "6.4"}   # superseded
  - {tenant: us_retail}                     # cross-tenant
expected_source_types: [approved_policy, system_of_record]
expected_behavior: abstain_if_risk_score_unavailable
max_context_tokens: 12000

The distinctive fields are required_evidence and forbidden_evidence. The first asserts recall: the pipeline must surface these items. The second asserts precision and safety: these items must not appear, whether because they are superseded, out of tenant, or outside the user's entitlements. Together they turn retrieval into something with a pass or fail condition.

Test categories that repay the effort include retrieval recall against known required evidence; ranking, asserting that the authoritative source outranks the secondary one; permission tests, running the same query under several identities and asserting different result sets; memory tests, asserting that a stored fact is retrieved when relevant and not when irrelevant, and that a corrected fact supersedes the original; freshness tests, asserting that superseded documents are excluded; assembly tests, asserting block presence, ordering, and per-category token allocation; tool selection tests, asserting the correct tool and parameters for a given request shape; multi-turn state tests, asserting that entity references resolve correctly across turns including after a topic switch; tenant isolation tests, deliberately probing for another tenant's known content; injection resistance tests, using a corpus seeded with instruction-bearing documents; overflow tests, asserting graceful degradation when candidate material exceeds the budget; and conflict tests, using a corpus with deliberately contradictory sources and asserting escalation or abstention rather than a confident pick.

Most of these run without invoking the model, which means they execute in seconds and can gate deployment the way any other test suite does. The subset that does require the model can run against fixed context fixtures, which removes retrieval variance from model evaluation and makes the results interpretable.

Metrics That Describe Context

None of the following has a universally standardized definition. Each is a way of asking a specific question, and the definitions have to be fixed locally before the numbers mean anything.

Retrieval recall asks what fraction of required evidence the pipeline surfaced. It is the ceiling on achievable accuracy. Retrieval precision asks what fraction of what was surfaced was actually needed. It bounds waste and noise. Context relevance asks, of the material finally assembled rather than merely retrieved, how much bears on the question. It differs from retrieval precision because filtering and reranking sit between them. Context utilization asks how much of the supplied context the answer actually drew on, usually approximated through citation analysis. Persistently low utilization suggests the budget is being spent on material the model ignores. Source authority distribution asks what proportion of assembled context came from approved versus unofficial sources, which is a governance signal rather than a quality signal. Freshness asks the age distribution of assembled material relative to effective dates. Permission accuracy asks whether authorization decisions matched what the source systems would have allowed, measured by comparing retrieval scope against an independent entitlement check. Duplicate context rate asks what share of assembled tokens are near-duplicates. Context token efficiency relates answer quality to tokens consumed, which is the metric that reveals whether adding retrieval depth is still paying for itself. Memory precision and recall ask whether stored memories are correct and whether relevant ones are retrieved when needed. Tool selection accuracy asks how often the correct tool with correct parameters was chosen. Citation correctness asks whether cited sources actually contain the cited claims, which is checkable automatically against the assembled context and is one of the more useful production guardrails available.

The value of tracking these is less in any individual number than in the fact that they decompose a single opaque quality score into components that different teams can own and act on.

The Economics of Context

Cost analysis in AI systems tends to focus on the model, because the model has a published price per token. That price is frequently a minority of the total.

Context imposes costs across the stack: embedding generation during ingestion and at query time, vector storage which for large corpora with high-dimensional embeddings is substantial, reranking which is a model inference cost of its own on every request, database queries against operational systems which may compete with production workloads, tool and API calls including those to metered third parties, the compute for ingestion and normalization pipelines, and the storage and processing of observability data which for full context tracing is not trivial.

There is also a latency budget, which converts into cost indirectly through user abandonment and through the infrastructure required to meet it.

This changes how architectural options should be compared. Consider two designs for the same task. The first uses cheap retrieval, a wide net of retrieved chunks, and a large expensive model to sort out the noise. The second invests in query transformation, hybrid retrieval, and a reranking pass, producing a tight context that a smaller model can handle. The first has lower context infrastructure cost and higher per-request model cost. The second inverts that, and typically has better latency at the model step and worse latency at the retrieval step.

Which is cheaper depends on volume, on the price gap between the models under consideration, on the corpus size, and on how much accuracy each design achieves, since a wrong answer has a cost too, in human review time if it is caught and in consequences if it is not. The comparison has to be made end to end on the specific workload. Optimizing model spend while treating retrieval infrastructure as fixed overhead produces conclusions that do not survive contact with the full bill.

Smaller Models, Sharper Context

A smaller model with precise, well-structured, authoritative context can outperform a larger model with poor context on narrow enterprise tasks. This is not a general claim, and it is worth being specific about when it holds.

It tends to hold when the task scope is narrow and well-defined, when the required knowledge is fully present in the supplied context rather than needing to be inferred, when the reasoning required is shallow, involving applying a stated rule to stated facts rather than synthesizing across many considerations, and when the output format is constrained. Under those conditions the model's contribution is comprehension and formatting, both of which smaller models do adequately.

It tends not to hold when the task requires multi-step reasoning over the supplied material, when the context contains ambiguity or conflict that must be arbitrated, when the required output involves judgment rather than rule application, when instructions are numerous and interacting, or when the context is long enough that reliable attention across it becomes the binding constraint.

The credit question sits near the boundary. Determining whether a stated utilization figure exceeds a stated threshold is well within a small model's range. Weighing an unusual payment pattern against a policy exception in a jurisdiction with an additional affordability requirement is not.

The broader point this supports is that model capability and system capability are different quantities. Upgrading the model raises a ceiling. Improving the context raises the floor, and for most enterprise workloads the floor is what determines whether the system is usable.

Routing Context, Not Only Models

Model routing is now a common pattern: classify the request, send simple ones to a cheap model and hard ones to an expensive one. The same logic applies to context, and it is discussed far less.

Different question types require fundamentally different information environments. A support question needs the knowledge base and the customer's ticket history, with recency weighted heavily. A financial decision needs structured data from systems of record plus the approved policy, with freshness treated as a hard requirement rather than a ranking signal. A technical incident needs logs, traces, runbooks, and recent deployment history, with a time window rather than a similarity threshold as the primary filter. A contract question needs clause-level retrieval with definitional context and version control.

Running all of these through one retrieval configuration produces a system that is mediocre at each. Context routing makes the strategy a function of intent:

Intent → Context Strategy → Sources → Retrieval Method → Budget → Model

For each intent class, the routing table specifies which sources participate, which retrieval method applies to each, what filters are mandatory, how the token budget is allocated across categories, what freshness requirement applies, and which model handles the assembled result. A credit assessment might specify: policy index with version filtering and reranking at 3,000 tokens, core banking API with live query at 800 tokens, case history at 400 tokens, freshness requirement of under five minutes on financial fields, abstention required if any mandatory source is unavailable, routed to the larger model.

This produces an orchestration layer that sits above both retrieval and inference and that encodes a substantial amount of organizational knowledge about which questions depend on which systems. That knowledge is currently held informally by the engineers who built the integrations, and writing it down as configuration is most of the value.

Context Policies

Once routing exists, a further step becomes available: making context selection policy-driven rather than entirely probabilistic.

Retrieval by similarity is a statistical process that produces a ranked list with no guarantees. For many questions that is fine. For questions with regulatory or financial consequence, it is reasonable to impose deterministic requirements on what must and must not be in the context.

Policies that organizations plausibly want to state explicitly include: financial calculations must use structured data from a system of record and may not rely on figures quoted in documents; legal and compliance answers may only cite approved, version-controlled documents; customer-specific queries must be scoped to the requesting user's tenant and entitlements with no exceptions; decisions above a defined risk threshold must include provenance for every material fact and must escalate if any is missing; superseded policy versions may not be used for prospective decisions and may only be retrieved for explicitly historical questions; and any answer that would rest on a source below a defined authority level must be marked as informational rather than authoritative.

Expressing these as configuration rather than as prompt instructions has two advantages. They can be enforced in the pipeline, where enforcement is deterministic, rather than requested of the model, where compliance is probabilistic. And they can be audited, which for regulated processes is often the actual requirement.

Context as Shared Infrastructure

At a certain point, multiple AI applications inside one organization begin rebuilding the same components. Each connects to the same document repositories, writes its own ingestion pipeline, maintains its own embeddings, implements its own permission filtering, and builds its own memory store. The duplicated effort is significant, and the duplicated permission logic is a genuine risk, since it means several independent implementations of access control over the same sensitive material.

The components that plausibly warrant consolidation include document ingestion and connector management, normalization and metadata extraction, entity resolution, embedding generation, search and reranking services, the knowledge graph, memory services, permission filtering, context assembly APIs, observability and tracing, and evaluation harnesses.

Consolidating these is not automatically correct. It centralizes a dependency, introduces a team whose roadmap constrains every consuming application, adds a network hop to a latency-sensitive path, and risks producing generic abstractions that fit no use case well. The threshold is usually a function of how many AI applications exist and how much they share in terms of underlying data. Two applications over disjoint data should not share infrastructure. Eight applications over the same document corpus and the same customer records almost certainly should share at least ingestion, permission filtering, and observability.

The strongest argument for consolidation is not efficiency. It is that permission filtering and provenance are correctness-critical and should have one implementation that is reviewed carefully, rather than eight that were each written under deadline.

The Context API and Its Limits

The abstraction that follows from consolidation is a context service. Rather than every application wiring itself to the CRM, the ERP, the document store, the vector index, the graph, the memory store, and the identity provider, the organization exposes a call:

context.get(
    user,           # identity and entitlements, not self-asserted
    task,           # intent class, which selects the strategy
    entities,       # resolved identifiers
    freshness,      # maximum acceptable staleness per category
    token_budget    # total and per-category allocation
) → ContextPackage {
    blocks[],       # labeled, ordered, with provenance
    omissions[],    # what was excluded and why
    unavailable[],  # required sources that failed
    trace_id
}

The appeal is clear. Permission logic exists once. Provenance is uniform. Observability is uniform. New applications start with working retrieval rather than a three-month integration project. Context policies are enforced centrally.

The difficulties are equally clear and should be stated honestly. A generic interface tends to serve the average case and to obstruct the specific one, and the specific one is usually where the value is; a credit assessment's context requirements differ from an incident response system's in ways that resist a common parameterization. Each additional hop adds latency to a path that is already latency-constrained. Ownership becomes contested, since the context team owns the mechanism while domain teams own the meaning, and defects tend to fall between them. Centralization creates a single point of failure for every AI application simultaneously. And the abstraction can become a platform dependency that is difficult to reverse once a dozen applications depend on its particular semantics.

A workable middle position is to consolidate the layers where uniformity is genuinely valuable, meaning ingestion, permission filtering, entity resolution, and observability, while leaving retrieval strategy and context assembly to the applications, which understand their own decisions. The shared layer provides authorized, normalized, traceable access to information. The application decides what to do with it.

Who Owns Context Quality

Asking which team owns context quality produces an unsatisfying answer: no single team can, because the layers have genuinely different owners and the boundaries are real rather than political.

Domain teams own meaning. Only the risk function can say which policy is authoritative, what a field means, which exceptions exist, and which sources are trustworthy. No amount of engineering competence substitutes for this, and systems built without it encode guesses.

Data and platform teams own pipelines and infrastructure: ingestion reliability, index freshness, connector health, embedding services, retrieval latency, and the operational characteristics everything else depends on.

Security owns access policy and its enforcement, including permission synchronization between source systems and indexes, tenant isolation, and classification handling.

AI engineering owns context assembly, retrieval strategy, prompt and instruction design, model interaction, and evaluation harnesses.

Knowledge management, where it exists, owns content lifecycle: authorship standards, versioning, supersession, ownership assignment, and retirement of stale material.

The organizational implication is that context quality problems are frequently misrouted. A wrong answer caused by a superseded policy that was never marked as superseded is not an AI engineering defect, and assigning it there produces workarounds rather than fixes. Systems that work tend to have an explicit map from failure category to owning team, derived from the failure taxonomy, plus a forum where cross-cutting defects can be assigned rather than debated.

Old Data Problems, New Visibility

Enterprise data quality problems have existed for decades and have been tolerable because humans compensated for them. An analyst who retrieves a duplicate customer record notices. An employee who finds an undated policy asks someone. A person who encounters two contradictory procedures escalates.

Retrieval-based systems remove that compensation layer. Duplicate records produce split evidence and inconsistent aggregates. Missing metadata makes version and authority filtering impossible, which means superseded content stays in the retrieval pool. Unowned documents cannot be corrected, because nobody is responsible for them. Outdated policies rank as well as current ones. Broken permission models in source systems propagate into indexes. Conflicting systems of record produce conflicting facts with no resolution rule. Knowledge that exists only in people's heads is simply absent.

The model does not repair any of this. It reads what it is given and reasons over it, which means data defects that were previously absorbed by human judgment now surface as confident, fluent, wrong output. In practice, deploying a retrieval system over a corpus is one of the more effective content audits an organization can run, because it exercises every document against real questions and makes the gaps visible.

This is why context engineering sits at the intersection of several existing disciplines rather than being an entirely new one. It draws on data engineering for pipelines and quality, knowledge management for lifecycle and ownership, information architecture for structure and metadata, search engineering for retrieval and ranking, and security engineering for authorization and isolation. The novel part is the assembly and evaluation layer that sits on top. The foundations are decades old, and organizations that invested in them find this transition considerably easier.

The Question, Answered Properly

Return to the original question with everything above in place.

Can we offer this customer a larger credit limit?

The prompt has not changed. What has changed is what surrounds it.

Identity establishes that the requester is a credit risk analyst in the EU retail tenant, resolved from the authentication layer rather than from anything typed in the message.

Authorization determines that this analyst may see full exposure and payment history for accounts in their tenant, and constrains retrieval to that scope before any search runs.

Entity resolution converts "this customer" into a specific account and customer identifier, with disambiguation if the reference is ambiguous rather than a guess.

Customer state is fetched live from the system of record: balance, current limit, utilization, account age, exposure across products, account status, each carrying the timestamp at which it was read.

Financial history is delivered as computed aggregates rather than raw transactions: late payment counts by severity band over the trailing twelve months, not thirty thousand tokens of statement lines.

Policy is the current approved credit policy, version-filtered to exclude superseded editions, retrieved at clause granularity with the section path and definitions intact, carrying its effective date, owner, and approval status.

Regional constraints are retrieved because the customer's region attaches to a supplement that modifies the global policy, a relationship found by traversal rather than by similarity.

Prior decisions surface a review from earlier in the year that was declined on utilization grounds, so the system does not contradict its own recent judgment without acknowledging it.

Tools are exposed as a filtered set relevant to this intent, with one of them reported as unavailable rather than silently omitted.

Provenance attaches to every material fact, so the eventual recommendation can be defended clause by clause.

Freshness is explicit per fact, so a balance read four seconds ago and a policy effective five months ago are distinguishable.

Memory contributes the task state if this review was started previously, and the analyst's stated working preferences.

Only then does the model reason. And in the traced example above, its correct output was not a recommendation at all. It was an abstention, because a required input was unavailable and the context said so explicitly rather than leaving a gap the model would have papered over.

RAW QUESTION
      ↓
CONTEXT CONSTRUCTION
      ↓
GROUNDED REASONING
      ↓
VALIDATED OUTPUT

The improvement between the empty prompt at the start of this article and the system just described did not come from adding another sentence to the instruction. The instruction is nearly identical. It came from constructing the information environment in which the question is answerable at all.

What the Model Cannot Bring With It

Foundation models will keep improving. Context windows will keep growing. Retrieval methods will keep getting better, agents will handle longer and more complex tasks, and a number of the engineering problems described in this article will become easier or disappear into infrastructure that nobody has to think about.

None of that changes the structural position. The information that determines whether an enterprise AI system produces a correct answer is information the model cannot contain, because it did not exist at training time, because it is private, because it changes continuously, or because it is subject to access rules that vary by requester. Company knowledge, customer state, operational state, entity relationships, decision history, current rules, permissions, and workflows all live outside the model, and a better model does not bring them any closer.

This is also where competitive difference actually accumulates. Every organization can access approximately the same frontier models within a short window of each other. What differs is the quality of the connection between those models and the organization's own reality: whether the right policy version is retrievable, whether entity resolution is reliable, whether permissions are enforced at the right layer, whether memory is accurate, whether provenance survives to the point of decision, and whether the system knows when to decline.

Models and context infrastructure are complementary, and neither substitutes for the other. A weak model over excellent context will reason poorly about good evidence. A strong model over poor context will reason excellently about nothing.

The practical difference worth ending on is this: a model that can reason is a general capability that anyone can buy, while a system that knows what it should be reasoning about is something an organization has to build.

Closing takeaway: Prompt engineering determines how an AI system is instructed; context engineering determines what it is able to know. As enterprise AI moves from generic writing into decisions, operations, and agents, the binding constraint shifts from the wording of the instruction to the construction of the information environment behind it, which is a problem of retrieval, structured data, memory, provenance, freshness, permissions, and assembly rather than of phrasing. The organizations that treat that environment as an architecture, with its own budget, evaluation, observability, and ownership, will get more from the same models than those that treat it as plumbing.