Prompting, RAG, or Fine-Tuning: The Production Decision Framework Most Teams Skip

A prompting demo, a RAG demo, and a fine-tuned model demo can produce the exact same answer to the exact same question. That is the trap. When three architecturally unrelated systems generate identical output in a fifteen-minute review, the people in that review reasonably conclude the choice between them is a matter of taste, or team preference, or which tutorial the engineer read first. It is not. Each of those three answers was produced by a system with a different relationship to time, a different cost curve as usage grows, a different way of failing, and a different answer to the question "what do we do when the vendor changes the model underneath us." None of that is visible in a demo. All of it becomes visible in production, usually between month six and month eighteen, usually at the worst possible moment — during a compliance review, a cost audit, or a scramble after a model deprecation notice.

This is a decision about how an AI feature will learn what it needs to know or how it needs to behave, and it gets made, in most organizations, the way a lot of early technical decisions get made: by whoever prototyped it first, using whichever approach that person already knew, validated against whichever handful of test queries happened to be lying around. The prototype works. It ships. Nobody revisits the decision until something forces the issue — a knowledge base that changed faster than anyone expected, an inference bill that grew faster than revenue, an evaluation team that cannot explain why the model's answers changed after last week's retrain, or a vendor deprecating exactly the fine-tuned checkpoint the product depends on. At that point the team is not improving an architecture. It is replacing one, under a deadline, with the original decision-makers usually still trying to figure out what assumption failed.

This article is a framework for making that decision on purpose, the first time, using the properties that actually predict whether an approach survives contact with a real product at real scale — not a walkthrough of what retrieval-augmented generation or fine-tuning are, which the official documentation from every major model provider already covers well. The goal is to give a technical decision-maker a way to reason about the choice along the axes that matter: how the approach behaves as the underlying data changes, what it costs at three different usage scales rather than one, where the latency actually comes from, how you test and evaluate it, what it exposes and to whom, what happens when the base model changes out from under you, who has to own it operationally, and when combining approaches is the right call rather than an excuse to avoid the decision.

What Each Approach Actually Changes

Before comparing them on any dimension, it helps to be precise about what each approach is mechanically doing, because the differences in cost, freshness, and risk later in this article all trace back to this.

Prompting changes nothing about the model. It changes what you send the model at inference time — instructions, a persona, formatting rules, few-shot examples, and whatever task-specific content you include in the context window. The model's weights are untouched; every property of the underlying model (its training cutoff, its general reasoning ability, its safety behavior) carries through unchanged. What you're doing is steering a fixed capability with a carefully constructed input. The practical ceiling on prompting is the context window: modern frontier models now offer very large windows — Anthropic's Claude Sonnet 5 and Claude Opus 5 support a 1 million-token context window, while Claude Haiku 4.5 supports 200,000 tokens (Anthropic, Claude models overview) — but a larger window is not a free resource. Every token in the context is billed, adds to latency, and research on long-context attention has repeatedly shown that relevance to a query does not scale linearly with how much you stuff into the window; models attend unevenly across a long context, and instructions or facts placed in the middle of a very long prompt are more likely to be under-weighted than the same content placed near the beginning or end. Prompting also inherits a subtler cost: the entire context is billed on every single call unless you use a caching mechanism, so a long, carefully engineered system prompt becomes a recurring cost multiplied by every request, not a one-time investment.

Retrieval-augmented generation (RAG) does not touch model weights either — it is, mechanically, still a form of prompting — but it automates and scales the process of deciding what goes into the context window. A RAG pipeline embeds your source documents into a vector space using an embedding model, storing each vector alongside the source text (typically chunked into passages of a few hundred tokens, since embedding an entire long document into a single vector loses too much fine-grained meaning to be useful for retrieval). At query time, the pipeline embeds the incoming question with the same embedding model and searches for the stored vectors most similar to it, usually by cosine similarity — nearby vectors representing semantically related text, distant vectors representing unrelated text. OpenAI describes this mechanism directly in its own embeddings guide: "the distance between two vectors measures their relatedness. Small distances suggest high relatedness and large distances suggest low relatedness" (OpenAI, Embeddings guide). Anthropic's own applied research on this problem is worth citing directly, because it corrects a common oversimplification: pure embedding similarity search alone is not the most reliable retrieval mechanism. Anthropic's Contextual Retrieval technique combines embedding-based semantic search with a traditional lexical method (BM25, a refinement of TF-IDF that accounts for document length and prevents common words from dominating relevance scores) in a hybrid search, and separately prepends short, chunk-specific explanatory context (generated by Claude itself, typically 50–100 tokens) to each chunk before it is embedded or indexed, so that a chunk pulled out of its source document still carries enough context to be matched correctly. Anthropic reports, in its own published engineering research, that contextual embeddings alone reduced retrieval failure rate by 35% (from 5.7% to 3.7%), combining contextual embeddings with contextual BM25 reduced it by 49% (to 2.9%), and adding a reranking step on top reduced it by 67% (to 1.9%) (Anthropic, Contextual Retrieval). Those are Anthropic-reported figures from Anthropic's own evaluation set, not independently replicated industry benchmarks, and they should be read as evidence that naive single-method retrieval leaves real accuracy on the table — not as a universal number that will reproduce on a different corpus. The retrieved passages are then assembled into a prompt alongside the user's question and sent to the generation model, which produces an answer grounded in whatever was retrieved. The generation model's weights are exactly as untouched as in pure prompting. What RAG buys you is a mechanism for keeping the content injected into that prompt current and relevant without a human manually curating a static prompt.

Fine-tuning is the only one of the three that changes the model itself. In full fine-tuning, you continue training a pretrained model's weights on a labeled dataset of your own input-output examples, adjusting every parameter in the network to shift its behavior toward your data. This is effective but expensive and carries real risk: adjusting all parameters on a comparatively small, narrow dataset can degrade capabilities the model had before, a phenomenon studied directly in the machine learning literature as catastrophic forgetting. An empirical study of continual instruction fine-tuning across models from 1 billion to 7 billion parameters found that forgetting is a real, measurable phenomenon during fine-tuning, and — counterintuitively — that larger models within that range often forgot more of their prior capability than smaller ones, plausibly because they had more capability to lose in the first place; the same study found that decoder-only architectures retained more of their original knowledge than encoder-decoder architectures under the same fine-tuning regime (An Empirical Study of Catastrophic Forgetting in Large Language Models During Continual Fine-tuning, arXiv:2308.08747). Full fine-tuning of a large model is also resource-intensive enough that most teams outside a handful of frontier labs cannot do it economically, which is precisely the problem that parameter-efficient fine-tuning methods were built to solve. The most widely adopted of these is LoRA (Low-Rank Adaptation), introduced by Hu et al. in 2021: instead of updating the full weight matrices of the network, LoRA freezes the pretrained weights entirely and injects small, trainable low-rank decomposition matrices into specific layers (commonly the attention layers) of the transformer architecture. The original paper reports that this can reduce the number of trainable parameters by up to 10,000 times and cut GPU memory requirements by roughly 3 times relative to full fine-tuning of GPT-3 at 175 billion parameters, while performing on par with or better than full fine-tuning on several benchmark tasks, with no additional inference latency once the adapted weights are merged or applied (Hu et al., LoRA: Low-Rank Adaptation of Large Language Models, arXiv:2106.09685). This is why LoRA and related PEFT techniques have become the default entry point for teams that want fine-tuning's behavioral control without full fine-tuning's cost and forgetting risk — though PEFT does not eliminate forgetting entirely; more recent research specifically on low-rank adaptation methods has continued to document and try to mitigate forgetting within LoRA-style fine-tuning itself, rather than treating it as a solved problem (see, for instance, ongoing 2026 research such as "Mitigating Forgetting in Low Rank Adaptation," arXiv:2512.17720, and "On Catastrophic Forgetting in Low-Rank Decomposition-Based Parameter-Efficient Fine-Tuning," arXiv:2603.09684).

One more mechanical distinction matters more than most comparisons acknowledge: fine-tuning, across essentially every major provider's own documentation, is positioned as a tool for shaping behavior — tone, format, instruction-following consistency, classification accuracy, tool-calling patterns — not as a reliable way to inject new, verifiable knowledge. OpenAI's own supervised fine-tuning guidance recommends the technique for classification tasks, nuanced translation, generating content in a specific format, and correcting instruction-following failures — and explicitly instructs teams to establish evaluation metrics before investing in fine-tuning at all, starting with as few as 50 well-crafted examples and only scaling up if that produces a measurable improvement (OpenAI, Supervised fine-tuning guide). Microsoft's Azure OpenAI fine-tuning guidance similarly frames fine-tuning as most effective for style and tone consistency, structured output formats, and reducing prompt engineering overhead when few-shot examples become unwieldy — while noting that retrieval-based methods remain the recommended complement for grounding a model in facts (Microsoft Learn, When to use Azure OpenAI fine-tuning). This distinction — behavior versus knowledge — is the single most useful mental model in this entire article, and most of the dimension-by-dimension analysis below is really a set of consequences that fall out of it.

The table below summarizes the mechanical differences before the framework builds on them.

Freshness: How Each Approach Ages

This is the dimension most decision-makers underweight at prototype stage and most regret two quarters later, because it is invisible until the underlying facts actually change.

Prompting and RAG both source their factual content from outside the model at the moment of the call. If the facts change, you update the source — a document, a database row, a retrieval index — and the very next call reflects the update. RAG in particular is designed around this property: because retrieval happens against a live index rather than baked-in weights, correcting a piece of information is an indexing operation, typically completed in minutes, not a model change. Fine-tuning does not have this property by construction. Whatever the model learned during training is fixed in its weights until you run another training job. There is no way to "patch" a single fact in a fine-tuned model the way you can update a single row in a vector database or a single paragraph in a system prompt. If your domain knowledge changes on any cadence faster than your retraining cadence, fine-tuning will systematically serve stale answers with exactly the same confident tone it uses for correct ones — which is worse than an system that visibly doesn't know something, because a confidently wrong answer doesn't prompt anyone to double-check it.

Consider a hypothetical, illustrative scenario that is common enough in regulated industries to be worth walking through in detail. A mid-sized fintech company builds an internal assistant for its compliance and support teams to answer questions about which transaction limits, disclosure requirements, and verification rules apply to a given customer segment and jurisdiction. The team fine-tunes a smaller open-weight model on a curated dataset of several hundred question-answer pairs built from the current compliance rulebook, reasoning that a fine-tuned model will be faster and cheaper per query than a prompting or RAG approach that has to carry the rulebook around in context on every call. The demo is excellent: the model answers rulebook questions fluently, in the company's preferred format, faster than a competing RAG prototype that had to wait on a retrieval round-trip. It ships.

The hidden assumption is that the rulebook is stable enough, over the model's useful lifetime, to be worth baking into weights at all. It is not. Regulatory transaction limits and disclosure thresholds in most jurisdictions get revised on a rolling basis — sometimes quarterly, sometimes with less notice than that, following regulator guidance updates. Three months after launch, a threshold changes in one jurisdiction. The fine-tuned model, having learned the old threshold as a fact rather than as a retrievable, sourced value, continues to state it with full fluency and no indication that anything is amiss, because from the model's perspective nothing is amiss — it is doing exactly what it was trained to do. A compliance analyst uses the assistant to answer a customer inquiry using the outdated limit. The error surfaces weeks later, during an internal audit, at which point the team has to determine how many other answers over that window were affected, retrain the model on the corrected rulebook, and — the part that actually costs the most time — build a process to catch this category of error before it reaches a customer again, because the original architecture had no mechanism for surfacing staleness at all.

The decision point the team faces after that audit is not "should we retrain more often." Retraining on every regulatory update, for every jurisdiction, indefinitely, is not an operating model; it's a standing commitment nobody sized correctly at the start. The better architecture separates the two things fine-tuning had blurred together: the authoritative, versioned rulebook belongs in a retrieval system that always points at the current version — correcting a threshold becomes a same-day content update, not a training run — while a lightweight fine-tune (or, in many cases, no fine-tune at all) is reserved only for the parts of the task that are genuinely about consistent behavior: response format, tone, and the specific phrasing compliance requires for certain disclosures. Knowledge goes in the retrieval layer, because knowledge changes; behavior goes in the model, because behavior is comparatively stable.

The Cost Curve Nobody Draws Before Prototyping

Cost comparisons between these approaches usually get reduced to a single sentence — "fine-tuning is expensive up front but cheap to run; RAG has ongoing infrastructure costs; prompting is free to start but scales badly" — which is directionally true and almost useless for an actual budgeting decision, because none of it says anything about your query volume. The right way to reason about this is as a fixed-cost-versus-variable-cost trade-off, the same framework you would apply to a build-versus-buy decision anywhere else in the business, and it produces a genuinely different recommendation depending on scale.

To make this concrete, walk through an illustrative — explicitly hypothetical — scenario using real, currently published per-token prices as the arithmetic inputs. Assume a product feature that answers customer questions using domain reference material, comparing three architectures for the generation step: (A) prompting with the full reference material included in context on every call (roughly 15,000 input tokens plus a 300-token question), (B) RAG retrieving only the relevant passages (roughly 2,000 input tokens plus the question), and (C) a lightly fine-tuned small model that needs only brief instructions and the question (roughly 600 input tokens), which also carries a training cost of $200 per retraining cycle, assumed to run once per month to keep pace with product changes. All three use the same generation model for the arithmetic to isolate the effect of context size alone; assume an average response length of 300 output tokens in every case. Using Claude Haiku 4.5's published API pricing of $1 per million input tokens and $5 per million output tokens (Anthropic, Claude models overview), and OpenAI's published text-embedding-3-small pricing, which the company's own guide estimates at roughly 62,500 pages of text embedded per dollar at approximately 800 tokens per page — an effective rate on the order of $0.02 per million tokens, making embedding cost immaterial to this comparison (OpenAI, Embeddings guide) — the per-query token cost for each approach works out as follows.

Approach A (full context in every prompt): 15,300 input tokens × $1/M + 300 output tokens × $5/M ≈ $0.0168 per query. Approach B (RAG, retrieved context only): 2,300 input tokens × $1/M + 300 output tokens × $5/M ≈ $0.0038 per query. Approach C (fine-tuned, short prompt): 600 input tokens × $1/M + 300 output tokens × $5/M ≈ $0.0021 per query, plus $200 amortized monthly training cost.

The illustrative chart below shows what that arithmetic produces at three different monthly query volumes. These are hypothetical usage scenarios, not measured customer data or vendor benchmarks — only the underlying per-token prices are real and sourced as noted above.

Chart 1 — Illustrative monthly generation cost by approach at three usage scales

Cost ($, log scale)                          Approach A (full-context prompting)
1,000,000 |                                        ●
          |                                    ╱
  100,000 |                              ●   ╱
          |                          ╱   ● Approach C (fine-tuned)
   10,000 |                    ●  ╱   ●
          |              ●  ╱  ● Approach B (RAG)
    1,000 |         ●  ╱
          |      ●
      100 |   ●
          +---------------------------------------------------
            10K queries/mo   1M queries/mo      50M queries/mo

The pattern this reveals is the entire point of drawing it: at low volume, the fine-tuned approach is the most expensive of the three, because its fixed monthly retraining cost dominates a small number of queries — a fact that contradicts the common assumption that fine-tuning is simply "the cheap option once you're past the up-front investment." At moderate volume, RAG's marginal cost advantage over full-context prompting starts to compound. At very high volume, the fine-tuned approach's lower per-query marginal cost finally overtakes both, because its fixed cost has been amortized across enough queries to matter less than its lower ongoing token spend. Full-context prompting is the worst-scaling option at every volume tested here, because its cost is dominated by a large context payload repeated on every single call with no retrieval step to shrink it. In practice, prompt caching (offered by both OpenAI and Anthropic, which reduces the cost of re-sending an unchanged prefix of a prompt) narrows this gap for approach A, but it does not eliminate the structural difference: a cached prefix is still typically billed at a reduced, non-zero rate, and caching does nothing to reduce the number of output tokens generated or the latency of processing a long prefill, which is the subject of the next section.

A second, realistic illustrative example makes the scale dimension concrete rather than abstract. A mid-market e-commerce marketplace with a catalog of 40,000 SKUs builds an AI shopping assistant that answers product questions by drawing on specification sheets, return policies, and inventory status. Early in development, an engineer prototypes the feature by including the top few hundred most-asked-about products' specification sheets directly in a long system prompt — the prompting approach — because it is the fastest thing to build and it demos well against the products the team happened to test with. The hidden assumption is that "the top few hundred products" is a stable, representative sample of what customers will actually ask about once the feature is live to the full catalog. It is not: real customer questions span the long tail of the catalog, seasonal top-sellers rotate, and inventory status changes by the hour. Within weeks of a full launch, the team faces an unpleasant choice — either keep growing the system prompt to cover more of the catalog, which increases token cost and latency on every single query regardless of which product the customer is actually asking about, or accept that most questions about products outside the curated list get answered poorly or not at all. The consequence is a cost curve that grows with the breadth of the catalog they try to support in-prompt, not with actual query volume, which is the wrong variable to be scaling against. The better approach, once the pattern is recognized, is RAG over the full product catalog and live inventory feed: cost per query becomes a function of how much relevant material a given question actually needs, not how much the team decided to preemptively stuff into every call, and inventory status updates propagate to the assistant's answers without touching the prompt or the model at all.

Latency: Where the Milliseconds Come From

Latency is the dimension most likely to be measured wrong, because teams tend to benchmark "time to first token" or "total response time" as a single number attributable to "the model," when in a RAG or long-context pipeline the model's own generation time is only one stage among several, and not always the largest one.

For prompting, the dominant latency cost as context grows is prefill time — the time the model spends processing the input tokens before it can begin generating output. Prefill scales with the number of input tokens processed, so a system prompt carrying 15,000 tokens of reference material adds meaningfully to response time on every single call, independent of how long or short the useful answer turns out to be. This is a direct, structural cost of the "just put everything in the context" approach that a demo using a handful of test queries against a short prompt will never reveal, because the effect only becomes visible once the context is large enough for prefill time to be a noticeable fraction of total latency.

For RAG, latency has to account for every stage of the pipeline, not just generation: embedding the incoming query, searching the vector index (and, in a hybrid setup like Anthropic's Contextual Retrieval, also running the lexical/BM25 search and combining results), and — if a reranking step is used, which the Anthropic research cited earlier found meaningfully improves retrieval accuracy — scoring and reordering the retrieved candidates before assembly into a final prompt. Each of these adds a discrete, sequential delay before generation even begins. The advantage RAG has over full-context prompting is that the prefill stage that follows is working with a much smaller, curated context, which is faster to process than a large undifferentiated one — but that advantage is partially offset by the retrieval stages that precede it, and a poorly optimized retrieval pipeline (an unindexed or oversized vector store, a network round-trip to an external vector database, an unnecessary reranking pass on every query regardless of whether the initial retrieval was already confident) can erase the advantage entirely.

For a fine-tuned model with a short, purpose-built prompt, latency is close to the theoretical floor for the underlying model: minimal prefill, and generation time driven mostly by output length. This is the concrete reason fine-tuning remains attractive for latency-sensitive, high-volume features even in an era of very capable prompting and RAG — not because the model is inherently faster, but because the amount of work the model has to do per call is smaller by construction, and because a fine-tuned small model can sometimes replace a larger general-purpose model for a narrow task, gaining speed from the model size itself as well as the prompt size.

The table below is an illustrative order-of-magnitude breakdown, not a measured benchmark from any specific vendor or deployment — actual latency depends heavily on model size, hosting infrastructure, network conditions, and implementation quality, and these figures should be read only as a directional illustration of where delay accumulates in each pipeline shape.

Chart 2 — Illustrative relative latency contribution by pipeline stage (not a measured benchmark)

The practical implication for a latency-sensitive feature — a live customer support widget, a real-time in-app assistant, anything where a user is actively waiting on a response rather than checking back later — is that both the size of what you put in context and the number of sequential stages before generation begins are levers you control, independent of which model you choose. A team chasing lower latency by upgrading to a faster model while leaving a 15,000-token context payload untouched, or leaving an unnecessary reranking pass in a pipeline where retrieval confidence is already high, is optimizing the wrong variable.

Testing and Evaluation Are Not the Same Problem for All Three

This is worth separating clearly from the broader discipline of managing what goes into a single call's context window — that is a distinct concern, addressed elsewhere, about tuning and structuring the content of one interaction. The evaluation question here is upstream of that: given that you've chosen prompting, RAG, or fine-tuning as your customization mechanism, what does "testing whether it works" actually require, and where does each approach hide its failures from a naive test suite?

Prompting is the easiest of the three to iterate on and the easiest to test superficially, which is itself a risk. Because there is no training step, you can change a prompt and get an answer in seconds, which encourages a workflow of eyeballing a handful of outputs and shipping when they look right. The failure mode this produces is that a single prompt change is a global behavior change — adjusting a system prompt to fix one failing case can silently shift behavior on cases that were previously passing, and without a maintained regression suite of representative queries run automatically on every prompt change, that shift goes undetected until a user reports it. OpenAI's own fine-tuning guidance makes a point that applies equally to prompting, even though it's written in the context of deciding whether to fine-tune at all: establish evaluation metrics before investing further in any customization approach, not after (OpenAI, Supervised fine-tuning guide). Teams that skip this step for prompting because "it's just a prompt, how wrong can it go" are the same teams that later can't explain why a prompt tweak six weeks ago quietly broke a use case nobody was watching.

RAG introduces a testing problem that a lot of teams don't realize is two separate problems until something goes wrong: retrieval quality and generation quality are independently testable, independently failable, and a generation-only evaluation can mask a retrieval failure entirely. If your test set happens to ask questions the underlying model already knew the answer to from its own pretraining, a broken retriever — one returning irrelevant or outdated chunks — can still produce a correct-looking final answer, because the generation model didn't actually need the retrieved context to get it right. This is precisely the scenario that makes a RAG evaluation misleading if it only measures final-answer correctness: the system can look like it's working while its actual retrieval mechanism is quietly broken, and the failure only surfaces later, on a question the model genuinely couldn't answer without correct retrieval. A properly designed RAG evaluation needs a labeled set of query-to-relevant-passage mappings to test retrieval precision and recall directly, separate from an end-to-end answer-quality evaluation, and both need to be run on a schedule, not just once at launch — because as the underlying corpus grows and changes, retrieval quality on old queries can silently degrade even if nothing about the pipeline code changed.

Fine-tuning has the least forgiving evaluation profile of the three, for a structural reason: every new fine-tuned checkpoint is, for evaluation purposes, effectively a new model, not a small change layered on a known-good baseline. A prompt change is a diff you can read; a retrained model is a new set of weights whose behavior across your entire task surface — not just the narrow slice of examples you fine-tuned on — needs to be re-verified. This is exactly why OpenAI's own documentation emphasizes checkpointing at every training epoch and comparative evaluation across checkpoints rather than assuming the final epoch is automatically the best one: without a held-out evaluation set distinct from the training data, teams cannot detect overfitting to that narrow training distribution or a regression in general capability caused by catastrophic forgetting, and the model's own confident, fluent output gives no signal that either has happened. A fine-tuned model that has quietly lost some general reasoning ability while gaining fluency on its narrow training task will not announce that trade-off; it will simply perform worse on inputs slightly outside the training distribution, in ways that a narrow task-specific eval set — the kind teams tend to write first — will not catch.

Security and the Data-Leakage Surface

Each approach exposes sensitive data differently, and the differences matter more than a generic "AI has security risks" framing suggests.

With prompting, whatever you place in the context window is transmitted to the model provider on every single call. This is a well-understood risk — data processing and retention terms with the provider matter, and a team including sensitive customer data directly in prompts inherits whatever data handling commitments that provider makes, on a per-request basis, indefinitely. There is also a distinct risk specific to prompting and RAG (since RAG assembles its final input the same way): prompt injection, where untrusted content included in context — a document, a user-submitted field, a webpage the system retrieved — contains instructions designed to hijack the model's behavior. This is a different failure mode from data leakage, but it lives in the same architectural layer and needs to be tested for separately.

RAG adds a new persistent data store to the security surface that didn't exist before: the vector index itself, which holds a derivative representation — not the raw text, but not nothing either, since embeddings can in some cases be partially inverted to recover information about the source text — of your source corpus. This index needs its own access controls, its own encryption posture, and its own retention policy, and in multi-tenant systems it needs a retrieval-time scoping mechanism that reliably prevents one customer's query from returning another customer's indexed content. That last point is a real and growing risk category as more products bolt AI features onto existing multi-tenant architectures, and it deserves its own dedicated engineering and testing attention beyond what this article covers — the point relevant here is narrower: choosing RAG means choosing to stand up and secure a new data store, not just a new API call, and that operational responsibility needs an explicit owner from day one, not a discovery six months in.

Fine-tuning creates the security property that is hardest to reverse: your training data becomes embedded, in a diffuse and hard-to-audit way, in the model's weights. This is not a hypothetical concern. Academic research on large language models has directly demonstrated that models can memorize and, under the right prompting conditions, regurgitate near-verbatim snippets of their training data — Carlini et al.'s "Extracting Training Data from Large Language Models," presented at USENIX Security 2021, demonstrated successful extraction of memorized training examples, including personally identifiable information, from a production language model, establishing training data extraction as a real, exploitable attack surface rather than a theoretical one (Carlini et al., USENIX Security 2021); follow-up work has continued to show extraction attacks scale to production systems (Scalable Extraction of Training Data from (Production) Language Models, arXiv:2311.17035). The practical consequence for a fine-tuning decision is direct: if your fine-tuning dataset contains anything you would not want a sufficiently determined user to potentially extract from the deployed model — customer PII, proprietary pricing logic disguised as example outputs, internal-only content — that risk does not go away because the data "just became part of the model." It also creates a genuine data-subject-rights problem that RAG does not share in the same way: a deletion or "right to be forgotten" request against a vector database is a delete operation against a specific record. The same request against a fine-tuned model has no equivalent surgical operation — there is no reliable way to remove one individual's information from trained weights without retraining the model from a checkpoint that predates including it, which is a materially heavier operational and legal-compliance burden than most teams account for when they decide to fine-tune on data that includes personal information.

A third, illustrative and explicitly hypothetical scenario shows how this plays out concretely. A healthcare technology company builds an AI-assisted intake tool that helps triage incoming patient messages by comparing them against fine-tuned examples drawn from anonymized historical intake conversations, reasoning that fine-tuning on real historical conversations will teach the model the specific phrasing patterns and urgency signals clinicians actually use. The hidden assumption is that "anonymized" fully removes the risk, when in practice free-text clinical conversation data is notoriously difficult to anonymize completely — indirect identifiers (a rare condition combined with a specific age and location, a distinctive phrasing pattern, a reference to a specific medication regimen) can survive de-identification and, when a model trained on that data is later probed with a targeted extraction attempt, resurface in ways the original anonymization process did not anticipate. The consequence, discovered during a pre-deployment security review rather than in production (the better outcome, but only because the review happened before launch, not by design), is that the compliance and security teams cannot sign off on the fine-tuned model without a data provenance audit the original project timeline never budgeted for, because there is no way to demonstrate, after the fact, exactly what the model can and cannot regurgitate from its training set. The decision point is whether to attempt that audit and accept the delay, or restructure the approach entirely. The better approach, adopted after that review, separates the two needs the original design had conflated: the triage logic itself — recognizing urgency signals, routing categories, response templates — is handled through a fine-tuned or carefully prompted layer trained or instructed on synthetic and clearly de-identified example patterns rather than verbatim historical transcripts, while any need to reference actual historical case data is handled through an access-controlled retrieval layer with proper per-record authorization, rather than folding raw historical conversation data into model weights at all.

The Day the Base Model Changes Under You

This dimension gets the least attention in most comparisons and deserves the most, because it is the one where the ground has genuinely shifted underneath the industry in the time since a lot of the generic advice about these three approaches was written — and it directly determines how much of your customization investment survives a vendor's own roadmap decisions, which you do not control.

Prompting is, structurally, the most portable of the three across a base model change. A prompt that works well on one model may need retuning on another — different models respond differently to the same instructions and formatting conventions, so "portable" does not mean "identical results with zero changes" — but retuning a prompt is a matter of testing and adjustment against your existing evaluation suite, not a training pipeline. There is no artifact tied to a specific model checkpoint that becomes worthless when that checkpoint is retired.

RAG sits in between, with one specific technical wrinkle worth being explicit about: swapping the generation model in a RAG pipeline is relatively low-friction, for the same reason prompting is — you re-validate that the new model uses retrieved context correctly, adjust instructions if needed, and move on. Swapping the embedding model is not low-friction, because embedding spaces from different models are not directly comparable to one another; a vector produced by one embedding model cannot be meaningfully compared against vectors produced by a different one. Changing your embedding model means re-embedding your entire indexed corpus from scratch before retrieval will work correctly again. This is a real, recurring maintenance cost that teams evaluating "RAG is more portable than fine-tuning" sometimes gloss over — it is more portable on the generation side, and has its own lock-in dimension on the retrieval side that needs to be planned for separately.

Fine-tuning is where base-model dependency becomes a genuine, current, and underappreciated business risk, and the clearest evidence for that claim is not hypothetical — it happened during the writing of this article. In May 2026, OpenAI announced it is winding down self-serve access to its fine-tuning API. Under the announced timeline, organizations that have not run inference on a fine-tuned model within a 60-day window lose the ability to create new fine-tuning jobs, and by January 2027, no organization — including existing, active fine-tuning customers — will be able to create new fine-tuning jobs on the platform at all. OpenAI's own stated rationale, as reported in coverage of the announcement, is that newer base models follow instructions and formatting well enough out of the box that prompt-based approaches now handle cases that previously required fine-tuning, reducing the population of use cases the company considers worth supporting through a self-serve fine-tuning product (coverage summarizing OpenAI's announcement, Tessl, "OpenAI is shutting down self-serve fine-tuning"; OpenAI, fine-tuning platform update). That is a vendor's own stated justification for a business decision, not an independently verified technical claim, and it should be read that way — but the operational consequence for any team that had built a product feature on a fine-tuned OpenAI model is not in dispute regardless of the reasoning: inference on existing fine-tuned models continues, according to OpenAI's own guidance, only until the underlying base model itself is deprecated, at which point the fine-tuned asset stops working entirely, with no path to recreate it through the same self-serve process that originally built it.

This is worth sitting with, because it inverts a common assumption. Teams often treat fine-tuning as the "durable" investment — you did the work once, trained a model, and it's yours — compared to a prompt, which feels ephemeral and easy to lose track of across iterations. In practice, the fine-tuned artifact is the one whose continued existence depends entirely on a third party's decision to keep serving its specific base model, a decision made on that vendor's schedule, for that vendor's reasons, communicated with whatever notice period the vendor chooses to give. A prompt or a RAG pipeline degrade gracefully when you swap the underlying model — you retest and adjust. A fine-tuned model built on a deprecated checkpoint does not degrade; it stops.

The landscape is also genuinely uneven across providers, which matters for anyone assuming "fine-tuning" is a single, stable category of capability rather than a shifting set of individual vendor product decisions. Google Cloud continues to actively invest in and document supervised fine-tuning for its Gemini models through Vertex AI, with dedicated guidance on when and how to tune Gemini models for specific tasks (Google Cloud, About supervised fine-tuning for Gemini models). Anthropic, meanwhile, has never offered general self-serve fine-tuning of its flagship Claude models directly; the one publicly documented fine-tuning path for a Claude model ran through a partner platform — Amazon Bedrock's fine-tuning support for Claude 3 Haiku, announced generally available in November 2024 — and it was scoped specifically to Haiku, the smaller model in Anthropic's lineup at the time, not to the frontier model (AWS News Blog, "Fine-tuning for Anthropic's Claude 3 Haiku model in Amazon Bedrock is now generally available"; Anthropic, "Fine-tune Claude 3 Haiku"). That pattern — fine-tuning access offered for a smaller or older model rather than the current frontier model — recurs often enough across the industry to be worth naming as a pattern in its own right: even where fine-tuning is available, it is frequently not available for the model you would otherwise most want to use, which is its own quiet cost of the approach.

The practical takeaway is not "never fine-tune." It is that a fine-tuning decision needs to explicitly account for vendor dependency as a first-class risk, the same way a team would evaluate dependency risk before building critical infrastructure on any other single-vendor API — with an honest answer to the question "if this vendor's fine-tuning product changes or disappears in twelve months, what does it cost us to recreate this capability, and on what alternative base model." Teams using open-weight models they host themselves face this risk differently — they retain the fine-tuned weights indefinitely regardless of any vendor's product roadmap — but they take on the operational burden of hosting and serving those weights themselves, which is its own cost and skill dependency, addressed next.

Who Owns This in Six Months

Every one of the technical trade-offs above eventually becomes an organizational question, because someone has to be responsible for keeping the system correct after the launch celebration is over, and the three approaches place that ongoing responsibility on very different roles.

A prompting-based system's ongoing ownership burden is comparatively light and comparatively cheap to distribute: maintaining and iterating on prompts is a skill a product manager, a technical writer, or a generalist engineer can develop without specialized machine learning background, provided the team has built the evaluation discipline discussed earlier to catch regressions from prompt changes. The risk on this side is not technical difficulty; it's the ease with which prompt changes get made informally, by whoever is closest to a customer complaint, without going through the same review rigor a code change would get.

A RAG system's ownership burden is split across two distinct skill sets that don't always live on the same team: someone needs to own the retrieval pipeline itself — chunking strategy, embedding model choice, index freshness, retrieval evaluation — which is closer to a data engineering and information retrieval discipline than a machine learning one, and someone needs to own the generation-side prompt and instructions that consume retrieved context. In practice, this frequently falls between an application engineering team, which owns the product surface, and a data or platform team, which owns the underlying corpus and index, and the handoff between them is exactly where staleness and retrieval-quality regressions tend to go unnoticed the longest, because neither side considers it fully their problem.

A fine-tuning-based system requires a genuinely different skill set to own responsibly: dataset curation and labeling discipline, an understanding of overfitting and forgetting risk well enough to design a proper held-out evaluation set, familiarity with the specific training infrastructure and hyperparameters involved, and — for teams hosting their own fine-tuned open-weight models rather than using a managed vendor product — ongoing infrastructure and serving expertise that most product engineering teams do not have in-house and do not want to build for a single feature. This is not a claim that fine-tuning requires a dedicated research team for every use case; parameter-efficient methods like LoRA have genuinely lowered that bar. But it is a real, distinct skill requirement, and a team that fine-tunes a model without anyone on staff who understands how to evaluate it for regression and forgetting is not saving effort by skipping RAG's infrastructure — it is deferring the effort to the first time the fine-tuned model behaves unexpectedly and nobody on the team can diagnose why.

Combining Approaches Without Combining Complexity Blindly

None of the preceding sections argue that these are mutually exclusive choices, and in a meaningful share of real production systems, the right answer is a deliberate combination — provided the combination is chosen because each piece is doing the specific job it's structurally good at, not because a team couldn't decide and layered on complexity to avoid deciding.

The most common and best-justified combination is RAG for knowledge, paired with a light fine-tune (typically LoRA-based) for behavior. This maps directly onto the behavior-versus-knowledge distinction established earlier: retrieval handles everything that needs to stay current — product data, policy documents, live status — while the fine-tune handles the parts of the task that are stable and repetitive enough to be worth training into the model directly: a consistent output format the retrieval-only version kept getting slightly wrong despite careful prompting, a specific tool-calling pattern the product relies on, a tone and structure that needs to be more consistent than prompting alone reliably produces across a high volume of varied queries. A useful way to think about this combination is that fine-tuning is teaching the model how to use and present information well, while retrieval is supplying what information it uses — and testing the two halves still requires the same separation of retrieval-quality evaluation from generation-quality evaluation described earlier, now with the added need to verify that the fine-tuned behavior hasn't degraded on inputs that fall slightly outside the fine-tuning dataset's distribution.

A second, less common but legitimate combination pairs fine-tuning with a smaller model for a narrow, extremely high-volume subtask, while a larger general-purpose model (with or without RAG) handles the smaller volume of more open-ended requests — a routing architecture rather than a single monolithic model handling every request the same way. This is where the cost curve from earlier becomes directly actionable: if 80% of a feature's query volume is a narrow, repetitive task (classifying a support ticket into one of a fixed set of categories, for instance) and 20% is open-ended, fine-tuning a small model for the narrow 80% and reserving a larger, more expensive general-purpose model (potentially with RAG) for the remaining 20% can produce a better cost and latency profile than routing every request through the same expensive, general-purpose path — provided the routing decision itself is reliable, since a misrouted open-ended request sent to the narrow fine-tuned model will fail in a way that's harder to detect than an outright error, because the narrow model will still confidently attempt an answer.

The Customization Commitment Index: A Working Framework

Rather than a generic maturity ladder, the following instrument is built around the six properties this article has argued actually predict whether a customization approach holds up in production: data volatility, required update speed, query volume at scale, latency sensitivity, available ML ownership capacity, and portability requirements across model vendors. Score your feature on each dimension, then use the total to identify the zone your feature falls into — not as a rigid formula, but as a structured way to surface disagreements on your team about the feature's actual requirements before you commit to an architecture based on an assumption nobody stated out loud.

Step 1 — Score each dimension from 0 to 2.

Step 2 — Interpret the total (0–12).

This is deliberately not a scoring system that produces "fine-tuning" as a default recommendation at high volume alone — notice that the highest zone still requires an explicit vendor-risk review as a precondition, not an afterthought, precisely because that is the dimension the previous section showed teams most often skip. A feature can score 12 on volume and latency alone and still be a poor fine-tuning candidate if its data volatility score is also high; the index is meant to be read across all six dimensions together, not by whichever single dimension feels most urgent in the room at decision time.

Three Situations Where the "Better" Approach Is the Wrong One

Frameworks like the one above are useful precisely because they also reveal their own exceptions. Three are worth stating explicitly, because each one contradicts a piece of conventional wisdom about these approaches.

RAG is the wrong choice when the underlying corpus is small, static, and well within a model's context window. Anthropic's own guidance on this point is direct: for a knowledge base smaller than roughly 200,000 tokens — about 500 pages of material — the company suggests simply including the entire knowledge base in the prompt rather than building retrieval infrastructure at all (Anthropic, Contextual Retrieval). Building a vector index, an embedding pipeline, and a retrieval evaluation process for a corpus that fits comfortably in a single prompt — and that doesn't change often enough to need the freshness benefits retrieval provides — is pure operational overhead with no corresponding benefit, and it introduces retrieval-quality risk (a chunk that should have been retrieved but wasn't) that simply including everything avoids entirely.

Fine-tuning is the wrong choice when the actual goal is teaching the model new facts, no matter how much training data you have. This is worth stating bluntly because it's the single most common category of fine-tuning project mismatch: teams collect a large dataset of documents and treat fine-tuning as a way to "teach the model our documentation," when — per the mechanical distinction established earlier — fine-tuning is far more reliable at shaping behavior and format than at reliably instilling new, precisely recallable facts, and a model fine-tuned this way will often still hallucinate details from documents it was "trained on," with no mechanism to cite or verify against a source, which a retrieval-based approach provides for free by construction.

Prompting-with-full-context is the wrong choice at meaningful scale even when it's technically sufficient, because the cost analysis earlier in this article demonstrates that its cost scales the worst of the three options as volume grows, independent of whether the answers it produces are correct. A team that validates a full-context prompting approach during a low-volume beta and never revisits the decision as volume grows by two orders of magnitude is not making an architecture mistake at launch — it's deferring a cost problem to whoever owns the infrastructure budget eighteen months later, at a scale where re-architecting is a much bigger project than it would have been to start with RAG from day one.

Startups, Scale-Ups, and Enterprises Carry Different Ledgers

The same technical trade-offs land differently depending on organizational stage, and the Customization Commitment Index above should be read through this lens rather than in isolation.

A pre-product-market-fit startup's binding constraint is usually engineering time and the cost of being wrong about the product itself, not infrequently the underlying customization architecture. Prompting-first is almost always the correct default here, even when the index above would technically justify RAG, because the cost of rebuilding a prompting-based feature once product-market fit clarifies the real requirements is far lower than the cost of building and maintaining retrieval infrastructure for a feature that might be redesigned entirely in three months. The exception is a genuinely data-volatile domain from day one — live pricing, live inventory — where prompting-with-stale-data isn't a viable minimum product at all, not even temporarily.

A scale-up, past product-market fit and dealing with real query volume for the first time, is typically the stage where the cost curve from earlier stops being theoretical and starts showing up as a real, noticed line item — and it's also the stage with the least slack to absorb a full architecture rebuild without disrupting a growing customer base. This is the stage where the Commitment Index earns its keep as a deliberate, written exercise rather than an implicit assumption, because it's cheaper to have this conversation on purpose, with the scoring table in front of the team, than to have it forced by a finance team asking why the AI feature's inference bill grew faster than its usage did.

An enterprise, with more resources to throw at any single approach, faces a different failure mode: the temptation to over-engineer a customization strategy — building fine-tuning infrastructure, a full RAG pipeline, and elaborate routing logic for a feature that a scored-honestly assessment would have placed comfortably in the prompting-first zone — because the organization has the resources to build all three and no forcing function pushing it toward the simplest approach that meets the actual requirement. Enterprises also carry the heaviest version of the security and compliance dimension discussed earlier: a fine-tuning decision that touches regulated data, or a RAG index that spans data governed by different retention or residency requirements across business units, needs the kind of formal review that a scale-up can sometimes defer and a startup usually hasn't encountered yet.

Questions to Bring Into the Next Architecture Review

Before approving a customization approach for a new AI feature, a technical decision-maker should be able to get a specific, non-hand-wavy answer to each of the following from whoever is proposing the architecture:

• How often does the underlying knowledge this feature depends on actually change, and who is responsible for propagating that change into the system within an agreed time window?
• What does this approach cost per query at our current volume, and separately, at 10x and 50x that volume — not just at prototype scale?
• Where, specifically, does latency accumulate in this pipeline, and which stage would we optimize first if response time became a problem?
• Do we have a way to evaluate retrieval quality and generation quality separately (for RAG), or task accuracy and general-capability regression separately (for fine-tuning) — or are we only measuring final output quality and hoping that's enough?
• What data, specifically, will be included in the prompt, the retrieval index, or the fine-tuning dataset, and have we reviewed it against our data handling and retention obligations for exactly that use?
• If our model provider deprecates or materially changes the base model this feature depends on with six months' notice, what is the actual plan, and has anyone estimated what it would cost to execute it?
• Who owns this system's ongoing correctness once the initial project team moves on to the next thing — by name, not by team name?

Frequently Asked Questions

Does fine-tuning add new knowledge to a language model? Not reliably. Every major provider's own guidance positions fine-tuning as most effective for shaping behavior — format, tone, classification accuracy, instruction-following consistency — rather than as a dependable way to instill new, precisely recallable facts. For grounding a model in specific, verifiable, and changing information, retrieval-augmented generation is the better-suited mechanism.

How much data do you need to fine-tune a model? OpenAI's own guidance sets a technical minimum of 10 examples and recommends starting with roughly 50 well-crafted demonstrations, evaluating results before deciding whether to scale up further; if 50 examples don't produce a measurable improvement, the recommendation is to reconsider the task rather than add more data. The right number is task-dependent, but quality and evaluation discipline matter more than raw volume.

Is RAG always cheaper than fine-tuning? No — it depends on scale. In the illustrative cost model in this article, fine-tuning's fixed training cost made it the most expensive option at low query volume, while RAG was cheapest at low and moderate volume, and fine-tuning's lower per-query cost only overtook RAG at very high volume once that fixed cost was amortized. The comparison has to be made at your actual and projected query volume, not treated as a fixed ranking.

What is catastrophic forgetting, and does LoRA solve it? Catastrophic forgetting is the tendency of a model to lose previously learned capabilities while being trained on new, narrower data. Parameter-efficient methods like LoRA reduce the risk relative to full fine-tuning by freezing the original weights and training only a small set of additional parameters, but current research shows forgetting still occurs within LoRA-based fine-tuning under some conditions — it is a reduced risk, not an eliminated one.

Can you fine-tune Claude or GPT models directly today? The landscape varies by vendor and changes over time. OpenAI announced in May 2026 that it is winding down self-serve access to its fine-tuning API, with new fine-tuning job creation ending for all customers by January 2027; existing fine-tuned models continue to serve only until their underlying base model is deprecated. Anthropic has not offered general self-serve fine-tuning of its flagship Claude models directly; the one documented path — fine-tuning Claude 3 Haiku through Amazon Bedrock — was scoped to a smaller model rather than the frontier model. Google continues to actively support supervised fine-tuning for Gemini models through Vertex AI. Confirm current availability directly with the relevant provider before committing an architecture to it.

Should we combine RAG and fine-tuning? Often, yes — but only when each layer has a distinct, articulable job: retrieval for knowledge that changes, fine-tuning for behavior that's stable. If a team can't clearly state which part of the task each layer handles, the combination is likely adding maintenance overhead without a corresponding benefit.

The Decision You Cannot Outsource to a Demo

The three approaches compared throughout this article will keep producing similar-looking answers in similar-looking demos, and that will keep tempting teams to treat the choice between them as low-stakes. It is not low-stakes; it is a decision about how your product will behave as your data ages, as your usage grows by orders of magnitude, as your evaluation team tries to explain a regression, and as a model provider you don't control changes its own roadmap. The concrete principle worth carrying out of this comparison is simple to state and consistently skipped in practice: score the decision against how your data and usage will look in a year, not against how your prototype performs against the ten test queries you happened to write this week. A fine-tuned model that performs beautifully in a demo built from last month's data is not evidence that fine-tuning was the right call — it's evidence that the demo didn't run long enough for the trade-off to show up yet.

The next time your team is choosing between prompting, RAG, and fine-tuning for a new feature, the question worth taking back to the room is not "which one works." All three probably will, in the demo. The question is: which one still works, on the same terms, a year from now — and who specifically signed up to make sure it does.

Getting that answer wrong is rarely a modeling problem. It's usually an evaluation and ownership gap that existed before a single line of the customization pipeline was written — nobody had defined what "still correct" would mean for this feature, or who would be accountable for checking. QAtronic works with engineering teams on exactly that layer: building the retrieval and generation evaluation harnesses that catch a staleness or regression problem before a customer does, and helping define, in writing, who owns a customization pipeline's correctness once the team that built it has moved on to the next feature.