A payment app on a commuter train loses signal for four seconds. The customer has already tapped "Pay $340." The client sends the charge request, the request reaches the payment processor, the processor authorizes the card and returns a success response — and the response never makes it back to the phone. The train enters a tunnel. The app's HTTP client hits its timeout and does what mobile clients are built to do: it retries the request automatically, using the same amount, the same card, the same order.

From the server's point of view, nothing went wrong the first time. The charge succeeded. The second request looks, on paper, like a completely ordinary new payment for $340 against the same card. Nothing in the payload says "this is a copy." Nothing in the transport layer says "the client already got what it needed." Unless something in the system was specifically built and specifically tested to recognize that second request as a duplicate of the first, the customer is charged twice, a refund ticket gets opened, support has to explain a discrepancy the customer can see on their bank statement before your team can, and finance has to reconcile a transaction that never should have existed. This is a hypothetical, deliberately simplified scenario — no specific vendor or incident is being described — but the mechanics are not exotic. They are the default behavior of almost every mobile HTTP client, every message queue configured for reliability, and every webhook provider that takes delivery seriously.

The interesting part is not that this can happen. Every engineer who has built a payment flow, a checkout, or a webhook consumer already knows retries exist. The interesting part is how rarely it gets tested. Look at almost any QA plan for a payment endpoint, an order-creation flow, or a webhook handler and you will find extensive coverage of the request succeeding once: correct amount charged, correct order created, correct email sent, correct inventory decremented. You will often find coverage of the request failing cleanly: invalid card, insufficient funds, malformed payload. What you will rarely find is a test that deliberately sends the exact same successful request a second time, or a near-duplicate with a new request ID but the same underlying intent, and asserts that nothing changes as a result. Idempotency is treated as an architectural property that presumably exists somewhere in the design, rather than a behavior that has to be demonstrated under test the same way correctness and error handling are.

This article treats idempotency as what it actually is: an explicit contract between a client and a server about what happens when a request is repeated, built out of specific mechanisms — idempotency keys, deduplication tables, delivery semantics — that can fail in specific, testable ways. It works through why duplication happens, how the major delivery guarantees your infrastructure already relies on actually behave, how to design tests that exercise the second request instead of only the first, and what a practical testing program looks like depending on how large and how regulated your organization is.

The stakes are not evenly distributed across a codebase, which is part of why this gap persists. A duplicated internal cache refresh costs nothing anyone notices. A duplicated charge, a duplicated shipment, or a duplicated legally significant notification costs real money, real support time, and — more durably — a specific kind of customer trust that is hard to earn back once someone has seen two identical charges on their bank statement for something they bought once. The engineering effort this topic deserves should track that unevenness: a handful of endpoints in almost every system carry the overwhelming majority of the real risk, and those are exactly the ones most likely to have been built, and tested, under the assumption that the first attempt is the only attempt that matters.

What "Idempotent" Actually Means Once Money or State Is Involved

The word gets used loosely, so it is worth being precise before going further. In HTTP semantics, an idempotent method is one where making the same request multiple times produces the same effect on server state as making it once. GET, PUT, and DELETE are defined as idempotent by the HTTP specification. POST is not, and this is not an oversight — POST is the method used for operations that are inherently about creating something new or triggering an action, where the framers of the specification could not assume repetition was safe by default.

This matters because most of the operations that cause real duplication problems are POST requests: create a charge, create an order, create a shipment, send a notification, provision a resource. The HTTP method tells you nothing about whether the underlying business operation is safe to repeat. A PUT /users/42/address that sets the address to a specific value is naturally idempotent — sending it five times leaves the address in the same final state as sending it once. A POST /charges that debits a card by a specified amount is naturally not idempotent — sending it five times, absent any additional protection, debits the card five times. The method being technically idempotent by specification and the operation being safe to repeat in practice are two different claims, and conflating them is one of the more common design mistakes in this space. Teams sometimes reason "we used PUT for this, so it's idempotent," when what actually makes an operation safe to repeat is the server-side logic that recognizes repetition and short-circuits it, not the verb in the request line.

It also helps to separate two properties that get bundled together under the same word:

• Idempotent: repeating the request produces the same end state as making it once. The card is charged $340 exactly once, regardless of how many times the charge request was sent.
• Safe: the request has no side effects at all, so repetition is trivially fine. A GET that reads an account balance is safe. A POST that creates a charge is neither safe nor, by default, idempotent — it has to be made idempotent deliberately.

There is a further distinction that matters even more once you are past a single API call: idempotency at the transport or API layer is not the same as idempotency at the business-operation layer. A single "charge the customer" action in a real system is rarely one atomic write. It typically involves authorizing the charge with a payment processor, writing a ledger entry, decrementing available inventory or seat count, triggering a fulfillment or provisioning workflow, and sending a confirmation email — five or six side effects fanning out from one logical action. An idempotency key that only protects the initial API call, without any thought given to the steps downstream of it, can still let the notification get sent twice or the inventory get decremented twice if those steps are triggered by an event that itself gets redelivered. Idempotency has to be reasoned about, and tested, at every layer where a side effect happens, not only at the front door.

Three Ways the Same Request Shows Up Twice

Duplication is not one failure mode with one cause. It is useful to separate it into three distinct mechanisms, because each one requires a different kind of test and, often, a different kind of protection.

Client-side retry after an ambiguous failure. This is the mechanism in the opening scenario. A client sends a mutating request, does not receive a response within its timeout window, and cannot distinguish between three very different underlying realities: the request never reached the server, the request reached the server but failed before doing anything, or the request reached the server, succeeded completely, and only the response was lost in transit. Network timeouts are symmetric with respect to these three cases — the client experiences the exact same symptom (no response) regardless of which one actually happened. A client that wants to be resilient to the first two cases has no choice but to retry, and in doing so it also retries in the third case, where retrying is the one thing that causes harm. This is not a bug in the client. It is a structural consequence of asynchronous networks: you cannot tell "it didn't happen" apart from "it happened but I didn't hear about it" without additional information, and that additional information is exactly what idempotency mechanisms are built to supply.

Infrastructure-level redelivery under at-least-once semantics. Message queues, event streams, and webhook systems that prioritize not losing messages almost universally choose at-least-once delivery as their default guarantee, because guaranteeing zero message loss and guaranteeing zero duplication at the same time is a much harder and more expensive problem — one that, as discussed in the next section, cannot be fully solved by the transport layer alone. Amazon SQS documents this directly: standard queues guarantee at-least-once delivery, and the documentation is explicit that this is a deliberate tradeoff of "reliability over exactly-once semantics" — messages are stored redundantly across servers, and if a server becomes briefly unavailable during message deletion, a redundant copy can still be delivered again after that server recovers. A consumer that is not built to expect this will process the same message twice. The same pattern shows up in webhook systems: Stripe's own documentation for webhook endpoints states plainly that "webhook endpoints might occasionally receive the same event more than once," and recommends that receiving applications log event IDs and skip ones they have already processed. This is not a defect in Stripe's delivery system. It is the same at-least-once tradeoff, applied to HTTP callbacks instead of a queue.

Human duplicate submission. A user double-clicks "Submit" because the button did not visibly respond fast enough. A user hits the browser back button after a slow page load and resubmits a form. A user has two tabs open on the same checkout page and completes the purchase in both. None of these involve any retry logic at all — they are simply two independent, genuine HTTP requests carrying the same business intent, often without even sharing a request ID, because each one was generated fresh by the browser. This is the category that idempotency keys alone do not automatically solve, because a client-generated key is only useful if the client is disciplined about generating it once per logical action and reusing it on retry — a double-click from two independent form submissions may generate two different keys for what is, to the user, the same intended purchase. Handling this class typically requires either UI-level submission locking (disable the button immediately, don't rely on server logic alone) combined with a short server-side dedup window keyed on a fingerprint of the request (user, amount, item, and a tight time window), or a client architecture where the idempotency key is generated once when the form is rendered and persists across any resubmission attempt.

These three mechanisms are worth keeping separate because a test plan built only around one of them — say, only around infrastructure redelivery — will still leave a system exposed to the other two. A queue consumer can be perfectly deduplicated against message redelivery and still allow duplicate orders from a user who double-clicks a checkout button, because that failure never touches the queue at all.

The three also compound rather than staying neatly isolated. A single real-world duplication incident frequently involves more than one of them at once: a flaky mobile connection triggers a client-side retry (mechanism one), and because the underlying operation is implemented as a chain of events on a queue, the resulting write also triggers a redelivered downstream event when a consumer briefly loses its connection to the broker mid-acknowledgment (mechanism two). A test suite that validates each mechanism in isolation — one test for "client retries," a separate test for "queue redelivers" — can pass cleanly while still missing the specific interaction between them, because neither test alone reproduces a client retry landing on a system that is simultaneously processing a queue redelivery of an earlier attempt. This is a reasonable argument for including at least one combined scenario in a mature test suite, not only the individually isolated ones.

The Delivery Guarantees Baked Into Your Infrastructure (Whether You Tested Them or Not)

Every messaging system, queue, and webhook provider makes an implicit or explicit promise about how many times a message will be delivered relative to how many times it was sent. Understanding these guarantees precisely — not the marketing summary, the actual documented behavior — is a prerequisite for knowing what your consumers need to defend against.

At-most-once delivery means a message is delivered zero or one times, never more. This favors avoiding duplicates over avoiding loss: if delivery is uncertain, the system drops it rather than risk sending it twice. This is rare in systems that care about business data, because losing a payment confirmation or an order event silently is usually worse than the alternative.

At-least-once delivery means a message is delivered one or more times, never zero. This is the default and by far the most common guarantee in production messaging systems, because favoring "the consumer might see it twice" over "the consumer might never see it" is almost always the right tradeoff for business-critical data. Amazon SQS standard queues, most webhook providers (including Stripe, as documented above), and the default configuration of many event-streaming platforms operate this way. The cost of this guarantee is duplication, and the system explicitly pushes the responsibility for handling that duplication onto the consumer.

Effectively-exactly-once processing is what teams actually want, and it is worth being precise about how it is achieved, because the phrase gets used in ways that overstate what is really happening. True exactly-once delivery at the transport layer — where the network itself guarantees a message arrives precisely once regardless of any failure — is not achievable in an asynchronous distributed system with unreliable networks; this is a well-established result in distributed systems literature, not a limitation specific to any one vendor. What cloud providers offer instead, and what is genuinely useful, is exactly-once processing: at-least-once delivery at the transport layer, combined with deduplication logic (either provided by the platform or built by the consumer) that makes the net effect of processing equivalent to exactly one execution, even though the message itself might physically arrive more than once.

Google Cloud's Pub/Sub is a clear, well-documented example of this distinction in practice. Pub/Sub's exactly-once delivery feature, generally available since 2022, guarantees that a successfully acknowledged message will not be redelivered — but the guarantee comes with real, documented boundaries: it applies only to pull subscriptions (push and export subscriptions are explicitly excluded), it holds only within a single cloud region (a subscriber application spanning regions will still see duplicates), and it carries measurably higher publish-to-subscribe latency than standard at-least-once subscriptions because of the additional coordination required. Google's own documentation is explicit that publish-side duplicates — where a publisher's own retry logic creates two distinct messages with different message IDs for what was semantically one publish attempt — are not covered by this guarantee at all, because from the broker's perspective those are two different messages, not one message delivered twice.

This is exactly the kind of nuance that gets lost between an architecture diagram and a test plan. A diagram that shows "Pub/Sub (exactly-once)" between two services is making a claim that is true only under specific, narrower conditions than the label suggests, and a QA process that takes the label at face value instead of testing the actual behavior — including the excluded push-subscription case, the cross-region case, and the publisher-retry case — will miss real gaps.

The practical takeaway is not "always demand exactly-once processing." It is that whichever guarantee your architecture relies on, that guarantee has documented conditions and documented exceptions, and a QA process that has not read those conditions is testing against an assumption rather than a specification.

The Idempotency Key Pattern — and the Four Ways Teams Implement It Incorrectly

The most direct tool for handling client-side retries and infrastructure redelivery at the API layer is the idempotency key: a unique value, generated by the client, attached to a mutating request, that the server uses to recognize "I have already processed this exact request" and return the original result instead of executing the operation again.

Stripe's implementation is a well-documented reference point for how this is supposed to work, and it is worth walking through precisely because the details matter more than the general concept. A client sends a POST request with an Idempotency-Key header:

curl https://api.stripe.com/v1/charges \
  -u sk_live_xxx: \
  -H "Idempotency-Key: 7f3c9e21-4b8a-4e2d-9c6f-2a1d5e8b0c44" \
  -d amount=34000 \
  -d currency=usd \
  -d customer=cus_ABC123

According to Stripe's documentation, the server saves the resulting status code and response body of the first request made with a given key, regardless of whether that first attempt succeeded or failed, and any subsequent request using the same key returns that saved result without re-executing the operation — including replaying a 500 error rather than trying again. Keys are retained for at least 24 hours before they can be pruned; after a key is removed, reusing it starts a genuinely new request. Critically, the documentation also notes that Stripe compares the parameters of a repeated request against the parameters of the original request under the same key and returns an error if they differ — this closes a real vulnerability where a client could reuse a key with a different amount and either get an unexpected cached result or, worse, silently overwrite the original operation's intent.

That is a fully worked reference implementation of the pattern, and it is a useful model for what "correct" looks like. In practice, teams building their own idempotency layer on top of internal APIs tend to get one or more of the following details wrong, and each of these should be its own explicit test case rather than something inferred to work:

1. Scoping the key too narrowly or too broadly. If the deduplication table keys only on the idempotency key string itself, without also scoping to the account, API key, or tenant that submitted it, then two different customers who happen to generate the same key value (more likely than it sounds if key generation is naive, for example based on a predictable counter rather than a UUID) can collide and one customer's request returns another customer's cached result. Conversely, if the key is scoped too broadly — including something like a timestamp that changes on every retry — then it stops functioning as a deduplication key at all, because every "retry" generates a technically distinct key and sails straight through.

2. Storing the key after the side effect instead of before it. A dedup check that works by first performing the charge and then recording the key is exposed to a race: if two copies of the same request arrive close enough together (a very real scenario when a client retries aggressively on a fast connection), both can pass the "have we seen this key before" check before either one has finished recording it, and both proceed to execute the charge. The correct pattern reserves the key — inserting a row in a "pending" state, using a unique constraint on the key column so that a concurrent second insert fails — before the operation begins, not after it completes.

3. No expiry, or an expiry misaligned with the retry window that actually matters. Storing idempotency keys forever is a real operational cost at scale, but expiring them too aggressively reopens the exact failure the key exists to prevent: if a mobile client can plausibly retry a request up to, say, six hours after the original attempt (a genuinely realistic window for a user who loses connectivity, closes the app, and reopens it later to find "did that go through?"), a dedup window of ten minutes does not protect against that retry at all. The window has to be chosen based on the actual retry behavior of the calling clients, not an arbitrary default.

4. Treating the API-layer key as sufficient protection for everything downstream of it. As covered in the previous section, an idempotency key on the initial POST /charges call does nothing to prevent the confirmation email from being sent twice if that email is triggered by a queue event that gets redelivered independently. Each side effect that has its own trigger needs its own dedup logic, even when they are all part of "the same" logical operation from a product perspective.

A minimal dedup table sketch, reflecting the "reserve before executing" pattern from point 2, looks roughly like this:

CREATE TABLE idempotency_keys (
    idempotency_key   VARCHAR(255) NOT NULL,
    account_id        BIGINT       NOT NULL,
    request_hash      VARCHAR(64)  NOT NULL,  -- hash of the normalized request body
    status            VARCHAR(16)  NOT NULL,  -- 'pending' | 'completed' | 'failed'
    response_code     INT,
    response_body     JSONB,
    created_at        TIMESTAMPTZ  NOT NULL DEFAULT now(),
    completed_at      TIMESTAMPTZ,
    PRIMARY KEY (idempotency_key, account_id)
);

The primary key on (idempotency_key, account_id) is what makes the "insert before executing" pattern safe under concurrency: a second, near-simultaneous request with the same key and account fails the insert with a constraint violation rather than racing past a read-then-write check, and the handler can catch that violation and either wait for the first request to finish and return its result, or return a "request in progress" response if it is still pending.

The Database as a Backstop, Not Just the Application Logic

Everything described in the idempotency-key pattern above lives in application code, which means it is only as reliable as that application code's own correctness — and application code has bugs, gets refactored by engineers unfamiliar with the original dedup logic, and occasionally gets bypassed entirely by a well-intentioned internal script, an admin retry button, or a data-backfill job that skips the normal request path. Treating the idempotency key check as the only line of defense leaves the system exposed to any of those cases.

A cheap, durable second layer is a uniqueness constraint enforced directly by the database, on a column or combination of columns that should never legitimately repeat for the same logical operation — an order's external reference number, a payment's provider transaction ID, a shipment's carrier tracking request ID. Unlike application-level dedup logic, a database constraint cannot be accidentally skipped by a code path nobody thought to add the check to, because it is enforced at the point of the write itself, independent of which code triggered that write.

ALTER TABLE charges
  ADD CONSTRAINT uq_charges_idempotency_key
  UNIQUE (account_id, idempotency_key);

ALTER TABLE shipments
  ADD CONSTRAINT uq_shipments_provider_reference
  UNIQUE (fulfillment_provider_id, provider_shipment_reference);

This is not a replacement for the application-level idempotency-key pattern described earlier — a database constraint only rejects a duplicate write after the fact, typically with a generic constraint-violation error that the application still has to catch and translate into the correct "here is your original result" response, rather than proactively short-circuiting the operation before any external side effect (like an actual card authorization) has already happened. The two layers do different jobs: application-level dedup logic decides early whether to execute the operation at all, and the database constraint guarantees that even if that logic is ever wrong, incomplete, or bypassed, the system still cannot end up with two rows representing what should be one transaction. A duplicate-request test suite should include a case that deliberately calls the write path a second layer beneath the normal API — directly against the service layer or repository, bypassing the idempotency-key check — specifically to confirm the database constraint catches what the application logic did not. Teams that skip this test frequently discover, only after a schema migration or a service rewrite quietly dropped or never carried forward a constraint, that their only protection against duplication was ever the application code.

When the Operation Itself Isn't Atomic: Partial Failure and Composite Side Effects

The idempotency key pattern assumes something convenient that is not always true: that the operation behind it either fully happens or fully does not happen. In reality, many of the operations most worth protecting are composites — several side effects, across several systems, that do not commit atomically as a group. A "place order" action might, in sequence, authorize payment, write an order record, decrement inventory, and enqueue a fulfillment event. If the process crashes after decrementing inventory but before enqueuing fulfillment, a naive retry that starts the whole sequence over will decrement inventory a second time for an order that, from the inventory system's point of view, has already been accounted for once.

This is the scenario that a request-level idempotency key alone does not fully solve, because the key protects against redoing the whole operation, not against safely resuming a partially completed one. Two practical approaches handle this, and the right choice depends on how tightly coupled the steps are:

The first is designing each individual step to be independently idempotent, and recording progress as the operation moves through them, so a retry can check "which steps have already completed for this operation ID" and only execute the remaining ones. This requires each step — the inventory decrement, the fulfillment event, the email send — to itself accept an operation ID and be safe to call twice, pushing the idempotency requirement down to every side effect rather than concentrating it at the entry point.

The second, more heavyweight approach is a saga-style pattern with compensating actions: if a later step fails, the system does not retry from the top, it runs explicit compensating operations to undo the steps that already succeeded (release the inventory hold, reverse the ledger entry) and then either retries cleanly from a known-good state or surfaces the failure for manual handling. This is considerably more implementation effort and is usually reserved for operations with enough steps, enough cost of error, or enough cross-service coordination to justify it — a multi-leg travel booking or a multi-party payout, for example, rather than a straightforward single-item purchase.

Either way, the QA implication is the same: a duplicate-request test suite that only exercises "send the whole request twice" is testing the easy half of this problem. The harder, more production-relevant half is "kill the process after step two of four completes, then trigger a retry, and verify steps one and two are not repeated while three and four correctly execute." That failure-injection style of test — deliberately interrupting a multi-step operation partway through — belongs in the same test suite as the simpler exact-duplicate case, and it is the one most teams skip, because it requires the ability to actually pause or fail a process mid-operation rather than just replaying an HTTP call.

Designing Tests That Actually Send the Second Request

Everything above is analysis. The following is what a QA program does with it. The core discipline is simple to state and consistently skipped in practice: for every mutating endpoint that matters, the test suite needs cases that deliberately send more than one request representing the same intent, under a range of timing conditions, and assert on the observable side effects — not just the HTTP response code, but the actual downstream state: how many charges exist, how many emails were queued, how many inventory units were decremented.

A useful way to organize this is by scenario category, because each one exercises a different part of the mechanism discussed above.

Exact duplicate, same idempotency key, sequential. Send the request, wait for the response, send the identical request (same key, same body) again. Assert: second response matches the first exactly (including status code); only one charge/order/record exists; no additional side effects (email, inventory change, downstream event) were triggered by the second call.

Exact duplicate, same idempotency key, concurrent. Fire two copies of the identical request at effectively the same time, without waiting for either response. This is the case that catches the "insert after executing" race described earlier. Assert: exactly one of the two requests actually executes the operation; the other either receives the same cached result or a well-defined "in progress" response; under no interleaving does the operation execute twice.

Near-duplicate, different idempotency key, same business intent. Simulate the double-click or double-tab scenario: two distinct requests, each with its own freshly generated key, but identical in every other respect (same user, same amount, same item, submitted within a short window). This is the case a pure key-based dedup system will not catch by design, since the keys genuinely differ. Assert whichever behavior the product has actually decided on — reject the second as a likely duplicate based on a fingerprint match, or accept both as legitimately separate actions — but assert it deliberately, rather than leaving the behavior as whatever falls out of the code by accident.

Retry after a timeout where the server actually completed the operation. This is the scenario from the opening of this article, and it is genuinely difficult to simulate without deliberately engineering the test harness for it, because it requires letting the server finish processing while making the client believe it did not get a response. One practical approach: hold a proxy or test harness between client and server that forwards the request, lets the server process it fully, and then drops the response before it reaches the client — then sends the retry through normally and observes what happens.

Retry after a timeout where the server had not started processing. The simpler, more benign sibling of the above: the connection drops before the server even received the request, so the retry is functionally the first real attempt. This case matters mainly as a control — it confirms the dedup mechanism does not falsely treat an unrelated first attempt as a duplicate and reject legitimate traffic.

Replayed webhook or redelivered queue message. Take a message the consumer has already processed and successfully acknowledged, and manually redeliver it — most queue and webhook tooling supports this directly (Stripe's dashboard and CLI both support manually resending a specific event, for example, and most queue consoles support requeueing a specific message ID). Assert that reprocessing does not repeat the side effect, and separately, that it does not silently fail either — a good dedup implementation returns a clean "already processed" outcome, not a crash or a swallowed error that looks the same as success from a monitoring dashboard but actually did nothing.

Expired idempotency key reuse. Advance the dedup window past its expiry (or configure a short window for the test environment specifically) and resend a request with a key that has aged out. Assert that this is treated as a genuinely new request — which may be the intended behavior — and confirm the team has consciously decided the expiry window is long enough relative to realistic client retry timing, not just short enough to keep storage costs low.

Out-of-order delivery. For queue-based systems in particular, deliver two related messages (for example, "order created" followed by "order cancelled") out of sequence, and separately, deliver a duplicate of one of them interleaved with the other. This tests whether the dedup logic is sensitive to ordering assumptions that at-least-once, non-FIFO delivery does not guarantee.

Pseudocode for the two hardest cases — concurrent duplicate and retry-after-server-completed — illustrates the shape these tests take in practice:

def test_concurrent_duplicate_charge_executes_once(test_client, payment_gateway_stub):
    key = generate_idempotency_key()
    body = {"amount": 34000, "currency": "usd", "customer": "cus_test_1"}

    # Fire both requests concurrently, not sequentially
    with ThreadPoolExecutor(max_workers=2) as pool:
        future_a = pool.submit(test_client.post, "/charges", json=body,
                                headers={"Idempotency-Key": key})
        future_b = pool.submit(test_client.post, "/charges", json=body,
                                headers={"Idempotency-Key": key})
        response_a, response_b = future_a.result(), future_b.result()

    charges = payment_gateway_stub.charges_for_customer("cus_test_1")
    assert len(charges) == 1, f"Expected exactly one charge, found {len(charges)}"
    assert {response_a.status_code, response_b.status_code}.issubset({200, 409})

def test_retry_after_response_lost_does_not_double_charge(test_client, response_dropping_proxy):
    key = generate_idempotency_key()
    body = {"amount": 34000, "currency": "usd", "customer": "cus_test_2"}

    # First attempt: server processes fully, proxy drops the response before
    # it reaches the client, simulating a lost-response timeout.
    response_dropping_proxy.drop_next_response()
    with pytest.raises(ClientTimeoutError):
        test_client.post("/charges", json=body, headers={"Idempotency-Key": key})

    # Client retries with the same key, as real retry logic would.
    retry_response = test_client.post("/charges", json=body,
                                       headers={"Idempotency-Key": key})

    charges = payment_gateway_stub.charges_for_customer("cus_test_2")
    assert len(charges) == 1
    assert retry_response.status_code == 200

These are illustrative test structures, not a specific framework's API — the pattern (drop the response but let server-side processing complete, then retry) is the part worth adopting regardless of the testing tools already in use.

A Duplicate Charge on a Flaky Connection

The following is a hypothetical, illustrative scenario built to demonstrate the failure mechanism, not a report of a real incident at any specific company.

Initial situation. A subscription SaaS product processes upgrade payments through a mobile app. The upgrade flow is a single POST /subscriptions/upgrade call that authorizes the new plan's price difference, updates the subscription record, and triggers a confirmation email. The mobile client has a 15-second timeout and, on timeout, automatically retries the request once.

The hidden assumption. The engineering team assumed that because the endpoint used an idempotency key generated client-side, retries were already handled safely. The key was implemented and tested — but only for the case where the retry happens after the first attempt has already returned an error. Nobody had specifically tested what happens when the retry fires because the response to a successful first attempt was lost, since that scenario requires a slower or interrupted connection to actually occur rather than a normal test environment where responses always return promptly.

The technical/organizational cause. On a genuinely degraded connection — the scenario the 15-second timeout exists to handle — the request reaches the server, the payment authorizes successfully, and the response begins its return trip just as the connection degrades further, arriving late enough that the client has already timed out and fired its automatic retry. Because the idempotency key was generated once per user tap of the "Upgrade" button and correctly reused on the automatic retry, the dedup layer worked exactly as designed at the API layer: the second request was correctly recognized as a duplicate and did not create a second charge. But the confirmation email was sent from a downstream event handler triggered by "subscription upgraded," which fired once per underlying database write — and because the idempotency check happened before that write, both times the request handler ran, it briefly appeared to the event system as though the write had happened again due to how the ORM's change-detection was implemented, generating two "subscription upgraded" events for one actual upgrade.

The consequence. No double charge occurred — the payment-layer idempotency key did its job. But customers received two nearly identical "Your subscription has been upgraded" emails, several minutes apart, for a single action. Support received a wave of "did this actually go through twice?" tickets that consumed real time to answer, even though the underlying transaction was correct, because customers reasonably could not tell from two emails whether they had been billed twice.

The decision to be made. Whether to treat this as low-priority, since no financial harm occurred, or to treat it as a real defect, since it directly damages customer trust in the billing system precisely because it looks like the failure mode customers fear most (being double-charged) even when it isn't one.

The better approach. Idempotency needs to be enforced at the point where each observable side effect originates, not only at the API entry point. In this case, the email-sending event handler needed its own dedup check keyed on the underlying business event ID (the specific upgrade transaction), not on an assumption inherited from the request-level key that the write itself only happened once. The fix was not to add more retries or backoff — it was to add a second, narrower idempotency boundary around the specific side effect that was actually duplicating.

The Order That Shipped Twice

This is also a hypothetical, illustrative scenario, constructed to demonstrate a realistic e-commerce failure mode rather than a description of any specific real event.

Initial situation. An e-commerce platform uses a third-party fulfillment provider that communicates order status via webhooks. When an order is marked "ready to ship" internally, the platform calls the fulfillment provider's API to create a shipment; the provider later sends a webhook back confirming the shipment was created and providing the tracking number, which the platform stores and forwards to the customer.

The hidden assumption. The team assumed that because they were calling the fulfillment provider's "create shipment" endpoint only once per order in their own code, duplicate shipments were not architecturally possible — the risk, as they saw it, lived entirely on the provider's side, not theirs.

The technical/organizational cause. During a period of elevated latency on the fulfillment provider's side, the platform's own outbound HTTP call to "create shipment" timed out on the platform's end before the provider's response returned — but the provider had, in fact, received and successfully processed the request. The platform's order-processing worker, built on a standard retry-on-timeout policy applied uniformly across all outbound API calls in the codebase (a policy that made sense for read operations but had never been specifically reviewed for this particular write operation), retried the "create shipment" call. The fulfillment provider, whose own API did not require or support a client-supplied idempotency key for this endpoint, correctly did what it was asked twice: it created two separate shipments for the same order, each with its own tracking number and its own physical pick-and-pack instruction sent to the warehouse floor.

The consequence. Two shipments went out for a single order. The financial cost was real but modest (duplicate shipping and duplicate inventory pulled), but the customer experience cost was worse — a customer receiving two boxes for one order, one containing a genuine surprise duplicate item, reasonably assumes something is systemically wrong with the platform's order accuracy, and the resulting refund-and-return process was more expensive in support time than the shipping cost itself.

The decision to be made. Whether the fix belonged on the platform's side (add idempotency protection to the outbound call, since the provider's API did not offer it) or should wait on the fulfillment provider to add idempotency key support to their endpoint.

The better approach. Waiting for a third party to fix their API is not a plan a platform's own reliability can depend on. The platform added its own idempotency safeguard on its side of the integration: before calling "create shipment," it checked a local table for an existing successful shipment record tied to that order ID, and treated a duplicate attempt within a defined window as a signal to first query the provider's API for the order's current shipment status rather than blindly creating a new one. This is the general pattern for any integration with a third party whose own idempotency support is unknown or absent: the protection has to live on the side that controls the retry, because you cannot assume the other side has built the safeguard you need.

The Shipment That Existed Before the Warehouse Knew About It

This third scenario is likewise hypothetical and illustrative, chosen specifically to demonstrate that this failure class is not limited to payments or e-commerce checkout flows.

Initial situation. A logistics and freight-visibility SaaS platform ingests tracking-status events from multiple carrier partners via a shared event bus, using a standard at-least-once queue configuration. Each event ("package picked up," "package in transit," "package delivered") triggers an update to the shipment's status record and, for the "delivered" event specifically, triggers a webhook out to the platform's own customers notifying them their shipment arrived.

The hidden assumption. The engineering team assumed that because each status event carried a monotonically increasing sequence number from the carrier, simply checking "is this sequence number higher than the last one we recorded for this shipment" was sufficient protection against redelivery — a reasonable-sounding heuristic that quietly assumed events would always redeliver with the same sequence number they originally carried.

The technical/organizational cause. During a partial outage on one carrier partner's side, that carrier's own systems retried publishing a batch of "delivered" events after recovering — but its retry path regenerated the events from source records rather than replaying the exact original messages, and the regenerated events carried new sequence numbers, higher than the ones already recorded, because the carrier's internal counter had continued to advance during the outage. The "is this sequence number higher" check, built to filter out simple redelivery of identical messages, had no way to recognize these as duplicates of an event it had already processed, because from its point of view they legitimately looked newer.

The consequence. Customers who had already received a "your package was delivered" notification received a second, identical one — sometimes hours later — for the same shipment. On its own this is a minor annoyance rather than a financial loss, but it eroded confidence in the accuracy of the platform's real-time tracking data specifically, which was the product's core value proposition to its own customers; a tracking platform whose "delivered" notification cannot be trusted to mean "delivered, once, definitively" undermines the entire reason a business pays for the service.

The decision to be made. Whether to trust the carrier's sequence numbers at all as a deduplication mechanism, or to build deduplication based on something the platform itself controlled.

The better approach. Sequence numbers and other identifiers supplied by an external party are a hint, not a guarantee, unless that party has explicitly documented them as stable and safe to dedupe on — and in this case, the carrier's documentation made no such promise. The platform moved to fingerprint-based deduplication: hashing a normalized combination of shipment ID, event type, and event timestamp (truncated to a coarse enough granularity to tolerate minor clock differences between retries) rather than trusting an externally generated sequence number as the sole source of truth for "have we seen this before." This is a broader lesson that generalizes past logistics: deduplication keys supplied by an upstream system are only as reliable as that system's own documentation says they are, and where the documentation is silent or ambiguous, the safer default is to build a dedup key from data your own side controls the semantics of.

Metrics That Reveal Duplication Before Customers Do

Idempotency defects are unusually good at hiding, because the retry itself often "succeeds" from a shallow monitoring point of view — the second charge attempt returns a 200, the second shipment creation returns a 201, the second email send reports as delivered. None of the standard error-rate dashboards fire, because nothing errored. The signal that duplication happened lives one level deeper than request success or failure: it lives in the relationship between how many requests came in and how many distinct business outcomes resulted.

This is precisely why duplication so often reaches production undetected for a stretch of time before anyone notices a pattern, rather than being caught on day one — every individual request, viewed on its own, looks correct. A monitoring approach built around per-request success rates will never surface this class of defect, no matter how sensitive its alerting thresholds are, because the defect is not that any single request failed. It is that two requests that should have collapsed into one outcome did not. Catching that requires metrics that are explicitly relational rather than metrics that describe one request in isolation.

A few metrics worth instrumenting specifically for this purpose, distinct from general API health monitoring:

Idempotency key hit rate — the proportion of requests to a protected endpoint that match an existing key rather than creating a new one. This number should be low but nonzero under normal operation, reflecting genuine client retries; a hit rate near zero across a system with known-flaky clients (mobile apps on cellular networks, for instance) is itself a warning sign that the idempotency key mechanism might not be wired up correctly on the client side, since real-world retry behavior should be producing at least occasional hits.

Reconciliation mismatch rate — for financial and inventory operations specifically, the count of cases where an internal record (number of charges recorded) diverges from an external system of record (number of charges actually settled with the payment processor). This is often the most reliable way to catch duplication that slipped past every other safeguard, precisely because it compares against a source of truth outside the system that might be the one generating the duplicates.

Downstream side-effect count per logical operation — specifically tracking, for a sample of operations, how many emails, webhooks, or fulfillment calls were triggered per single logical order or transaction, rather than only tracking aggregate volume. A shift in this ratio from close to 1.0 upward is a direct signal of the composite-operation failure mode discussed earlier, where the entry point is protected but a downstream side effect is not.

Retry rate versus duplicate-detected rate — tracking how often clients actually retry (visible from request headers, repeated idempotency keys, or repeated client-generated request IDs) against how often the server's dedup layer actually engages. A gap between these two — clients clearly retrying, but the dedup layer rarely engaging — suggests the dedup key scoping, expiry window, or matching logic has a gap letting retries through as if they were new requests.

Chart 1 — Where Duplicate Requests Typically Originate in Distributed Systems

Recommended chart type: horizontal bar chart. Categories on the Y axis (source of duplication); illustrative share of incidents on the X axis.

This distribution is illustrative — based on the common architectural failure patterns described in the delivery-semantics documentation and engineering writing referenced throughout this article, not a measured statistic from any specific dataset or industry survey. It is included to show the relative proportions worth planning test coverage around, not to be cited as a benchmark. What it demonstrates: client-side retries and infrastructure-level redelivery together account for the large majority of realistic duplication sources, which is why both need dedicated test scenarios rather than treating "duplicate testing" as a single generic checkbox.

Chart 2 — Relative Cost of a Duplicate-Charge Defect by Stage of Detection (Illustrative)

Recommended chart type: vertical bar chart. Detection stage on the X axis; relative cost multiplier on the Y axis, normalized to the earliest detection stage as 1x.

These multipliers are illustrative, not measured figures for any real organization or dataset — they are included to visualize a widely discussed general engineering principle (defects found later cost disproportionately more to resolve) applied specifically to the duplicate-charge scenario used throughout this article. Actual costs vary enormously by organization, payment volume, and dispute-handling process, and no specific study is being cited here. What it demonstrates: the cost gap is not linear — it accelerates sharply once a duplicate reaches a customer, because chargeback disputes carry direct fees, potential card-network penalties for high dispute rates, and support time on top of the refund itself, none of which apply when the same defect is caught by a test case before release.

How Much Rigor You Need Depends on Where You Are

Not every organization needs the same depth of idempotency infrastructure, and treating a five-person startup's payment flow with the same rigor as a regulated enterprise's is usually a sign of misallocated engineering time in one direction or the other.

The point of this comparison is not that a startup should feel behind for not having enterprise-grade dedup infrastructure. It is that a startup should be deliberate about which two or three endpoints get real idempotency protection and testing — almost always payment and account creation — rather than either skipping the topic entirely or trying to build a fully general solution before there is enough traffic or enough integrations to justify it.

A Framework for This Quarter, Not Someday

The following is a practical sequence a team can realistically execute over one quarter, built specifically for this article rather than adapted from a generic testing checklist.

Step 1 — Inventory every mutating endpoint that has an external or financial side effect. Payment captures, order creation, account creation, subscription changes, anything that sends an email or SMS, anything that provisions a resource, anything that calls a third-party API with a side effect on their end. This list is usually shorter than engineering teams expect — most systems have a handful of genuinely high-risk mutating operations and a long tail of lower-stakes ones.

Step 2 — For each endpoint on that list, classify it by consequence of duplication, not by traffic volume. A rarely called endpoint that triggers an irreversible external action (a wire transfer, a legal notice, a physical shipment) deserves more scrutiny than a high-traffic endpoint whose duplication is merely wasteful but harmless (re-triggering a cache refresh, for instance).

Step 3 — For the top-priority endpoints, document the idempotency contract explicitly. What key, if any, does the client provide? What is the dedup window? What happens if the same key arrives with different parameters? What happens if two copies arrive concurrently? This should be a written artifact, not tribal knowledge, because it is the specification the test suite in the next step is written against.

Step 4 — Build the duplicate/replay/partial-failure test suite against that contract, covering, at minimum: exact sequential duplicate, exact concurrent duplicate, near-duplicate with a different key, retry after a response is lost post-success, replayed webhook or redelivered message, and — for any composite operation — interrupted-then-retried partial execution.

Step 5 — Instrument the metrics from the previous section (idempotency key hit rate, reconciliation mismatch rate, downstream side-effect count per operation) for the same priority endpoints, before assuming the test suite alone is sufficient — tests catch what you thought to test for; monitoring catches what you didn't.

Step 6 — Run a deliberate chaos exercise against at least one high-priority flow, either in staging or, for teams with the operational maturity for it, in production with careful scoping: manually replay a webhook, manually redeliver a queue message, or use a proxy to simulate a lost response, and confirm the system behaves as documented in step 3, not merely as assumed.

Step 7 — Extend the contract-and-test pattern to the next tier of endpoints, and put idempotency contract documentation into the standard checklist for any new mutating endpoint going forward, so this becomes a design-time question rather than a recurring cleanup project.

This sequence deliberately does not try to cover every endpoint in the first pass. The goal for one quarter is a small number of genuinely well-protected, well-tested, well-monitored endpoints — the ones where duplication is expensive — rather than shallow, unverified coverage spread across everything at once.

When Not to Chase Perfect Idempotency

It is worth being honest about the limits of this discipline, because treating idempotency as a universal requirement for every endpoint is its own kind of mistake.

Operations that are genuinely incremental by design — "add one item to cart," "increment a view counter," "append a log line" — are not supposed to be idempotent in the sense discussed throughout this article, because their entire purpose is to accumulate, not to converge to a fixed end state. Forcing a dedup key onto an intrinsically additive operation either breaks its actual purpose or adds meaningful complexity and latency for a risk that does not really exist there. The relevant design question for these operations is not "how do we make repeats safe" but "how do we make each individual call unambiguous about how much it should add," which is a different problem.

Similarly, low-traffic, internal-only, easily reversible operations — an internal admin tool used by three employees to relabel a support ticket, for instance — rarely justify the engineering investment of a full idempotency-key infrastructure, even though technically a double-click could cause a duplicate relabel. The cost of the occasional manual cleanup is lower than the cost of building and maintaining protection nobody outside the team will ever notice was missing.

There is also a latency cost to dedup infrastructure that is worth acknowledging honestly rather than treating idempotency as free: a dedup check that requires a database round trip before every mutating request adds real latency to the critical path, and for extremely high-throughput, low-value operations, that tradeoff may not be worth making. The discipline this article argues for is not "add idempotency keys everywhere." It is "make a deliberate, tested decision about which operations need this protection, instead of an accidental one."

Frequently Asked Questions

Is idempotency only a concern for payment systems? No, though payments are where the consequences are most visible and most immediately expensive. Any operation with an external, hard-to-reverse, or customer-visible side effect — account creation, provisioning a cloud resource, sending a legally significant notification, creating a physical shipment — carries the same risk. Payments simply make the cost of getting it wrong obvious in a way that a duplicated internal log entry does not.

Do GET requests ever need duplicate-request testing? Generally no in the sense discussed here, because a correctly implemented GET should have no side effects to duplicate. The exception worth watching for is a GET endpoint that was implemented carelessly and triggers a side effect anyway — a "mark notification as read" action implemented as a GET, for example, which is a design smell independent of the idempotency discussion, but one that duplicate-request testing tends to surface.

How long should a deduplication window actually be? Long enough to cover the realistic retry behavior of your actual clients, not a round number picked for convenience. A mobile client that can retry hours after the original attempt (because a user closed and reopened the app) needs a window measured in hours, not minutes; a server-to-server integration with tight, immediate retry logic may only need a window of seconds to minutes. This should be derived from client behavior, not assumed.

Can idempotency keys themselves become a security or privacy problem? Yes, in a specific way worth guarding against: an idempotency key that is predictable (a sequential integer, a simple hash of user ID and timestamp) can allow one party to guess or collide with another party's key, particularly if the dedup table is not properly scoped per account, as discussed earlier. This is one of the reasons vendors like Stripe explicitly recommend high-entropy, randomly generated keys such as UUIDs rather than anything derived predictably from user data.

Is "exactly-once delivery" ever something a team should just trust and stop testing for? No, even where a provider offers an exactly-once delivery feature. As the Pub/Sub example earlier shows, these guarantees come with documented exceptions — specific subscription types, regional boundaries, publisher-side retry behavior — that fall outside the guarantee entirely. The guarantee reduces how often you need to think about duplication; it does not eliminate the need to test for the cases it explicitly does not cover.

How is this different from general regression testing or load testing? Regression testing generally verifies that a single request still produces the correct result after a code change. Load testing verifies the system holds up under volume. Neither one, by default, deliberately sends the same logical request more than once and checks the relationship between requests — that is a distinct dimension, orthogonal to both, and needs its own dedicated test cases rather than being assumed to be covered by either.

Does a database unique constraint make an idempotency key unnecessary? No, and the two solve different problems. A unique constraint stops a duplicate write from ever committing, but it typically does so only after the operation has already attempted to run — which is too late if the operation involves an external side effect, such as authorizing a card with a payment processor, that already happened before the database rejected the duplicate row. An idempotency key checked before the operation begins can avoid attempting that external side effect a second time at all. The constraint is a backstop against the key check failing or being bypassed, not a substitute for it.

Should idempotency testing be the QA team's responsibility or the backend team's? In practice it works best as a shared contract rather than something owned exclusively by either side. Backend engineers are usually best positioned to implement the mechanism correctly (key scoping, storage, constraint design), while QA is often better positioned to design and maintain the adversarial test scenarios — concurrent duplicates, lost-response retries, replayed webhooks — because that testing mindset of deliberately trying to break an assumption is exactly what the discipline requires, and it is easy for the engineer who built the mechanism to only test the cases they already thought to defend against.

How QAtronic Approaches Retry-Safety Testing

Idempotency failures rarely show up in a demo, a staging walkthrough, or a standard regression pass, because all three tend to send each request exactly once under clean network conditions — the one scenario where duplication cannot occur. QAtronic builds duplicate, replay, and partial-failure test coverage as a distinct layer of a QA strategy for teams whose systems process payments, webhook events, or queued messages: mapping which mutating endpoints carry real consequences if repeated, defining the idempotency contract for each one, and building the automated test cases and monitoring signals that verify the contract holds under the retry conditions real clients actually produce, not just the ones a happy-path test plan happens to exercise.

The Principle to Carry Back to Your Team

Idempotency is not a property that emerges automatically from a well-drawn architecture diagram, a message queue with a reassuring name, or a payment provider's SDK. It is a specific, testable contract that has to be defined deliberately for each operation that matters, implemented correctly at every layer where a side effect occurs, and verified the same way correctness is verified — by actually sending the second request, not by inferring that it would probably be fine.

The distinction worth holding onto is between a system that has never been sent a duplicate request and a system that has been tested against one and passed. The first can look identical to the second in every demo, every staging review, and every clean-network manual test — right up until a real network, a real retry, or a real redelivery exposes the difference, usually in front of a customer.

The question worth taking back to your own team is a narrow one: pick the single mutating endpoint in your system where a duplicate would be most expensive — financially, reputationally, or operationally — and ask whether anyone can point to a specific, automated test that sends that exact request twice and verifies only one thing happened as a result. If the honest answer is no, that is the place to start, not because every endpoint needs this treatment immediately, but because that is the one where the answer already matters.