Idempotency Keys: Making Retries Safe

Every guide to retries and backoff comes with a warning: only retry idempotent operations. GET is fine. PUT of a full resource is fine. POST /charges is not — retry it and a customer pays twice.

But the operations you most need to retry are exactly the ones with side effects. Idempotency keys make a non-idempotent endpoint behave idempotently, one request at a time.

The ambiguous failure

A client sends a request and gets back a timeout. There are three possibilities:

1. The request never reached the server.
2. The server received it, and crashed or failed before finishing.
3. The server finished the work, and the response was lost.

The client can't tell them apart. Not retrying is wrong in cases 1 and 2. Retrying blindly is wrong in case 3. The fix has to live on the server, because only the server knows what it already did.

The protocol

The client generates a unique key — a UUID — once per logical operation, before the first attempt, and sends it in a header on every attempt. The server remembers each key and the response it produced, and replays that response when it sees the key again.

POST /v1/charges
Idempotency-Key: 5d1e2b4a-7f3a-4c1e-9b2d-8a6f0e3c1d47
Content-Type: application/json

{ "amount": 4000, "currency": "usd", "source": "card_123" }

The key identifies the intent, not the HTTP request. Two different attempts at the same charge share a key. Two different charges — even for the same amount, to the same card — must not.

A server-side implementation

The core is a table with a unique constraint on the key:

CREATE TABLE idempotency_keys (
  key           text PRIMARY KEY,
  request_hash  text NOT NULL,
  status        text NOT NULL,        -- 'in_progress' | 'done'
  response_code int,
  response_body jsonb,
  created_at    timestamptz NOT NULL DEFAULT now()
);

async function handleCharge(req, res) {
  const key = req.get('Idempotency-Key');
  const hash = sha256(JSON.stringify(req.body));

  // 1. Claim the key. The unique index makes this atomic: exactly one
  //    concurrent request wins the insert.
  const claimed = await db.query(
    `INSERT INTO idempotency_keys (key, request_hash, status)
     VALUES ($1, $2, 'in_progress')
     ON CONFLICT (key) DO NOTHING
     RETURNING key`,
    [key, hash]
  );

  if (claimed.rowCount === 0) {
    const existing = await db.one(
      'SELECT * FROM idempotency_keys WHERE key = $1', [key]
    );
    if (existing.request_hash !== hash) {
      return res.status(422).json({ error: 'key reused with a different body' });
    }
    if (existing.status === 'in_progress') {
      return res.status(409).json({ error: 'request in progress, retry later' });
    }
    return res.status(existing.response_code).json(existing.response_body);
  }

  // 2. Do the work and record the result.
  const charge = await payments.charge(req.body);
  await db.query(
    `UPDATE idempotency_keys
     SET status = 'done', response_code = 201, response_body = $2
     WHERE key = $1`,
    [key, charge]
  );
  return res.status(201).json(charge);
}

The details that matter

Claim before you act. Checking "does this key exist?" and then inserting it is a race: two concurrent retries both see nothing and both proceed. The INSERT … ON CONFLICT DO NOTHING is the check and the claim in one atomic step.

Fingerprint the request. If a client bug reuses a key for a different payload, replaying the old response silently does the wrong thing. Store a hash of the body and reject mismatches. Hash a canonical form — keys sorted, whitespace removed — so the same payload serialised by two client versions doesn't look different.

Decide what "in progress" means after a crash. If the server dies between claiming the key and finishing, the record sits in in_progress forever and every retry gets 409. Either expire in-progress claims after a timeout, or — better — make the work itself resumable from recorded checkpoints.

Put the key and the side effect in the same transaction when you can. If the charge is a row in the same database, write it and mark the key done in one transaction, and the crash window disappears. When the side effect is an external API call, you can't, and you need that API to accept an idempotency key of its own — pass one derived from yours, so your retry becomes its retry.

Cache errors carefully. A 400 for a malformed body is deterministic; replaying it is correct. A 503 because a dependency was down is transient; caching it means the client can never succeed with that key. Only store responses the server would give again.

Expire keys. Stripe keeps them for 24 hours. Pick a window longer than any client's retry horizon, then delete — the table otherwise grows forever. Partitioning the table by day makes cleanup a cheap DROP TABLE of yesterday's partition instead of a slow DELETE that competes with live traffic; a background job that deletes in small batches works too.

Two retries at once, step by step

The most common misunderstanding is that retries arrive one after another. On mobile networks they often overlap: the first attempt is still running when the client times out and sends the second. Trace both through the handler:

1. Attempt 1 runs INSERT ... ON CONFLICT DO NOTHING and claims the key.
2. Attempt 2 runs the same insert a millisecond later. The unique index rejects it; rowCount is 0.
3. Attempt 2 reads the row: status = 'in_progress'. It returns 409.
4. Attempt 1 charges the card, updates the row to done, returns 201 — to a client that has already given up on it.
5. The client, having seen a 409, waits and retries. Attempt 3 finds done and replays the stored 201.

The charge happened exactly once, and the client ends up with the right answer. That 409 should carry a Retry-After, so clients wait a sensible interval instead of hammering the endpoint while the first attempt finishes.

Making the work resumable

The "stuck in progress" problem has a better fix than timeouts: break the request into atomic phases and record which phase completed. Each phase is a local database transaction; external calls sit between them and use their own idempotency keys.

async function handleCharge(key, req) {
  let rec = await keys.claimOrLoad(key, req);           // as before

  if (rec.phase === 'started') {
    await db.tx(async (t) => {
      const order = await t.one(
        'INSERT INTO charges (key, amount, status) VALUES ($1, $2, $3) RETURNING id',
        [key, req.amount, 'pending']);
      await keys.setPhase(t, key, 'charge_created', { chargeId: order.id });
    });
    rec = await keys.load(key);
  }

  if (rec.phase === 'charge_created') {
    // external call; the provider dedupes on our derived key
    const result = await provider.charge(req, { idempotencyKey: `${key}:provider` });
    await db.tx(async (t) => {
      await t.none('UPDATE charges SET status = $2, provider_id = $3 WHERE id = $1',
        [rec.data.chargeId, 'succeeded', result.id]);
      await keys.finish(t, key, 201, { id: rec.data.chargeId, status: 'succeeded' });
    });
  }
  return keys.response(key);
}

A crash anywhere leaves the key at the last completed phase. The next retry — or a background sweeper — picks up from there instead of starting over or giving up. Stripe has described building its own API this way, with the phases called recovery points.

Scope, storage and lifetime

Scope keys to the caller. Store (account_id, key), not just key. A UUID collision across customers is astronomically unlikely, but a client bug that sends the same hard-coded key for everyone is not — and without scoping, one customer could receive another customer's stored response.

Store the response you'd send. Status code, body, and any headers the client relies on, such as Location. Keep it small; if a response is large, store a reference and rebuild it.

Choose where the table lives. If the side effect is in a relational database, keep the keys table in the same database, so the key and the effect commit together. Redis is a reasonable home only for the "in progress" claim with a TTL, when the durable record is elsewhere.

Pick a retention window and say so. Document it in the API: "keys are remembered for 24 hours". After that, the same key is treated as new.

A reusable middleware

Once one endpoint needs this, others will. In Express, the pattern fits in a middleware that wraps any mutating route:

function idempotent({ ttlHours = 24 } = {}) {
  return async (req, res, next) => {
    const key = req.get('Idempotency-Key');
    if (!key) return next();                       // or 400 if you require it
    const scoped = `${req.user.accountId}:${key}`;
    const hash = sha256(req.method + req.path + JSON.stringify(req.body));

    const existing = await keys.claim(scoped, hash, ttlHours);
    if (existing?.mismatch) return res.status(422).json({ error: 'key reused' });
    if (existing?.inProgress) return res.status(409).set('Retry-After', '1').end();
    if (existing?.response) {
      return res.status(existing.response.status).json(existing.response.body);
    }

    const originalJson = res.json.bind(res);
    res.json = async (body) => {
      if (res.statusCode < 500) await keys.finish(scoped, res.statusCode, body);
      else await keys.release(scoped);             // let the client retry 5xx
      return originalJson(body);
    };
    next();
  };
}

app.post('/v1/charges', idempotent(), chargeHandler);

Note the method and path in the fingerprint: reusing a key across two different endpoints is a client bug and should be rejected, not replayed. There is also an IETF draft standardising the Idempotency-Key header for HTTP APIs; following its conventions makes your API behave the way client libraries increasingly expect.

Generating keys on the client

The server can only dedupe what the client labels correctly:

• Generate the key when the user acts, not when the request is sent. The "Pay" button click creates the key; every retry of that click reuses it.
• Persist it for long-running flows. If the app might be killed and restarted mid-payment, store the key with the pending operation so the retry after restart uses the same one.
• Derive it from the operation when you can. A job that charges a subscription for a billing period can use sub_123:2026-10 as its key, which survives any number of restarts without storing anything.
• Never create it inside the retry loop. A fresh key per attempt turns every retry into a new charge.

Testing it

The tests that matter all involve duplicates and timing:

• Send the same request twice sequentially: one effect, identical responses.
• Send it twice concurrently: one effect, one 201, one 409 (or both 201 after the retry).
• Same key, different body: 422, no effect.
• Kill the server between claiming the key and finishing: the next retry completes the work exactly once.
• Make the downstream fail with a 503: the key is released, and a retry can still succeed.

Common mistakes

Every row in that table has shipped in a real payments API at some point. None of them shows up in a test that sends one request at a time.

Idempotency beyond HTTP

The same shape shows up anywhere delivery is at-least-once:

• Message consumers. A queue will redeliver after a consumer crash. Store processed message IDs alongside the effect, in one transaction — the consumer side of the outbox pattern.
• Natural keys. Sometimes the data already has one. "Apply payment inv_42 to the ledger" can use a unique constraint on the invoice ID; no separate key table needed.
• Idempotent by construction. SET balance = 100 is safe to repeat; SET balance = balance + 10 is not. Where you can, design operations that state the desired end result instead of a delta.

Networks guarantee at-least-once at best. Exactly-once effects aren't something the network gives you; they're something you build on the receiving end — and an idempotency key is the smallest version of that. Pair it with retries, backoff and jitter on the client and an outbox on the server, and "the request timed out" stops being a question anyone has to answer by hand.