Retries, Backoff, and Jitter: How to Retry Without Causing an Outage

Retrying a failed request is the most obvious reliability measure there is, and one of the easiest to turn into an outage. The failure mode is well known enough to have a name — the thundering herd — and the fix is three lines of code that most retry loops are missing.

The naive loop

// Please do not ship this.
async function fetchWithRetry(url, attempts = 3) {
  for (let i = 0; i < attempts; i++) {
    try {
      return await fetch(url);
    } catch (err) {
      if (i === attempts - 1) throw err;
    }
  }
}

Two problems. It retries immediately, and it retries everything.

Immediate retries triple your traffic against a service that is already struggling — the retries arrive while the original overload is still in progress. And if the service is down rather than flaky, all three attempts fail, so the caller waits three timeouts to learn what the first one already told it. And it can't tell errors apart: the moment someone adds if (!res.ok) throw inside the try, it starts retrying 400 Bad Request too — a request that will be rejected identically every time — wasting two more round trips before surfacing a bug the caller needed to see immediately.

Exponential backoff, and why it isn't enough

Waiting longer between attempts gives the dependency time to recover: 1s, 2s, 4s, 8s.

That helps with duration but not with shape. When a service blips, every client fails at roughly the same moment, so every client's 1-second timer starts at roughly the same moment — and fires at roughly the same moment. The retries arrive as one spike, the service falls over again, and the next spike is scheduled for 2 seconds later:

The top timeline is exponential backoff done "correctly". The retries are spaced out in time and completely synchronised in phase. The service sees idle stretches punctuated by bursts it can't absorb.

Jitter is the actual fix

Randomise the wait:

const sleep = (ms) => new Promise((r) => setTimeout(r, ms));

async function retry(fn, { attempts = 5, base = 200, cap = 20_000 } = {}) {
  for (let i = 0; ; i++) {
    try {
      return await fn();
    } catch (err) {
      if (i >= attempts - 1 || !isRetryable(err)) throw err;
      const ceiling = Math.min(cap, base * 2 ** i);
      await sleep(Math.random() * ceiling);        // full jitter
    }
  }
}

That's "full jitter": wait a random amount between zero and the exponential ceiling. The same retry budget, spread across the window instead of concentrated at its edge, which is the difference between the two timelines above.

Variants worth knowing:

Full jitter is the default worth reaching for. Equal jitter is a reasonable choice when a too-short first retry is genuinely harmful.

Also cap the ceiling. Uncapped doubling reaches absurd waits by attempt fifteen, and a retry two hours later is not a retry — it's a scheduled job in disguise.

One blip, three policies

Numbers make the difference concrete. A service normally handles 1,000 requests a second with room for about 1,500. It restarts and refuses connections for two seconds. Clients make up to three attempts.

• Immediate retries. During the outage, every request fails and is retried twice at once: ~3,000 requests a second hit a service that's down. When it comes back, the first second brings new traffic plus the tail of retries — well over its 1,500 capacity. It fails again. The two-second blip becomes a minute of flapping.
• Exponential backoff, no jitter. The 2,000 requests that failed during the outage all come back at exactly +1 s and +2 s after their failures, on top of the normal 1,000 a second. The service recovers to a queue of synchronised retries, around 2,000–3,000 a second, and struggles to catch up.
• Backoff with full jitter. Those same retries are spread uniformly over the next few seconds: an extra few hundred requests a second on top of the normal 1,000, inside the 1,500 capacity. The service absorbs them and the blip is over in a little more than two seconds.

Same number of retries in all three cases. Only the timing changed, and the timing decided whether the outage was two seconds or a minute.

Retry only what is retryable

Retrying a request that will never succeed wastes capacity and delays the error the caller needs to see.

The timeout row is the subtle one: a timeout means you don't know whether the request was processed. Retrying a POST /payments that actually succeeded charges the customer twice.

Idempotency makes retries safe

A retry is only safe if repeating it is harmless. GET, PUT, and DELETE are naturally idempotent; POST is not. The fix is an idempotency key — a client-generated ID the server records:

await fetch('/payments', {
  method: 'POST',
  headers: { 'Idempotency-Key': paymentAttemptId },  // stable across retries
  body: JSON.stringify(payment),
});

The server stores the outcome against that key. A second request with the same key returns the original result instead of charging again. Crucially, the key must be generated once per logical operation, not once per attempt — regenerating it inside the retry loop defeats the entire mechanism.

Retry budgets and amplification

The dangerous case is retries stacking across layers. If every hop retries three times, a request through four services can become 81 requests at the bottom. Under load, that amplification is what turns a slow dependency into a total collapse.

Two guards:

• Retry at one layer. Usually the one closest to the user, or the one that knows whether the operation is idempotent. Not every layer.
• Retry budgets. Cap retries as a fraction of overall traffic — say 10%. When failures are widespread, retries are dropped rather than multiplied, because a broadly failing dependency will not be fixed by more requests.

A retry budget is a token bucket that refills from successful traffic:

class RetryBudget {
  constructor(ratio = 0.1, minPerSecond = 10) {
    this.ratio = ratio;            // retries allowed per request
    this.tokens = minPerSecond;
    this.min = minPerSecond;
    setInterval(() => { this.tokens = Math.max(this.tokens, this.min); }, 1000);
  }
  onRequest() { this.tokens = Math.min(this.tokens + this.ratio, 1000); }
  tryRetry() {
    if (this.tokens < 1) return false;
    this.tokens -= 1;
    return true;
  }
}

Every request deposits a tenth of a token; every retry spends a whole one. When things are healthy, the occasional retry always finds a token. When a dependency fails broadly, retries quickly use up the deposits and further retries are refused — total traffic can rise by at most 10%, instead of tripling. The small floor keeps low-traffic clients able to retry at all. gRPC and service meshes like Envoy implement exactly this idea.

Deadlines, not just attempt counts

"Three attempts" says nothing about how long the caller waits. Three attempts with 10-second timeouts and backoff can keep a user staring at a spinner for over half a minute. Bound the whole operation instead:

async function retryWithDeadline(fn, { deadlineMs = 3000, base = 100 } = {}) {
  const deadline = Date.now() + deadlineMs;
  for (let i = 0; ; i++) {
    const remaining = deadline - Date.now();
    try {
      return await fn({ signal: AbortSignal.timeout(remaining) });
    } catch (err) {
      const wait = Math.random() * base * 2 ** i;
      if (!isRetryable(err) || Date.now() + wait >= deadline) throw err;
      await sleep(wait);
    }
  }
}

Each attempt gets only the time that's left, and the loop gives up when another wait would overrun the deadline. Better still, propagate the deadline downstream — pass the remaining budget in a header such as x-request-deadline, as gRPC does with its deadline field — so a service three hops away doesn't keep working on a request whose caller has already given up. See AbortController and cancellation for the client side of this.

Listen when the server tells you

A 429 or 503 with a Retry-After header is the server doing your backoff calculation for you. Honour it, and add a little jitter on top so all the clients told "30 seconds" don't return in the same millisecond:

function retryDelay(res, attempt, base = 200) {
  const header = res?.headers?.get('Retry-After');
  if (header) {
    const seconds = Number(header);
    const ms = Number.isFinite(seconds)
      ? seconds * 1000
      : new Date(header).getTime() - Date.now();   // HTTP-date form
    return Math.max(0, ms) + Math.random() * 1000;
  }
  return Math.random() * base * 2 ** attempt;
}

The server side of the same contract: when overloaded, fail fast with 503 and a Retry-After, instead of accepting work you'll time out on. A quick, honest rejection is what lets well-behaved clients back off — see load shedding and rate limiting.

Hedged requests: retrying for latency

Retries usually respond to failure. Hedging responds to slowness: if a request hasn't answered by the 95th-percentile latency, send a second copy to another replica and take whichever finishes first. It cuts tail latency dramatically for a few percent of extra load, because the slow responses are usually slow for reasons specific to one server (a GC pause, a busy disk). Only hedge idempotent reads, cancel the loser, and include hedges in the retry budget so they can't multiply load during an incident.

Retries in background jobs

Queues and job runners retry for you, and the same rules apply with longer timescales. BullMQ, Sidekiq and SQS all support exponential backoff per job; configure it rather than relying on immediate redelivery. Add jitter for jobs that fail together (all of them hitting the same broken API). And set a final destination — a dead-letter queue or a failed state that alerts — so a job that will never succeed stops consuming retries and gets a human's attention.

Retries and circuit breakers

Retries handle the individual request that hit a blip. They're the wrong tool for a dependency that's genuinely down — that's what a circuit breaker is for. The two compose: the breaker decides whether it's worth calling at all, and the retry policy handles transient failures while the breaker is closed.

Testing retry behaviour

Retry logic is rarely exercised until an incident, which is the worst time to discover it retries 400s or never gives up. Test it on purpose:

• Unit-test the policy with a fake clock and a fake fn that fails N times: assert the number of attempts, that waits fall within the jitter range, and that non-retryable errors throw on the first try.
• Inject faults in staging — a proxy that returns 503 for 20% of requests, or adds two seconds of latency — and watch the downstream request rate. It should rise modestly, not multiply.
• Graph retries as their own metric. A rising retry rate is often the earliest sign of a dependency degrading, well before error rates move.

The checklist

• Exponential backoff with a cap.
• Jitter — this is the one people skip, and it matters most.
• Retry only retryable errors.
• Idempotency keys for anything that mutates state.
• A bounded number of attempts, and a bounded total deadline.
• Retry at one layer, with a budget.
• A circuit breaker for the case where nothing is going to work.
• Respect Retry-After when the server sends it.
• A total deadline, propagated downstream, rather than an attempt count alone.
• Retries visible as their own metric, so a degrading dependency shows up before it fails outright.