Exponential Backoff: The Right Way to Retry Failed Webhooks

When a webhook delivery fails, you have two options: give up or retry. Most production systems retry, but how you retry matters enormously. Retry too aggressively and you will overwhelm an already-struggling endpoint. Retry too slowly and you lose the event or make customers wait too long.

Exponential backoff is the standard solution. It is worth understanding how it works before you implement it.

Why naive retry logic fails

The simplest retry strategy: wait 5 seconds, try again. If that fails, wait another 5 seconds. Repeat until it works or you hit a limit.

This works fine when failures are brief and isolated. But webhook delivery failures usually happen because your destination endpoint is having trouble: high load, a deployment, a dependency outage. In those cases, retrying every 5 seconds:

1. Makes the problem worse - your retries add load to an endpoint that is already struggling
2. Wastes resources - you burn queue capacity and compute on requests that will not succeed yet
3. Creates thundering herd problems - if many sources are retrying at the same interval, they all hit the endpoint at the same time

The same logic applies at the provider level. If Stripe retries 10,000 subscriptions every 60 seconds, they will DDoS your server. They do not, because they use exponential backoff with jitter.

How exponential backoff works

The idea: each retry waits longer than the previous one, with the wait time growing exponentially.

Attempt 1: immediate (no wait)
Attempt 2: wait 1 minute
Attempt 3: wait 2 minutes
Attempt 4: wait 4 minutes
Attempt 5: wait 8 minutes
Attempt 6: wait 16 minutes (or cap)

The base formula is delay = base_delay * 2^(attempt - 1).

This means:

• If a failure is brief (30-second deploy), it will resolve by attempt 2 or 3
• If a failure is longer (hours-long outage), retries do not stack up and overwhelm the endpoint on recovery
• The total retry window spans hours, giving a reasonable chance of eventual delivery

Adding jitter

Pure exponential backoff has one remaining problem: if many events failed at roughly the same time (because the endpoint went down), they will all retry at the same scheduled times, creating simultaneous spikes.

The fix is jitter: randomise each retry delay within a range.

function getRetryDelay(attempt: number, baseMs = 60_000): number {
  const exponential = baseMs * Math.pow(2, attempt - 1);
  const cap = 2 * 60 * 60 * 1000; // 2 hours max
  const capped = Math.min(exponential, cap);

  // Add ±30% jitter
  const jitter = capped * 0.3 * (Math.random() * 2 - 1);
  return Math.max(baseMs, capped + jitter);
}

With jitter, events that failed at the same moment will retry at slightly different times, spreading the load evenly across the window.

Implementing retry logic with BullMQ

If you are using BullMQ for background job processing, exponential backoff is built in:

import { Queue, Worker } from 'bullmq';

const webhookQueue = new Queue('webhook-delivery', {
  connection: redisConnection,
});

// Add a job with exponential backoff
await webhookQueue.add(
  'deliver',
  { eventId, destination, payload },
  {
    attempts: 5,
    backoff: {
      type: 'exponential',
      delay: 60_000, // base delay: 1 minute
    },
    removeOnComplete: true,
    removeOnFail: false, // keep failed jobs for inspection
  }
);

// Worker processes jobs
const worker = new Worker(
  'webhook-delivery',
  async (job) => {
    const { destination, payload } = job.data;
    const response = await fetch(destination.url, {
      method: 'POST',
      headers: { 'Content-Type': 'application/json' },
      body: JSON.stringify(payload),
      signal: AbortSignal.timeout(10_000), // 10s timeout
    });

    if (!response.ok) {
      throw new Error(`Delivery failed: ${response.status}`);
    }
  },
  { connection: redisConnection }
);

BullMQ handles the retry scheduling and job state automatically. job.attemptsMade tells you which attempt you are on.

What to do when all retries are exhausted

After the final retry fails, you have a dead event. At this point you need to:

1. Notify the operator - email, in-app notification, Slack - someone needs to know
2. Keep the event - do not delete it; you need to be able to replay it
3. Provide a replay mechanism - let operators re-trigger delivery manually once the issue is resolved

The notification should include enough context to diagnose the problem: which destination failed, how many attempts were made, what the last error was, and a direct link to investigate.

worker.on('failed', async (job, error) => {
  if (job && job.attemptsMade >= job.opts.attempts!) {
    // Final attempt exhausted
    await notificationService.sendDeliveryFailed({
      workspaceId: job.data.workspaceId,
      eventId: job.data.eventId,
      destinationName: job.data.destination.name,
      attempts: job.attemptsMade,
      lastError: error.message,
    });

    // Mark delivery as permanently failed
    await db.deliveries.update({
      where: { id: job.data.deliveryId },
      data: { status: 'failed', failedAt: new Date() },
    });
  }
});

Circuit breakers: knowing when to stop retrying

Exponential backoff assumes eventual recovery. But what if a destination is permanently broken: wrong URL, deleted endpoint, a server that will never come back up?

A circuit breaker prevents you from burning retry cycles on a destination that is clearly not going to recover. The pattern:

• Track consecutive failures per destination
• After N consecutive failures (e.g. 5), "open" the circuit and stop attempting delivery to that destination
• After a cooldown period, try one request to see if it has recovered ("half-open")
• If it succeeds, close the circuit and resume normal delivery
• If it fails, stay open and reset the cooldown

async function shouldAttemptDelivery(destination: Destination): Promise<boolean> {
  if (destination.circuitState === 'closed') return true;

  if (destination.circuitState === 'open') {
    const cooldownMs = 30 * 60 * 1000; // 30 minutes
    const elapsed = Date.now() - destination.circuitOpenedAt.getTime();

    if (elapsed > cooldownMs) {
      // Transition to half-open: allow one probe request
      await db.destinations.update({
        where: { id: destination.id },
        data: { circuitState: 'half_open' },
      });
      return true;
    }

    return false; // still cooling down
  }

  // half_open: allow the probe
  return true;
}

async function handleDeliveryResult(
  destination: Destination,
  succeeded: boolean
) {
  if (succeeded) {
    // Reset circuit on success
    await db.destinations.update({
      where: { id: destination.id },
      data: {
        circuitState: 'closed',
        consecutiveFailures: 0,
        circuitOpenedAt: null,
      },
    });
  } else {
    const failures = destination.consecutiveFailures + 1;
    const threshold = 5;

    await db.destinations.update({
      where: { id: destination.id },
      data: {
        consecutiveFailures: failures,
        circuitState: failures >= threshold ? 'open' : 'closed',
        circuitOpenedAt: failures >= threshold ? new Date() : null,
      },
    });
  }
}

Retry budget: do not retry forever

Decide upfront how long you will keep trying. Common approaches:

• Fixed attempts (Stripe: up to 25 over 72 hours; GitHub: 3 attempts over a few hours)
• Time-bounded (retry for up to 24 hours, however many attempts that is)
• Hybrid (up to N attempts, but do not retry after T hours)

The right choice depends on how stale an event becomes. A payment.succeeded event is still worth processing 12 hours late. An event that triggers a real-time notification probably is not.

---

Retry logic looks simple until a partial outage leaves you with thousands of stuck jobs. The full pattern (exponential backoff with jitter, circuit breakers, dead-letter handling, and manual replay) is the same one battle-tested infrastructure uses.

Webhook Watchtower implements this pattern on top of your existing webhook setup: plan-dependent retries with fixed waits (1m, 5m, 30m, 2h; no jitter), circuit breakers per destination, notifications on exhaustion, and one-click replay for failed events. The retries and delivery docs cover exactly how the retry schedule works and what happens when attempts are exhausted.