Building an Idempotent Webhook Handler

Every webhook handler needs to be idempotent. Most teams know this in the abstract and get it subtly wrong in practice, usually because the naive version looks correct in testing and only breaks under concurrent delivery, which local testing rarely produces.

This post covers why duplicates happen only briefly. The focus here is building a handler that's actually safe against them.

The naive version, and why it has a race condition

The obvious first attempt: check if you've seen the event ID, and if not, process it.

async function handleEvent(event: WebhookEvent) {
  const seen = await redis.get(`event:${event.id}`);
  if (seen) return;

  await processEvent(event);
  await redis.set(`event:${event.id}`, '1', 'EX', 86400);
}

This works fine in a single-threaded test. It breaks the moment two workers process the same event concurrently, which happens more often than people expect: a provider's retry can arrive while the original attempt is still in flight, or you can have multiple worker processes pulling from the same queue.

Worker A: checks redis, sees nothing
Worker B: checks redis, sees nothing (A hasn't written yet)
Worker A: processes event
Worker B: processes event  ← duplicate, both checks passed

The check and the claim aren't atomic, so there's a window where two workers can both pass the check before either has recorded that it's handling the event.

The fix: commit the marker and effects together

An atomic insert prevents two workers from claiming an event. It is only safe when the claim and the work commit together. This PostgreSQL example uses pg (node-postgres); create the tables once in a migration. orders is your existing business table.

CREATE TABLE processed_events (
  account_id TEXT NOT NULL,
  provider TEXT NOT NULL,
  event_id TEXT NOT NULL,
  processed_at TIMESTAMPTZ NOT NULL DEFAULT now(),
  PRIMARY KEY (account_id, provider, event_id)
);

CREATE TABLE webhook_outbox (
  account_id TEXT NOT NULL,
  provider TEXT NOT NULL,
  event_id TEXT NOT NULL,
  effect_type TEXT NOT NULL,
  payload JSONB NOT NULL,
  delivered_at TIMESTAMPTZ,
  PRIMARY KEY (account_id, provider, event_id, effect_type)
);

Verify the raw request signature before calling this handler. Resolve accountId from your trusted endpoint configuration, not an untrusted payload field. The example handles a verified Stripe payment_intent.succeeded event.

import { Pool } from 'pg';
const pool = new Pool({ connectionString: process.env.DATABASE_URL });

type PaymentSucceeded = {
  id: string;
  data: { object: { id: string } };
};

async function handleEvent(event: PaymentSucceeded, accountId: string) {
  const client = await pool.connect();
  try {
    await client.query('BEGIN');
    const claim = await client.query(
      `INSERT INTO processed_events (account_id, provider, event_id)
       VALUES ($1, 'stripe', $2)
       ON CONFLICT DO NOTHING RETURNING event_id`,
      [accountId, event.id]
    );
    if (claim.rows.length === 0) {
      await client.query('COMMIT');
      return; // A committed transaction already applied this event.
    }

    const order = await client.query(
      `UPDATE orders SET payment_status = 'paid'
       WHERE account_id = $1 AND payment_intent_id = $2 RETURNING id`,
      [accountId, event.data.object.id]
    );
    if (order.rows.length === 0) throw new Error('Order not ready; retry later');
    // Schedule the external effect in the SAME transaction. Do not send here.
    await client.query(
      `INSERT INTO webhook_outbox
       (account_id, provider, event_id, effect_type, payload)
       VALUES ($1, 'stripe', $2, 'send_receipt', $3::jsonb)`,
      [accountId, event.id, JSON.stringify({ paymentIntentId: event.data.object.id })]
    );
    await client.query('COMMIT');
  } catch (error) {
    await client.query('ROLLBACK');
    throw error; // Return a failure or retry the durable job; never acknowledge success.
  } finally {
    client.release();
  }
}

Concurrent inserts wait for each other. If the first transaction commits, the second sees the conflict and skips. If the first rolls back, the second can claim and complete the work. A crash after commit is safe too: the retry sees the marker and the outbox effect is already durable. Every business write must use this same transaction connection.

External effects need an outbox worker

A worker claims an undelivered outbox row with a lease, sends the external request, and then marks the row delivered. Recover expired leases after crashes. Use a stable key such as account:stripe:event:send_receipt for the external service's idempotency mechanism on every retry.

There is still a crash window after the remote service accepts the request and before your worker marks it delivered. The outbox provides durable, at-least-once execution; the remote service must deduplicate the stable key within its supported retention window. If it cannot, use a reconciliation process or a naturally idempotent operation. Do not promise exactly-once email, payment, or API calls merely because you used an outbox.

Scoping the claim correctly

Whatever ID you dedup on has to actually be unique to the event, not to the event type. A common mistake: deduplicating on something like ${orderId}-updated, which collapses multiple genuinely distinct updates to the same order into a single claim.

// Wrong: collapses every update to this order into one claim
const claimKey = `order-updated:${orderId}`;

// Right: unique per actual event, not per resource
const claimKey = event.id; // Stripe's evt_xxx, GitHub's delivery UUID, etc.

Use whatever unique identifier the provider actually assigns to the delivery, not something you construct yourself from the payload's contents.

Cleaning up claim records

Keep deduplication keys through the longest supported resend or replay window, plus any recovery backlog and safety margin. Stripe supports manual resends for 15 days in the Dashboard and 30 days with the CLI. A relay may retain events longer: Watchtower's Bastion plan defaults to 90 days. Deleting keys after 24–72 hours would let legitimate later replays repeat effects.

If historical events can be imported or replayed indefinitely, retain keys indefinitely or define an explicit archival cutoff that your handler enforces. Provider retry expiry alone does not mean a duplicate can no longer arrive.

Watchtower's ingest deduplication reduces repeat traffic, but deliberate manual replay and failures between your endpoint's commit and its HTTP response still require an idempotent handler. See the supported providers doc.