Debugging Webhooks in Production: A Practical Approach

A customer emails to say an order didn't sync. Somewhere between the provider firing an event and your system reflecting it, something went wrong, and you have no idea which of the six steps in that chain is the problem. This is the moment most webhook debugging starts, and it goes a lot faster with a method than without one.

Start by finding the boundary

Every webhook failure sits on one side of a boundary: either the event never reached your system, or it arrived and something inside your system mishandled it. Everything else follows from figuring out which side you're on.

Provider  →  [ingest]  →  your queue  →  your handler  →  your database
          ↑                                             ↑
     did it get here?                          did it finish here?

Check the provider's own delivery log before anything else. Stripe, GitHub, and Shopify all keep one, and all show the exact response your endpoint sent back, including the status code and body. If the provider shows a successful 2xx delivery, the event definitely reached you, and the bug is downstream. If the provider shows a failed or missing delivery, you're debugging a network or endpoint problem, not application logic.

If it never arrived

A few things to check, roughly in order of likelihood:

The URL is wrong. Compare the URL configured in the provider's dashboard against the one your server actually listens on, character by character. A trailing slash, an old staging URL left in place after a migration, or a copy-paste error are all common and all invisible until you compare them directly.

A firewall or proxy is blocking the provider. If you've added IP restrictions, confirm the provider's current IP ranges are on the list. Providers do update these occasionally, and a restriction that worked six months ago can silently start blocking new ranges.

The request never got to your application code. Check for a reverse proxy, WAF, or load balancer that might reject the request before it reaches your handler, often invisible to application-level logging since your code never sees the request at all.

If it arrived but something went wrong

Once you've confirmed delivery, the search moves inside your own system. A few habits make this much faster the next time it happens, even if they don't help retroactively.

Log a correlation ID at every stage. Whatever unique ID the provider sends (Stripe's evt_ ID, GitHub's X-GitHub-Delivery, Shopify's X-Shopify-Webhook-Id) should appear in every log line related to that event, from ingest through to final processing. Searching your logs for one ID and seeing the full lifecycle beats reconstructing it from timestamps.

async function processEvent(event: WebhookEvent) {
  const log = logger.child({ eventId: event.id, type: event.type });

  log.info('Processing started');
  try {
    await handleEvent(event);
    log.info('Processing completed');
  } catch (err) {
    log.error('Processing failed', { error: err.message, stack: err.stack });
    throw err;
  }
}

Keep the raw payload, not just what you extracted from it. If your handler pulls three fields out of a payload and discards the rest, you lose the ability to check what else was in there when something looks wrong. Storing the full raw body (even just in a database column, even if you only query it rarely) turns "what did the payload actually contain" from a guess into a query.

Separate queue failures from handler failures. If you're using a job queue, its dashboard or logs usually show you whether a job failed to run at all (a connection error, a worker crash) versus ran and threw an application error. Those point at completely different problems.

Reproducing the bug without waiting on the provider

Once you've found the failing event, the fastest way to iterate on a fix is replaying the exact same payload against your handler, rather than trying to trigger a fresh one from the provider and waiting for it to arrive.

// If you stored the raw event, replay it directly
const failedEvent = await db.webhookEvents.findUnique({ where: { id: eventId } });
await processEvent(failedEvent);

This only works if you kept the original payload somewhere. Providers themselves offer manual redelivery from their dashboards (GitHub's Recent Deliveries, Stripe's event log), which resends the real signed request, but that still means switching to another dashboard, finding the event, and waiting on the round trip each time you want to test a fix.

A checklist for next time

The gap between "this took four hours" and "this took ten minutes" is usually a handful of things set up in advance:

• A correlation ID that ties a request through every log line it touches
• The raw payload stored somewhere queryable, not just the fields you extracted
• A clear separation in your logs between "we didn't receive it" and "we received it but failed to process it"
• A way to replay a specific past event without regenerating it from the provider

None of these are hard to add individually. The value comes from having all of them in place before the incident, not scrambling to add logging while a customer is waiting on an answer.

---

The storage and replay half of this list is what Webhook Watchtower handles by default, so it's one less thing to build: raw headers and body kept per event, full attempt history with response codes and timing, and replay without re-triggering from the provider. The debugging guide walks through using the dashboard to narrow down where a specific event got stuck.