Why Webhooks Fail (And How to Fix Them)

Webhooks are supposed to be simple. Your payment processor fires an HTTP POST at your server. Your server handles it. Done.

In practice, webhooks fail all the time, and when they do, they usually fail silently. No error on your side. No obvious log entry. Just a missed event and a confused customer.

These are the most common reasons it happens.

The fundamental problem with webhooks

Unlike an API call you make (where you control when it happens and you wait for the response), webhooks arrive on the sender's schedule. Your server has to be ready at exactly the right moment. If it is not, the event is either lost or you have to hope the sender retries.

Most providers do retry. But retry policies vary wildly, and "retried a few times over 24 hours" is not the same as "reliably delivered."

The most common failure modes

1. Your endpoint was temporarily unavailable

Deployments, restarts, scaling events, cold starts on serverless functions: any of these can cause your endpoint to return a non-2xx status or simply not respond in time. Most webhook providers consider a 5xx response or a timeout a failure and will retry, but after a few attempts they give up.

If your deploy takes 30 seconds and a webhook arrives mid-deploy, it either fails or hits an old instance that may not have the latest code.

The fix: Use a dedicated ingest endpoint that does the minimum work. Acknowledge the event immediately, push it to a queue, and process it asynchronously. This keeps your endpoint fast and almost always available.

2. Timeouts

Webhook providers typically give your endpoint somewhere between 5 and 30 seconds to respond. If your handler does any non-trivial work synchronously (database writes, calls to other services, sending emails) you will hit this limit eventually.

GitHub's default timeout is 10 seconds. Stripe gives you 30. Shopify gives you 5.

The fix: Return a 200 immediately and do the actual processing in a background job. Your handler should do nothing more than validate the signature and enqueue the payload.

3. Non-2xx responses

Webhook senders require a 2xx response to consider a delivery successful. A 301 redirect, a 404, a 500 - these all count as failures.

Common causes:

• The URL was changed in your code but not updated at the provider
• Authentication middleware is blocking the request (common when you accidentally put webhooks behind a login)
• Your server is returning HTML error pages instead of JSON

The fix: Check what status code you are actually returning. Add logging at the ingest layer that records every incoming request before any processing happens.

4. Signature verification failures

Most providers sign their webhooks with an HMAC signature. If your verification logic is wrong (wrong secret, wrong algorithm, incorrect body parsing) the verification will fail and you will typically reject the event yourself.

A very common cause: body parsing middleware that reads and modifies the raw request body before your verification code sees it. HMAC verification requires the exact byte-for-byte original payload.

The fix: Read the raw request body before any other middleware touches it. Most webhook providers have a specific section in their docs about this.

5. Duplicate processing

Webhook delivery is "at least once." Providers will retry if they do not get a clear 2xx, which means the same event can arrive multiple times. If your handler is not idempotent, you can end up with duplicate orders, double-sent emails, or twice-charged customers.

The fix: Store processed event IDs and check before processing. Most providers include a unique event ID in the payload or headers.

6. Ordering issues

Webhooks from the same provider do not always arrive in the order they were sent. A customer.updated event can arrive before the customer.created that preceded it. If your handler assumes order, it will break in subtle ways that are hard to reproduce.

The fix: Design handlers to be resilient to out-of-order delivery. If you need ordering, fetch current state from the API rather than relying on the event sequence.

7. The staging URL in production

This is embarrassingly common. A developer tests a webhook integration locally or on staging, sets the URL in the provider's dashboard, then forgets to update it before going to production. Events go to the wrong place and nobody notices until a customer complains.

The fix: Keep webhook configurations separate per environment. Better yet, use an ingest proxy that makes it easy to see exactly what is arriving and where it is going.

---

Why these failures are so hard to spot

Most of the above happens without any obvious error. Your application does not throw an exception. Your server does not log anything. The event just never arrives.

The places you would normally look, your own application logs, often have nothing useful because the request never made it far enough to be logged. The provider's delivery logs (if they expose them) show their side of the story, not yours.

The result is a gap between "the provider says it was delivered" and "your application processed it" that is almost impossible to debug without dedicated tooling.

---

The setup that actually works

These five things together make webhooks reliable in production:

1. Dedicated ingest endpoint - one URL per source, no auth middleware, returns 200 immediately
2. Signature verification on receipt - validate HMAC before doing anything else
3. Enqueue, do not process inline - push the raw payload to a job queue, return 200
4. Idempotent handlers - check event IDs before processing
5. Visibility into every delivery - log every event, every attempt, every response

This is roughly what Webhook Watchtower handles for you. Your provider fires at a Watchtower ingest URL, we verify the signature, store the full payload, and forward to your endpoint asynchronously with retries. If delivery fails, you see exactly what happened and can replay with one click.

The key insight, whether you build this yourself or use a tool, is treating webhook delivery as unreliable infrastructure rather than something that just works. If you're already using Webhook Watchtower, the debugging guide walks through a systematic approach to diagnosing exactly which part of the chain broke.