Monitoring Webhook Health: The Metrics That Actually Matter

A webhook pipeline can look completely healthy on a standard uptime dashboard while quietly failing every delivery to one destination. The server is up, responding to health checks, serving traffic. None of that tells you whether the specific thing you actually care about (events arriving, verifying, and reaching their destination) is working.

Webhook health needs its own set of signals. Here's what actually matters, in roughly the order you'd want to notice a problem.

Delivery success rate, broken down per destination

The obvious starting metric, with one important caveat: blend it across every destination and you'll miss the case where one destination is failing consistently while others mask it in the average.

All destinations blended:  94% success   ← looks fine
Destination A:              99% success
Destination B:              72% success   ← this is the actual problem
Destination C:              98% success

A 94% blended rate looks like a minor, unremarkable amount of noise. Broken out, destination B is having a real, ongoing problem that the blended number was hiding. Track success rate per destination, not just as a single pipeline-wide figure.

Response time trend, not just pass or fail

A destination that times out is an obvious, visible failure. A destination that's getting slower, but hasn't crossed the timeout threshold yet, is invisible to a pass/fail metric and is often the leading indicator of the outage that's about to happen.

// Not just: did this succeed
recordMetric('delivery.success', destination.id, succeeded ? 1 : 0);

// Also: how long did it take, even on success
recordMetric('delivery.duration_ms', destination.id, durationMs);

A handler that used to respond in 300ms and is now consistently taking 4 seconds hasn't failed a single delivery yet, but it's heading toward the timeout, and you'd rather know now than after it starts actually failing.

Retry rate as a leading indicator

Every delivery that needs a retry succeeded eventually, so it never shows up in a simple failure count. But a destination whose retry rate is climbing (more and more deliveries needing attempt 2 or 3 before succeeding) is degrading, even if its final success rate still looks fine.

recordMetric('delivery.attempts_to_succeed', destination.id, attemptNumber);

This is one of the earliest signals available. A destination trending from "usually succeeds on attempt 1" to "often needs attempt 2" is worth investigating before it crosses into outright failures.

Verification failure rate, tracked separately from delivery failure

A signature that fails to verify is a completely different problem from a destination timing out, and lumping them into one "failure" bucket hides which one you're actually dealing with. A spike in verification failures usually means a rotated secret that wasn't updated everywhere, or a provider change to their signing format. A spike in delivery failures means something's wrong with the destination itself.

recordMetric('event.verification_status', source.id, verified ? 'passed' : 'failed');
recordMetric('delivery.status', destination.id, succeeded ? 'succeeded' : 'failed');

Keeping these as separate metrics, rather than one combined error rate, means an alert on one immediately tells you which half of the pipeline to look at first.

Queue depth, if you're running your own

If webhook processing goes through a queue you operate, queue depth is worth watching independently of any of the above. A queue that's steadily growing means your processing throughput can't keep up with incoming volume, a capacity problem that a per-event success rate won't necessarily surface until much later.

What to actually alert on versus just chart

Not everything here needs to page someone. Response time trend and retry rate are useful on a dashboard you check periodically or that feeds a slow-moving alert (sustained degradation over an hour, say). Delivery success rate dropping sharply for one destination, or a verification failure spike, are worth a more immediate alert, since both usually mean something needs attention now rather than eventually.

---

None of these metrics are exotic. The value is in tracking them per destination and as trends rather than a single blended number, and in keeping verification and delivery failures as distinct signals rather than one combined error rate.

Building this from scratch means wiring up your own event tracking for all of it. Webhook Watchtower records the underlying data per event instead, so verification status, delivery status, and every attempt's response code and timing are already there, broken out per destination. The debugging webhooks guide covers using it to run down a specific incident.