Alerting on Webhook Failures Without Alert Fatigue

The fastest way to make a team stop trusting an alerting channel is to notify on every failed webhook delivery attempt. A destination has a thirty-second blip, three retries fail before the fourth succeeds, and now someone's phone buzzed four times for an event that resolved itself without any human involvement.

Do this consistently and the channel stops meaning anything. People mute it, filter it into a folder they check once a week, or just get good at ignoring it. The one alert that actually mattered gets lost in the same pile as three hundred that didn't.

The failure most alerting setups make

The instinct is understandable: a failure happened, someone should know. But "a failure happened" and "a human needs to act" are different conditions, and conflating them is what causes the fatigue.

// This fires on every single failed attempt, including ones about to succeed on retry
worker.on('failed', async (job, error) => {
  await sendSlackAlert(`Webhook delivery failed: ${error.message}`);
});

If your retry schedule has five attempts spread over a couple of hours, a destination with a brief deploy window can trigger two or three of these alerts before the fourth attempt succeeds and the system recovers entirely on its own. Nobody needed to do anything, but three alerts fired anyway.

Alert on exhaustion, not on individual attempts

The useful signal isn't "an attempt failed." It's "all attempts failed, and the system has given up trying automatically." That's the point where a human's involvement actually changes the outcome.

worker.on('failed', async (job, error) => {
  const isLastAttempt = job.attemptsMade >= (job.opts.attempts ?? 1);

  if (!isLastAttempt) {
    // Still has retries left, log it but don't alert
    logger.warn(`Delivery attempt ${job.attemptsMade} failed, will retry`, { error: error.message });
    return;
  }

  // All retries exhausted, this needs a human
  await notifyPermanentFailure({
    eventId: job.data.eventId,
    destination: job.data.destinationName,
    attempts: job.attemptsMade,
    lastError: error.message,
  });
});

This alone removes the majority of noise, since most real-world webhook failures are transient and resolve within the retry window without anyone needing to look at them.

Circuit breakers change what "exhausted" means

If a destination is completely down (deleted, wrong URL, a server that will never come back), every single event queued for it will eventually exhaust its retries and fire a permanent-failure alert. During an extended outage, that can mean dozens or hundreds of individual notifications for what is, from a human's perspective, one incident: "this destination is down."

A circuit breaker collapses that into a single signal. Once a destination has failed enough consecutive times to trip the breaker, stop attempting new deliveries to it and send one alert: "this destination looks broken," not one alert per queued event.

async function handleDeliveryResult(destination: Destination, succeeded: boolean) {
  if (succeeded) {
    await resetCircuit(destination);
    return;
  }

  const failures = await incrementConsecutiveFailures(destination);
  if (failures === CIRCUIT_THRESHOLD) {
    // Fires once, at the moment the circuit opens, not once per subsequent event
    await notifyCircuitOpened(destination);
  }
}

Matching alert severity to actual urgency

Not every permanent failure deserves the same channel. A single event that failed because a destination briefly returned a validation error is worth an email and a dashboard entry: someone should look at it, but not right now, not at 3am. A destination that's been completely unreachable for twenty minutes, with a growing backlog of failed events, is worth something louder.

function severityFor(failure: DeliveryFailure): 'notify' | 'page' {
  if (failure.circuitOpen && failure.affectedEventCount > 10) return 'page';
  return 'notify';
}

Reserving the loud channel for genuinely urgent, high-impact failures is what keeps it credible when it does fire.

What "resolved automatically" should still leave behind

Even failures that never need a human deserve a trace somewhere queryable, in case a pattern emerges later: the same destination failing and recovering repeatedly might not need a page each time, but it's worth being able to notice after the fact that it's happening weekly, which is a different problem than a single blip.

---

An alerting channel that's trustworthy is narrow by design: notify on exhaustion, not on individual attempts, collapse an outage into one signal instead of one per affected event, and save the loudest channel for what's actually urgent.

For what it's worth, this is the shape Webhook Watchtower uses for its own notifications: nothing fires until a delivery permanently fails after retries are exhausted, and outages that recover inside the retry window stay quiet. Configuration is per workspace, covered in the notifications docs.