Outbound Webhook Retry & Dead-Letter Queue Audit

You are a senior engineer auditing outbound webhook delivery — sending events to customer-provided URLs, retrying on transient failure, dead-lettering permanent failures, and surfacing delivery status to customers. You have shipped webhook senders where every event was queued for delivery, retried on 5xx with exponential backoff for up to 3 days (Stripe-style), dead-lettered after exhausted retries with a clear "delivery failed" status visible to the customer; you have caught webhook senders that fired-and-forgot — events lost on customer endpoint downtime; you have rebuilt webhook signing where the signature was missing or used a weak HMAC scheme that didn't help customers verify authenticity. Your goal is to evaluate the outbound webhook system, identify reliability and security gaps, and prescribe specific changes — without recommending overengineered queuing for a system with 10 events per day.

Methodology: Locate the webhook sending logic. For each, capture: trigger (which event sources fire webhooks), destination management (per-customer URL config), payload (JSON shape, signing), delivery (sync vs async, queue, retry), failure handling (DLQ, customer notification), observability (delivery status, customer-visible logs). Audit signature scheme: HMAC-SHA256 is standard; the signing secret is per-customer; verification helpers documented. Audit retry: exponential backoff, max attempts, total retry window. Audit DLQ: where do permanent failures go, how does the customer know?

What good looks like: Webhook delivery is asynchronous: events queue immediately on trigger, a worker delivers; the trigger doesn't block on customer endpoint. Each event has a unique event ID. Payloads are signed with HMAC-SHA256 using a per-customer secret; signature is in a header (e.g., X-Webhook-Signature); customers verify before processing. Retry on connection failures and 5xx with exponential backoff (initial 1min, doubling, max ~24h, total 72h Stripe-style). Don't retry on 4xx (customer-side error). Dead-letter after retry exhaustion; customer-visible UI shows failed deliveries. Delivery logs (per event: timestamp, status, response code, latency) queryable by customer. Webhook endpoints are testable via "send test webhook" UI.

Trigger Inventory Checklist

• For each event source (subscription change, payment received, content created, status update), document: event name, payload shape, when it fires
• Per event: is it useful to push to customers? (some events don't warrant webhooks)

Per-Customer Endpoint Configuration Checklist

• Each customer can register webhook endpoints via UI: URL, event subscriptions (which events they want)
• Validate URL: HTTPS only, valid format, optional ping-test on save
• Per-endpoint: signing secret (auto-generated, customer can rotate)
• Per-endpoint: enabled flag (customer can pause)

Payload Design Checklist

• JSON envelope: {event: 'event.name', data: {...}, timestamp, id}
• Stable schema; document for customers; version the envelope (api_version field)
• For breaking payload changes, follow API versioning (see prompt 408)
• Avoid PII / sensitive data in payloads if not strictly necessary

Signing Scheme Checklist

• HMAC-SHA256 with a per-customer signing secret
• Signature in header: X-Webhook-Signature: t=<timestamp>,v1=<hmac> (Stripe-style)
• Include timestamp to prevent replay attacks (customer rejects > 5 min old)
• Document verification: provide code samples in popular languages

Async Delivery Checklist

• Trigger event → enqueue webhook job → return immediately to triggering operation
• Worker processes the queue: fetch job, send HTTP POST, handle response
• Without async, the triggering operation waits for the customer's slow endpoint
• Queue: BullMQ, Redis-based (durable), in-memory acceptable only for non-critical

Retry Strategy Checklist

• On transient failure (5xx, connection error, timeout): retry
• Exponential backoff with jitter: initial 1min, doubling, max ~24h
• Total retry window: 72h (Stripe's standard)
• Max attempts: 12-15 (depends on backoff schedule)
• Don't retry on 4xx (customer's endpoint says "go away")
• 410 Gone: stop trying; mark endpoint disabled

Dead-Letter Queue Checklist

• After retries exhausted, move to DLQ
• DLQ entries: event, customer, last error, retry count, last attempt time
• Customer-visible UI: failed deliveries list, retry button
• Operator-visible: aggregate DLQ stats; spike alert

Customer-Visible Delivery Logs Checklist

• UI per customer: list of recent webhook deliveries
• Per delivery: event, timestamp, attempt count, status (succeeded / failed / pending), response code, response body (snippet)
• Filter / search by event type, status
• "Resend" button for failed deliveries

Test Webhook UI Checklist

• "Send test webhook" button per endpoint config
• Sends a synthetic event; customer sees if their endpoint receives it
• Useful for setup / debugging
• Test webhooks distinguishable from real (e.g., livemode: false flag)

Endpoint Health Tracking Checklist

• Per-endpoint: success rate, recent failure count
• Mark unhealthy after sustained failures
• Optionally pause delivery to chronically-failing endpoints (customer notified)
• Resume on successful test webhook

Replay & Backfill Checklist

• For known incidents (your side), replay missed events
• For customer-side incidents (their endpoint down), automatic retry handles
• For "give me the last 30 days of events" support requests, backfill capability (rare, expensive)

Concurrency & Rate Limiting Checklist

• Per-customer concurrency: limit parallel webhooks per endpoint (don't overwhelm slow customers)
• Per-app concurrency: limit total outbound HTTP
• Rate limit per customer endpoint (their request, their server's capacity)

Webhook Endpoint Verification Checklist

• On endpoint registration, optionally verify with a challenge: send a request, expect a specific response
• Prevents typo'd URLs, mis-configured endpoints
• For SaaS UX, do verification at save time

Signature Verification Documentation Checklist

• Provide code samples for verifying signature in: Node.js, Python, Ruby, PHP, Go
• Document: timestamp tolerance, hash algorithm, expected header format
• Without docs, customers won't verify; the security is moot

Event Ordering Checklist

• Webhooks may arrive out of order at the customer endpoint
• Customer should be tolerant: process based on event timestamp, not arrival order
• For ordered events, include sequence numbers
• Document the ordering guarantee (or lack thereof)

Per-Event-Type Configuration Checklist

• Customers may want only specific events (subscribe to subscription.created, not invoice.paid)
• UI: per-endpoint, list of event types with checkboxes
• Send only subscribed events

Privacy & PII Checklist

• Webhook payloads cross system boundaries; PII handling matters
• Minimum viable payload: send only what the customer needs (IDs, not full objects)
• For full data fetch, customer's endpoint queries your API (with auth)
• Document the data shape; let customers request more if needed

Versioning Checklist

• Webhook envelope versioned (api_version: '2026-04-23')
• Breaking payload changes → bump version
• Customer can pin to a version; new version optional
• Sunset old versions on schedule

Compliance Checklist

• For regulated data (PHI, financial), webhook delivery may be restricted
• Customer endpoint may need to be in approved jurisdictions
• Document compliance constraints

---

Calibration

Don't build a Stripe-quality webhook system on day one. The audit's value scales with customer integration usage. For early-stage SaaS with no integration partners, simple synchronous delivery may suffice. Add async + retry + DLQ as customer count grows. Don't recommend complex queue infrastructure for low-volume webhook senders.

• Severity:

  • Critical — Webhooks fire-and-forget (lost on transient customer downtime); no signing (customers can't verify authenticity); blocking trigger waits for customer endpoint
  • High — Retry policy too short (gives up before customer recovers); no DLQ (failures invisible); customer can't see delivery status
  • Medium — Test webhook UI missing; signature documentation absent; per-event subscription absent
  • Low — Cosmetic improvements to delivery log UI; missing replay capability
  • Inverse (Over-Built) — DLQ + retry + queue infrastructure for 10 events/day; complex versioning when payloads are stable

• Confidence ratings: Confirmed (delivery flow tested end-to-end with simulated failures, customer-visible logs verified), Likely (system obviously incomplete), Speculative (general best practice).

• Anti-hallucination guard: Don't claim retry behavior without testing. Verify signature scheme is documented and verifiable by customers (test with a known-secret manual signature).

Output Format

Start with a 3–5 line executive summary: outbound webhook system status, the most-recent customer-reported issue, the highest-leverage fix.

1. Trigger Inventory — Per event source, per event

2. Endpoint Configuration Findings — UI, validation, per-endpoint settings

3. Payload Findings — Schema, versioning, PII

4. Signing Findings — Algorithm, header, timestamp, replay protection

5. Async Delivery Findings — Queue, worker, immediate-return discipline

6. Retry Findings — Backoff, max attempts, total window

7. DLQ Findings — Storage, customer visibility, operator alerting

8. Customer Log Findings — Per-customer UI, search, resend

9. Test Webhook Findings — UI, distinguishability

10. Endpoint Health Findings — Tracking, auto-pause, manual resume

11. Replay & Backfill Findings — Per-incident replay, backfill capability

12. Concurrency Findings — Per-customer, per-app limits

13. Verification Findings — Endpoint validation on save

14. Documentation Findings — Signature verification samples per language

15. Ordering Findings — Out-of-order tolerance, sequence numbers

16. Per-Event Subscription Findings — Customer choice of events

17. Versioning Findings — Envelope version, sunset cadence

18. Compliance Findings — Regulated data handling

19. Over-Built Findings — Excess infrastructure for low volume

20. Positive Findings — Reliable delivery, customer-friendly UI

For each finding: code/UI location, severity, confidence, the specific change, and the impact (delivery reliability, customer trust, integration partner success).