Three Lines of Defense, All Asleep: A Webhook Outage, and How We Make Sure a Machine Notices First

Part One: The Story, for Human Readers

We have a messaging channel that matters a lot to us: a collaborating AI on the outside sends tasks in through a webhook, our system receives them, files them, and then wakes a patrol program to handle them. It normally works quietly, delivering several urgent tasks a day, and we had even replaced its unstable temporary URL with a fixed domain name just for it.

On the afternoon of November 1, 2026, that channel went down for more than three hours. The most ironic part was not that it broke, but that — all three automatic lines of defense were asleep, and in the end it was the user who noticed first.

Starting at around 15:23 that afternoon, the two programs that receive messages quietly died; the last message successfully received was timestamped 12:03. The first line of defense was supposed to be a "keeper" program: it checks every 5 minutes and automatically restarts the whole service stack if something is broken. But that day it was in a disabled state, so the self-healing never kicked in at all — that was layer one.

The second line of defense was a "canary" we had launched only that noon: a zero-cost dead script that, every 5 minutes, walks the complete public path, sends a self-test message, and requires the other end to reply with a random code — only a matching code counts as truly connected. It was actually working — it showed a red light every single round, and the end-to-end test returned 502 every single round. But its escalation logic was written as "report once, then stay quiet," and that one single report was delivered to the wrong chat room. The red light stayed on for more than three hours and nobody was woken up — that was layer two.

The third line of defense was a "backup mailbox": when the main channel cannot deliver, the sender instead drops the message into a cloud mailbox, which we poll every 30 seconds. This layer actually saved us — not one of the 4 messages sent during the outage was lost; they were all sitting in the mailbox. The problem was that the program responsible for fetching the mail had died together with the main program, so the mail sat there for more than three hours, until the whole stack recovered and it was pulled in and filed all at once at 18:52. The backup existed, but the backup's courier was dead too — that was layer three.

At 18:38, the user noticed that "something seems wrong with messaging." We verified on the spot and confirmed that all three layers were dead at once. At 18:52 we ran a full-stack restart and verified segment by segment: the local health check was normal, the public health check was normal, and a self-test sent through the public URL returned 200. The channel was back, and the 4 messages in the mailbox were promptly backfilled and filed.

The second part of this article is the complete runbook for this incident: what the architecture looks like, how the canary and the keeper program are designed, how to locate a dead segment, every step of the repair and how to verify it, and the three fixes we added afterwards — so that the next time the channel breaks, a machine notices first, and it is loud enough to wake someone up.

One thing to say up front: the value of monitoring is not in "having it installed," but in "its alerting still being alive when things break." The fact that the alerting path shared the same fate as the system it monitored was the most expensive lesson of this incident.

Other PlAIground Lab products: Home｜ComputeQuote｜Axiom｜Ideal Type

The Complete Runbook Follows

1. Architecture: The Fleet Webhook Chain

Sender (external collaborating AI) → public endpoint → tunnel layer → receiver → filing → waking the patrol. Backup path: sender → cloud mailbox (KV) → poller program → filing.

Health check endpoints: local GET http://127.0.0.1:8791/health and public GET https://hook.example.com/health, both returning {"ok": true} when healthy.

Receiver validation rules (the four causes of a 400, each returning an error string): body size mismatch, JSON cannot be parsed, text missing or empty, task_id format invalid; there are also 401 (key mismatch) and 409 (duplicate). When troubleshooting, read the error string first — don't start by guessing that the sender is broken.

2. Incident Timeline (2026-11-01, America/Phoenix)

3. Locating the Dead Segment: Segmented Health Checks

Don't just test "connected or not." Split the chain into segments that can be read independently, have each segment keep its own health record, and when an incident happens you can read the dead segment straight off:

1. Receiver local segment: curl http://127.0.0.1:8791/health, expecting 200 with a body containing "ok": true.
2. Reverse tunnel process segment: check that the tunnel process exists and that its command line contains the expected port-mapping parameters; the process existing does not mean the function works (see the zombie listener in Section 6).
3. Public segment: curl https://hook.example.com/health, expecting 200. Note: some CDNs block POSTs without a User-Agent, so the canary must send a custom UA.
4. Poller segment: read the poller health file (such as poller-health.json) and check timestamp freshness (less than 10 minutes) and the consecutive failure count (less than 3).
5. End-to-end segment: send a test message of type: selftest through the public endpoint, carrying a random nonce; when the receiver gets it, it echoes the nonce directly without filing anything. Only a matching echo means the whole chain is truly connected — testing health alone misses the case of "the process is alive but the receiving path is dead."

How this incident read: segment 1 failed + segment 3 returned 502 → the dead point was the receiver itself (not the tunnel or the domain), and the correct response was a full-stack restart rather than poking at segments at random.

4. Repair Steps

Prerequisite: confirm there is a full-stack restart script start-all.sh (idempotent; it brings up, in order, the receiver, the poller, and the reverse tunnel, and ensures the tunnel on the home-computer side).

1. Run start-all.sh once. Expected: the script produces no error output, and each subprocess's PID file is updated.
2. Wait for convergence: rebuilding the tunnel and the public path takes time; wait at least 45 seconds before re-checking — judging failure 3 seconds after a restart is a known source of false alarms.
3. Re-check the five segments from Section 3 one by one. Expected: all five segments green.
4. End-to-end re-check: send another selftest through the public URL. Expected: returns 200 with a matching nonce, and a CANARY or selftest line appears in the receiver log.
5. Backfill check: look at the poller log and confirm that the messages accumulated in the mailbox during the outage have been filed. Expected: every message has an ACCEPT record, and the corresponding files appear in the task repo's pending area.
6. If, after step 2, some segment is still not green: stop, preserve the scene records first (each segment's status, the process list, the most recent logs), then narrow the scope by dead segment — do not keep re-running the restart script and hoping for luck.

5. Prevention Design: Canary, Keeper, and Escalation

The design finalized after the incident (all dead scripts with zero LLM cost, waking an agent only when a failure threshold trips):

Canary

• Runs every 5 minutes: first the segmented checks, then a selftest through the public path (nonce echo verification).
• The state file stores: consecutive failure count, whether escalation has fired, cumulative sent and succeeded counts, most recent latency, and each segment's status.
• Thresholds: 2 consecutive failures → automatically run start-all.sh to repair, wait 45 seconds, and re-test once; 6 consecutive failures (about 30 minutes) → escalate to wake the responsible team, with the dead-segment determination attached.
• Escalation must be repeatable (the core fix from this incident): after the first escalation, if the outage continues, re-report every additional 12 failures (about 1 hour); when an escalated incident recovers, send a follow-up "recovered" notice. Alerts must be delivered to the primary chat room, never to an unattended location.

Keeper Program and Restart Mutual Exclusion

• Both the keeper and the canary can trigger start-all.sh, and two simultaneous restarts will collide on the port (address already in use). The fix: start-all.sh takes a file lock at the start (flock single-instance; if the lock cannot be acquired, skip), and before binding a port it polls and waits for the old port to be released (probe every 0.5 seconds, up to 10 seconds).
• The keeper's "wait before re-checking after repair" must not be shorter than the tunnel convergence time (45-second scale for this system); otherwise it will misjudge a system that is still recovering as failed and restart it repeatedly.

Poller Backoff

• When the poller's upstream produces errors such as RemoteDisconnected, apply exponential backoff: starting at 30 seconds, doubling, capped at 300 seconds; each round's result is written to the health file (timestamp, consecutive failure count, most recent error) for the canary to read. Backoff is not giving up — it keeps the upstream from being hammered while it recovers, and lets the health file show "backing off" instead of "dead."

Sender Backup Convention

• When the sender's webhook calls fail consecutively (for example, hitting a 502), it switches to dropping messages into the cloud mailbox, which our poller pulls, so no message is lost. This convention was battle-tested within an hour of going live: after one 502, the sender switched to the mailbox and the message was delivered successfully (duplicate copies deduplicated and closed out by task_id). That 502 left no trace in our receiver's log and was determined to have died at the tunnel layer — which is also why the segmented records must include the tunnel segment.

Structural Assessment (not implemented; assessment conclusion only)

• The recommended long-term direction is "pull as the main path, push as acceleration": the sender writes to the mailbox and we proactively pull as the primary path, with the webhook tunnel demoted to an acceleration channel; first dual-write for a week to compare success rates and latency, then decide whether to migrate. The rationale: the pull model removes an entire segment of tunnel dependency and is inherently indifferent to endpoint URL changes.

6. Known Pitfalls Checklist

• Zombie listener: on the home-computer side, an old SSH session once held the mapped port without releasing it — the client side was dead, the connection was half-open through the proxy, and the system sshd had no ClientAlive configured, so it was never released; the tunnel process looked like it was running, but new connections could not bind. How to tell: from the home computer itself, curl 127.0.0.1:18791/health fails locally while the tunnel process exists — suspect this item; the remedy is to clear the old session processes, let the tunnel loop bind to a new session, and verify with a local curl.
• Egress credential rotation: the cloud VM's egress proxy credentials rotate with sessions, and a long-lived tunnel loop will keep retrying with dead credentials (process alive, function dead). A full-stack restart must also rebuild the tunnel loop so it picks up fresh credentials — and the keeper's periodic full-stack restarts objectively bound the maximum impact window of credential expiry.
• Timing oversensitivity: convergence after a restart takes time; re-checking too fast only manufactures false alarms and secondary restarts. Every re-check wait must be finalized from measured convergence time and written into the script, not left to feel.
• Report-once escalation: any design that "reports once, then stays quiet" is equivalent to staying quiet forever when that one report is lost or delivered to the wrong place. Escalation must re-report on a rhythm, and recovery must be reported too.

7. Verification Checklist (tick each item when the repair is complete)

• [ ] Local health returns 200 with "ok": true
• [ ] Public health returns 200
• [ ] Selftest via the public path returns 200 with a matching nonce
• [ ] Poller health file timestamp is less than 10 minutes old, with 0 consecutive failures
• [ ] All mailbox messages from the outage period are filed (verify each against its ACCEPT record)
• [ ] Canary state file's consecutive failure count is back to zero, and the recovery notice has been sent
• [ ] Keeper program is in an enabled state

© PlAIground Lab. Quoting and use for AI learning are welcome; please credit the source when republishing. Commercial resale is prohibited.

Agent version: a clean Markdown version of this article is also available (the English version, 2026-11-01-webhook-outage.en.agent.md, is ready, and a Chinese agent version is provided as well), so your AI can read it and build along with it. Original manuscript timestamp: 2026-11-01 (preserved via repo commit)