Non-Diagnostic Recovery

Flew home from DEFCON. Landed in Tulsa at 3am. Head down a little after four. At 4:30, PagerDuty went off.

I won’t tell you what the incident was. It doesn’t matter, and not knowing is the whole point. On ninety minutes of sleep I didn’t know what was wrong either. Nobody does at 4:30am. Kernighan’s law says debugging is twice as hard as writing the code, which means the system you built at your sharpest already sits at the edge of what you can diagnose at your sharpest. At 4:30am you’re nowhere near your sharpest. You’re not a full-powered engineer in that moment. You’re running on whatever fumes are in the tank and whatever was decided before you went to sleep.

So “what’s wrong” was never the question. At 4:30am there are only two: what does the alarm say, and what am I cleared to do. I had an answer to the second one, because somebody had already written the list: a handful of actions pre-decided as safe to take blind.

Shooters and pilots have been writing that list for decades.

The drill has a name

Defensive pistol training gave it the cleanest name. When a pistol stops firing mid-fight, you don’t stand there examining a mechanical system while someone closes distance on you. You run a fixed sequence: tap the magazine, rack the slide, shoot. If that fails, reload and rack. If that fails, strip the magazine, rack three times, reload. Eyes stay on the threat the whole time. The drill is called non-diagnostic linear malfunction clearance. Non-diagnostic because you never stop running the gun long enough to figure out what’s wrong with it. Linear because the sequence is fixed and advances on its own when a step fails.

The military wrote the same idea into doctrine. The Army’s rifle manual defines immediate action as a correction applied “without performing troubleshooting procedures to determine the actual cause.” Aviation went further and made it typography: the emergency actions a pilot must execute from memory are printed in bold in the flight manual, so “boldface” became the name for the procedures you run before you open the checklist. The FAA teaches why with a body count. In 1972, all four aviators on Eastern Air Lines Flight 401 got absorbed in diagnosing a burned-out landing gear bulb while their L-1011 descended into the Everglades. A hundred and one people died. Diagnosis, at the wrong moment, is the thing that kills you.

Every one of these disciplines converged on the same design: when something breaks under pressure, execute a pre-decided sequence first. Understand later.

None of this stays on the range. A leaked API key, an agent burning through a queue with valid credentials, a box beaconing somewhere it shouldn’t. All of them break at contact speed, and all of them have a tap-rack equivalent: revoke the token, contain the host, freeze the agent. Pre-decided, pre-authorized, executed blind. Software has been quietly running this doctrine for years, and the cleanest proof ran in production for a decade.

Every 24 hours, whether you needed it or not

Heroku ran this doctrine as a platform. For over a decade, the dyno manager restarted every dyno at least once a day. Every 24 hours, plus up to 216 random minutes of jitter so a fleet never restarted in unison. No diagnosis. No ticket. No one asked you. The official rationale was a single vague phrase: “to help maintain the health of applications running on Heroku.”

And it worked. A generation of Rails apps with real memory leaks ran boringly reliable for years. Engineers said the quiet part out loud on Hacker News: “programs with slow memory leaks tend to work just fine.” One admitted the only reason their client’s leak went undetected was that Heroku restarted the dynos daily. Microsoft made the same bet with IIS, recycling app pools every 29 hours because it’s the smallest prime number over 24 and a prime interval never falls into resonance with the daily traffic pattern. Academia has a name for the whole family: software rejuvenation, in the literature since 1995.

Your gut says this is crude. The uptime says your gut is wrong.

The hard part isn’t the sequence

Tap-rack-bang is three words. Anyone can memorize it. What turns it from trivia into a drill happened earlier: someone with authority decided, in advance, that this is the sequence, and that the person holding the gun doesn’t ask permission to run it.

That second half is the part most organizations skip. Everyone has runbooks. Almost nobody has granted the 4:30am authority. If your on-call engineer needs an incident commander’s approval to isolate a host, you don’t have a drill. You have a meeting with a countdown timer. The entire value of the boldface page is that the thinking moved earlier in time, into a calm room, and the operator inherits a decision instead of making one. NIST’s incident handling guide says it plainly: containment decisions “are much easier to make if there are predetermined strategies and procedures for containing the incident.”

Write the sequence, and you’ve documented the fix. Grant the authority, and you’ve built the capability. They are not the same artifact.

You already run these drills

This doctrine is not something you’d be importing into security. It shipped with your stack.

CrowdStrike network containment is one click. The host loses every network connection except the channel back to the Falcon console, before anyone knows what’s on the box, and the reverse is one click too: lift containment. Okta session clearing kills every active session and forces re-authentication; suspension keeps the user’s apps and groups intact and restores them on unsuspend. Credential rotation runs on suspicion, not proof. CISA’s federal playbook lists rotating keys “where compromise is suspected” as a containment activity, and AWS tells you to disable exposed keys rather than delete them precisely so the action stays reversible. Microsoft went all the way to removing the human: Defender’s attack disruption auto-contains devices and revokes sessions mid-attack, and its published justification is that every automatic action “can be undone by your security team.” That is the vendor’s claim about its own product. Reversibility you have not rehearsed is a datasheet feature, not a control.

Your incident response framework blessed all of this a long time ago. In PICERL and NIST 800-61 alike, containment comes before eradication and recovery. The ordering is the doctrine: you contain before you fully understand, because containment “provides time for developing a tailored remediation strategy.” NIST calls the alternative, watching the attacker while you build a complete picture, dangerous. The 2025 revision goes further and recommends configuring your tools to contain automatically. The framework has been non-diagnostic at its core the entire time. It just never used the word.

Seven tests for a blind action

Not every action earns a spot on the list. The list is exactly the set of moves that pass seven tests:

0. Captures before it clears. The drill’s first step snapshots the state the drill is about to destroy. An action that wipes its own evidence fails, whatever else it passes.
1. Safe on a false positive. If firing it on a healthy system causes real damage, it can’t be run blind.
2. Cheap enough to run every time. A drill you hesitate to fire isn’t a drill.
3. Reversible, or bounded. Lift containment. Unsuspend. Re-enable the key. The undo is what makes blind execution defensible.
4. Rehearsed. The first live run cannot be during the incident.
5. Safe even when the attacker pulls the trigger. Your playbook is not a secret.
6. Ends in a handoff. The last step of every sequence files the diagnosis: the captured snapshot, a named owner, a date. The Army applies immediate action once; a second stoppage of the same kind means you stop and look at the weapon. A drill that ends with the system green has ended early.

Run the security stack through them: network contain passes, key rotation passes, snapshot-and-restart passes, each with its capture step named and its handoff filed. Session revocation passes only with a rate bound the attacker can’t reach, because a revocation an attacker can trigger at will is the account lockout weapon by another name. Now run the board’s favorite control through them: the kill switch. Hard-killing a system fails test one on its face. It demands a diagnosis-grade decision, under maximum uncertainty, at the worst moment, from whoever happens to be awake, and it destroys in-flight state on the way down. That’s why it lives in board decks instead of runbooks. The kill switch is the feeling of control. The drill list is control.

The same tests score your AI operations. Restart the agent: passes, if state is snapshotted first. Revoke and reissue its credentials: passes. Quarantine the agent and route its work to a fallback: passes. Drop it to shadow mode, where it keeps deciding but stops acting: passes, and it keeps generating evidence while contained. Roll back its state to a known-good checkpoint: passes test three by definition, with one caveat that gets its own section.

Step zero: capture before you clear

There’s a reason boldface works in aviation and it isn’t pilot discipline. The flight recorder is running the whole time. A crew can act before understanding because the evidence survives the action.

Most software recovery actions destroy their own evidence. A restart throws away the heap. Revoking a session throws away the session context. Resetting an agent throws away the conversation state that explains what it was doing. Run those blind, on a schedule, and you’re not recovering. You’re wiping the crime scene daily and calling it hygiene.

So every drill on the list gets the same first step: snapshot before you act. Memory image before the restart. Session export before the revocation. Full state dump before the agent reset. This is not a novel idea either; it’s written into forensic doctrine. RFC 3227: “Don’t shutdown until you’ve completed evidence collection.” NIST lists evidence preservation as a formal criterion for choosing a containment strategy and tells you to capture memory, connections, and process lists before they’re gone. The order of operations is the whole discipline: contain in a way that preserves, then clear.

The AI version of this has a sharp edge. When an agent misbehaves, the temptation is to ask it what happened. Its answer is not evidence. An agent’s account of its own state is a self-report from the thing you’re investigating. Snapshot what it actually holds, then reset it, and let the snapshot testify.

The attacker has read your playbook

Everything above was invented against failures that don’t fight back. Jammed pistols don’t adapt. Memory leaks don’t learn your restart schedule. Your adversary does both, and a documented, pre-authorized, automatic response is part of your attack surface the moment it exists.

This isn’t paranoia; it’s in the standards. NIST’s incident guide warns about containment that’s booby-trapped: malware that pings a peer host, notices when you disconnect the network, and encrypts the drive in response. The official remediation, executed on cue, becomes the trigger. The pattern generalizes everywhere you look. Account lockout policies get weaponized as denial of service against your own users. Password spraying exists specifically to glide under the lockout thresholds you published. Banks learned in 2012 that a DDoS can be a smokescreen, occupying the responders while the wire transfer goes out the back; in one Kaspersky survey, over half the businesses hit by DDoS believed it was cover for something else. And the newest version: a crew calling itself Ransom Busters, which GuidePoint’s researchers assess is likely a ransomware affiliate, marketing itself as an incident recovery firm and reaching out helpfully before the attack is even public. The adversary is not just reading your playbook. Sometimes he’s offering to run it.

This is why test five exists. Analyze every drill under the assumption the attacker fires it. A rollback the attacker can trigger is a denial-of-service primitive. A rollback to a checkpoint the attacker poisoned is a persistence mechanism with an audit trail that says you did it. Quarantine that reroutes work to a fallback hands the mission to whatever the fallback trusts. Sometimes this test will disqualify a drill, and that is the test working. A test everything passes is not a test. The bar it sets: safe when triggered by failure, safe when triggered by the enemy.

The tests apply to the scaffolding too. The vault where step zero concentrates memory images and state dumps is now the most valuable box in the room. A published threshold invites the attacker to stay one fire under it. A scheduled restart is a persistence metronome. And the agent reading captured state during the RCA is reading bytes the attacker may have authored. Every piece of recovery machinery is machinery, and machinery gets attacked.

The restart works, so the leak never gets fixed

Now the other half of the Heroku story. Daily cycling made leaky apps reliable, and Heroku’s own documentation says what it didn’t do: restarts “won’t actually fix any memory problems, but instead will cover them up.” A stop gap, in their words. The stop gap ran for a decade as the platform default, and a generation of leaks went not just unfixed but undetected.

Notice what actually disappeared. The restart didn’t remove the leak. It removed the signal. Before cycling, a leak eventually became an outage, and outages get investigated. After cycling, the same leak became nothing at all. Silent automatic recovery converts a recurring defect into a defect with no evidence stream, and no amount of telling engineers to be more curious survives contact with a system that reports nothing wrong.

This is the standing objection to everything this post argues, and it’s a fair one. Act before you diagnose, automate the action, and you’ve built a machine for never diagnosing anything. The fix is not to slow the drill down. The fix is accounting.

Count every fire

Every drill execution is an event. Which drill, on what target, triggered by what, carrying what captured context. Log it like an incident, because it is one; a drill that fired is a failure that got survived, not a failure that didn’t happen.

Then count them, and give the count a threshold with teeth. The count needs an owner with a name, and it’s the same name that signed the authorization: whoever granted the 4:30am authority owns the fire count and reads it on a cadence, because the authority and the accounting are the same grant. An unread counter is a silent restart with extra steps. Google’s SRE culture has run this play for years. Carla Geisser’s formulation: “If a human operator needs to touch your system during normal operations, you have a bug.” The error budget works because the threshold was agreed on in advance, so when it’s crossed, engineering effort shifts by prior agreement instead of by argument. Do the same for drills: N fires against the same fault triggers a mandatory root cause with a named owner and a date. Heroku’s decade of invisible leaks happened because the restart was silent. A counted restart is the opposite of silence. A drill firing daily is a bug report the system is filing on its own behalf, a vulnerability disclosure you’re making to yourself. The drill is the tourniquet. Somebody still owns the surgery.

The threshold is the backstop, not the bound. Test six already set the bound: a second occurrence of the same fault means eyes on the weapon now, not at fire number N. And a drill that fires on a schedule never fires against a fault at all, so scheduled drills get audited on a calendar instead. Heroku’s restart would never have crossed a per-fault threshold. That is exactly how a decade of leaks stayed invisible.

AI holds both ends now

The honest objection to act-first was always the tradeoff: speed now, understanding never, because by the time the incident is over, the diagnosis budget is spent and Tuesday’s meeting has other topics. AI collapses that tradeoff from both ends.

On the action end, the sequence executes in seconds, at 4:30am, with the snapshot taken first because step zero is in the playbook and scripts don’t skip steps. LLMs are not scripts. So the drill itself is code, and the agent’s job is to pull the trigger, not to improvise the steps. Verify the snapshot exists before trusting that it ran. An agent reporting it followed the sequence is a self-report too. On the analysis end, the root cause work starts the moment the drill fires instead of the moment someone gets around to it. This is already happening in the field. When Hugging Face got breached, hosted frontier models refused to process the attack commands and payloads, so the responders ran their forensics on an open-weight model on their own infrastructure. Agents walked more than 17,000 recorded attacker events, and the team did “in hours what would usually take days,” with no attacker data or referenced credentials ever leaving their environment. And on Unchained’s Uneasy Money, Kain Warwick described a coding agent deleting his entire database, and two AI models independently noticing the process was still running, dumping the database from memory, and rebuilding it in thirty seconds. The evidence was volatile state. Fast hands saved it. The width of the snapshot is the ceiling on the analysis. An agent can only interrogate what step zero actually kept, and an agent walking thin telemetry produces a confident wrong answer faster than no agent at all.

Diagnosis was expensive, so leaders tolerated skipping it. AI repriced the writing of the analysis. It cannot reprice the tolerance. The learning still costs what it always did: people close to the work reading the analysis, arguing with it, and changing what they believe about the system. A cheap debrief nobody reads is still a document. The machine’s analysis counts when a named owner turns it into a change on the drill list and the operators rehearse the change.

Drills come from postmortems

None of the drills in this post were invented in a workshop. Tap-rack-bang compresses a century of stoppage data. Every boldface page was paid for by an accident report. Heroku’s daily restart encoded years of watching dynos die. The drill is what a postmortem looks like after it’s been compiled.

Which means your lessons-learned phase has a concrete output format. Every root cause analysis ends with two questions: what would the immediate action have been, and who is now authorized to run it without asking? Answer them, add the action to the list, grant the authority, and the next incident starts with a better opening move. NIST’s newest revision points the same direction, telling teams to share lessons “as soon as they are identified, not delayed until after recovery concludes.” A lessons-learned phase that doesn’t mint drills mints documents.

Run it before you need it

A drill you’ve never run under pressure is a theory. AWS lists the failure mode by name in its Well-Architected guidance, as an anti-pattern: “You document your procedures, but you never exercise them.” The fix is rehearsal against real systems. Game days. Chaos engineering, which exists precisely to build “confidence in the system’s capability to withstand turbulent conditions in production.” Netflix let Chaos Monkey kill production instances during business hours so that instance death became a non-event.

Security got its own edition of this discipline. Kelly Shortridge and Aaron Rinehart’s Security Chaos Engineering is the book-length version of the argument, and it names exactly what an unexercised playbook is worth: plans are “better for creating a sense of comfort rather than a sense of confidence during real response activities. Incidents rarely unfold exactly as people imagine.” Comfort is what your runbook wiki gives the board. Confidence comes from the experiment: inject the failure, watch your drill run against it, and fix what the watching reveals.

So here’s the rep. Pick your most likely 4:30am page. Write the immediate actions for it, run them through the seven tests, and name who is cleared to fire them without asking. Put the snapshot in step zero. Put the counter on every fire. Then run the whole thing live, on a Tuesday afternoon, while everyone is rested and watching. The first execution of your boldface cannot be the night it’s real. Neither can the last.

That night in Tulsa, the list ran. I was back in bed before five. The diagnosis happened the next morning, rested, with the evidence waiting.

Pre-authorized blind action sounds like recklessness. It’s the most disciplined thing an organization can do: the hard thinking done early, in a calm room, by the person who owns the outcome, handed to the operator as freedom to move. Act before the root cause analysis. Never skip the root cause analysis. Defer diagnosis. Never delete it.