Runbooks drift and lie
A wiki runbook is written once and is wrong two releases later — and it looks authoritative right up until the step fails, mid-incident.
Ops has two memory failures PriorStates is built to fix. Runbooks drift and lie — you find out during the outage. And incident knowledge evaporates — the next on-call re-walks the dead ends the last one already paid for. PriorStates turns runbooks into runnable docs that prove their own freshness and post-mortems into searchable memory. Recall by symptom before you firefight, record the post-mortem as it resolves.
The doc you trust is wrong, and the one person who remembers the last time is asleep.
A wiki runbook is written once and is wrong two releases later — and it looks authoritative right up until the step fails, mid-incident.
A post-mortem is the richest gotcha doc anyone writes — and it lands where nobody searches it. The next on-call re-derives “didn't we hit this before?” under pressure.
Most rotations only do steps 2–3. PriorStates is steps 1, 4, 5 — the brackets that make ops memory compound instead of resetting each on-call.
Before touching anything, journal_search the symptom. Last
quarter's root cause and the relevant runbook surface. You skip the dead end
the last incident already paid for.
The rollback procedure is a runnable mdlab doc — it executes, with
captured output. No “is this still accurate?” doubt at 3am.
Mitigate, verify recovery, stand down. The step ends with the service healthy and an MTTR you can stand behind.
The incident isn't done when it's green — it's done when the next on-call
can find it. A bug (root cause), a decision (what
changes), an updated runbook — with the metric attached.
Different person, different tool, six months later — the symptom search returns this incident. Ops memory compounds across rotations.
Same symptom search, two incidents apart. The recall that opens tonight's page already carries what the last outage taught — and tomorrow it carries this one.
Step 1 — recall before you firefight. Prior findings surface the moment the alarm fires:
$ journal_search topic=latency-spike
outcome title TL;DR
─────── ────────────────────────────────────────────── ─────────────────────────────
loser Bumped the connection pool, 2024-11 outage didn't help — it was DNS
gotcha p99 lags error rate by ~90s on billing page on p99, not error rate
ref Roll back billing svc: rollback.mdlab.md runnable, ~4 min, tested 2026-06You skip the connection-pool rabbit hole, you trust the runbook because it ran clean last month, and you know to watch p99. The last incident is working for you.
Step 4 — record, then recall again. You found the cause and journaled it. The same search now carries four:
$ journal_search topic=latency-spike
outcome title TL;DR
─────── ────────────────────────────────────────────── ─────────────────────────────
loser Bumped the connection pool, 2024-11 outage didn't help — it was DNS
gotcha p99 lags error rate by ~90s on billing page on p99, not error rate
ref Roll back billing svc: rollback.mdlab.md runnable, ~4 min, tested 2026-06
bug Cache stampede on cold deploy → p99 5x add request coalescing (MTTR 22m)The next on-call inherits the root cause and the fix. No one re-debugs the stampede from scratch.
A wiki runbook can be wrong silently. A runnable one fails loudly — which is exactly what you want before you rely on it.
An mdlab runbook is prose, commands and captured
output in one file — and it executes:
priorstates mdlab run rollback-billing.mdlab.mdIf a step has rotted since the last release, the run fails now,
in a calm moment — not at 3am while you're trusting it. Pair it with
as_of-decayed incident notes (an old “always restart X” sinks below a
recent contradicting one) and the store stays trustworthy without manual gardening.
Each a plain Markdown file with the metric attached, searchable by the symptom — not buried in a doc tree.
The root cause, with impact. “Cache stampede on cold deploy → p99 5×.” Recalled the next time the symptom appears.
The fix that didn't work. “Bumping the pool didn't help — it was DNS.” The single most expensive thing to rediscover under pressure.
A trap in the system's behavior. “p99 lags error rate by 90s.” Changes how you read the dashboards mid-incident.
The procedure, runnable and dated. “Roll back billing, ~4 min, tested 2026-06.” Recalled by symptom, trusted because it runs.
Once installed, your agents and runbooks do it on their own.
priorstates init gives the team a shared
memory and journal and
wires the recall / record protocol into Claude Code,
Codex, Gemini, Copilot and Cursor. Runbooks live as mdlab files that
run; incidents are recalled by symptom. Store the diagnosis, not the log
firehose — and never secrets, which belong in a vault.
Free, open source, and yours to run.