Runbooks are for the 3am version of you
A runbook isn't documentation. It's a script for someone tired, scared, and holding a pager.
- sre
- operations
The version of you reading a runbook is not the version of you who wrote it. They are tired. They are being asked questions in a chat window. They have already tried the thing that seemed obvious, and it didn't work.
Write for that person.
What that means in practice
A runbook that says "check the database is healthy" is not a runbook. It's a prompt for someone who already knows the answer.
# Is the primary accepting writes, or just answering reads?
psql -h primary.internal -c 'select pg_is_in_recovery();'
Every step should be something you can copy, run, and recognise the output of.
The rules I keep failing to follow
- One action per step, not one idea per step
- Say what success looks like, not just what to run
- Say what to do when it fails, even if that's "escalate"
- Date the thing, and say which version of the system it was written against
The test
Hand the runbook to someone who has never touched the system, at a normal hour, with the pager off. Watch where they hesitate. Those are the steps that will cost you an hour at 3am.
If they can't get through it while well rested and caffeinated, it isn't going to work in the dark.