On-call runbooks AI agents actually read first

A runbook is only useful when it's in front of the person — or agent — handling the incident. This is obvious and mostly not achieved.

The standard on-call runbook is a Confluence page, a Notion doc, or a README. An engineer gets an alert at 2am, opens an AI assistant to help triage, and starts describing the problem. Whether the relevant runbook is in that session depends on whether the engineer included it, whether they remembered which page it was, and whether they had the presence of mind at 2am to locate and include the right document.

Most don't. The runbook exists. It's rarely the first thing the AI sees.

The session-load problem

For an AI assistant to apply runbook procedures, it needs to have them in context. Context has to either be provided by the user or loaded automatically.

User-provided context at 2am during an incident is unreliable — not because engineers are careless, but because incident response is cognitively demanding and adding "find and paste the runbook" as a prerequisite step to every triage session introduces friction at exactly the wrong moment.

Automatic loading is the correct model. When an engineer opens an AI session tagged to a service or alert type, the relevant runbooks load as memories at session start. They don't need to be found and pasted. They're in the context window before the first message.

Tagging runbooks for automatic load

In Memory Stack, runbook content is stored as memories tagged by service, alert type, or incident category. A runbook for the auth service latency alert is tagged service:auth, alert:latency. An engineer starting an incident triage session for an auth latency alert loads that runbook automatically.

The granularity of tagging is flexible. A team that wants runbooks loaded for every auth-related incident tags broadly. A team that wants specific procedures for specific alert signatures tags narrowly. The tagging model reflects the team's operational structure.

What this means in practice: the AI assistant in the session starts with the relevant runbook in context. When the engineer asks "what are the diagnostic steps," the answer draws on the runbook — not on generic troubleshooting heuristics that may or may not apply.

Runbooks as living documents

The failure mode of traditional runbooks is staleness. They're written when a service is stood up or when an incident happens, and they're not updated consistently as the service evolves. An engineer following a runbook from 18 months ago may be following procedures that no longer apply to the current architecture.

Runbook memories, like all memories in Memory Stack, can be surfaced by the Stale Shelf mechanism when they haven't been reviewed recently. A runbook for a service that's been significantly refactored in the past year will surface for review, prompting an explicit decision: update the procedure or confirm it's still accurate.

The update when made is versioned: the prior procedure is preserved with its dates; the new procedure carries the author and timestamp of the update. A post-incident review can trace which version of the runbook was loaded during the incident — useful when evaluating whether the procedure was correct.

The knowledge capture direction

The other direction the runbook relationship should flow: insights from incidents captured back into the runbook layer.

An engineer who discovers a new diagnostic step that isn't in the current runbook, or who finds that step 3 doesn't apply in a specific configuration, can log that as a memory update during or after the incident. Passive capture surfaces these as candidates when the session contains decision-relevant operational content.

Over time, the runbooks reflect accumulated incident knowledge, not just the original procedure. The memory layer is the mechanism for that accumulation — the path from "we discovered this in an incident" to "this is now in the runbook that loads for every future incident in this category."

The honest version

Runbook memories loaded at session start are only useful if the session is associated with the right service and alert type. An engineer who starts a generic chat session without specifying the incident context may not get the right runbook loaded automatically. Session tagging discipline — associating the session with the service being investigated — is what makes automatic load work reliably.

Also: runbooks are starting points. Novel incidents, cascading failures, and cross-service issues often outrun the documented procedures. The runbook in context is the baseline; the engineer's judgment is what handles the cases the runbook doesn't cover.

An on-call runbook that needs to be found and pasted before it's useful isn't meeting the use case it was written for. On-call context that loads automatically at session start — procedures available before the first question is asked — is the operational behavior that makes runbooks actually functional at 2am.