It’s 2:47 AM. PagerDuty just went off. Something called PolicyConstraintSyncFailed is firing, and the only thing you know for certain is that you wrote a runbook for this exact situation four months ago, in a moment of great personal discipline, and you now cannot remember which file it's in, what it's called, or whether you ever actually finished writing it.
So you do what everyone does: you open the repo, you
grep -ri "syncfailed" ., you get six matches across three files, half of them are Terraform variable names that happen to contain the word "sync," and you start reading top to bottom while the alert timer keeps ticking in the corner of your screen.
I’ve done this enough times that I finally sat down and built something to stop doing it. It’s called infra-runbook-rag, and the short version is: it's a small tool that answers on-call questions using your own docs, and now it also takes a real alert and tells you which runbook section it's about. Nothing fancier than that. But getting the boring parts right — the chunking, the retrieval, the "should this even call an LLM" decision — turned out to be the actual work, and that's what this post is about.
The itch
There are basically three ways people deal with “where’s the runbook for this”:
- Grep and skim. Works, but it’s slow under pressure, and it has no idea that “reseal the demo secret” and “rotate the sealing key” are the same operation described two different ways in two different files.
- Dump everything into a general chatbot. This is the thing everyone reaches for now — paste your whole wiki into ChatGPT’s context window, or wire up a generic “chat with your docs” SaaS product. It works until it doesn’t, and when it doesn’t, you have no idea why. Did it hallucinate? Did it just not have that doc in context? You can’t tell, because the answer doesn’t point back to anything.
- Ask the person who wrote it. Great, if it’s 2:47 PM and they’re awake.
What I actually wanted was something narrower and more honest: search this repo’s own markdown, and when it answers, tell me exactly which file and which section the answer came from — down to the paragraph, not just the filename. If it doesn’t know, say so. Don’t guess.
What it actually is
infra-runbook-rag its a Python CLI (plus a small Streamlit UI, plus — as of the most recent thing I added — a webhook receiver) that:
- Ingests a repo’s markdown docs, chunks them by header, and embeds every chunk locally.
- Answers a typed question by retrieving the closest chunks and asking an LLM to synthesize from only those excerpts, citing each claim.
- Takes a real Prometheus Alertmanager payload and routes it to the runbook section most likely to explain it — no LLM call on that path, on purpose (more on that below).
Here’s the whole flow, start to finish:
rag ingest ../minikube-gitops-platform --glob "*.md"
# Embedding 56 chunks from ../minikube-gitops-platform ...
# Stored 56 chunks in rag.db
rag ask "why would ArgoCD show policy-constraints as SyncFailed?"And the answer I actually got back, verbatim, the first time I ran this against a real sibling repo:
ArgoCD may show policy-constraints as SyncFailed or OutOfSync initially because it depends on policy-templates' CRDs to exist first. If they have not finished reconciling yet, it could lead to this transient state. ArgoCD automatically retries to sync it with backoff until the dependencies are resolved, usually within a couple of minutes [RUNBOOK.md § 1. Spin it up (part 4/4)].
That citation isn’t decorative. It’s the actual mechanism I built the tool around: [source_file § section name (part N/M)] — file, section, and which fragment of a long section, if it got split. Click through, and you're at the exact paragraph, not "somewhere in a 400-line file."
How the pieces fit together
Chunking. The naive version of this splits a markdown file on ##/### headers and calls it a day. I did that first. It's wrong, and I'll get to exactly how wrong in a second, but the fix is simple: any section over 600 characters gets further split into paragraph-packed sub-chunks, so a single embedding vector never has to represent an entire five-paragraph "How to run this" section.
# rag/chunk.py
MAX_CHUNK_CHARS = 600def _split_long_text(text: str, max_chars: int) -> list[str]:
"""Split text into paragraph-packed chunks up to max_chars, never splitting
a paragraph (or a fenced code block) in half."""
...That “never splitting a fenced code block in half” part matters more than it sounds like it should. Runbooks are full of shell commands. Splitting a code block mid-command produces a chunk that’s syntactically garbage and semantically useless — worse than not chunking it at all, because now it’s actively diluting the good match with noise.
Embeddings. All local — sentence-transformers, all-MiniLM-L6-v2. No API key needed for this half, no cost, runs offline after the model downloads once. This was a deliberate choice: the embedding side runs on every ingest, potentially over hundreds of chunks, and I didn't want "can I re-index this repo" to depend on an API being up or a bill going up.
Storage and retrieval. A SQLite table, brute-force cosine similarity in numpy. No FAISS, no pgvector, no managed vector database. A repo’s worth of docs is a few dozen chunks — brute force is the fast path here, and reaching for a real vector index would have been solving a problem I don’t have.
Answering. This is the only part that touches a real API — OpenAI’s gpt-4o-mini, with a system prompt that's basically one sentence doing all the work:
SYSTEM_PROMPT = (
"You are an on-call assistant answering questions about an infrastructure repo using only "
"the provided doc excerpts. Each excerpt is labeled with its source file and section. "
"Answer concisely, and after any claim, cite the excerpt it came from as "
"[source_file § section]. If the excerpts don't contain the answer, say so plainly instead "
"of guessing."
)“Say so plainly instead of guessing” is the line I care most about in this whole codebase. An on-call tool that confidently makes something up is worse than no tool at all, because it costs you the time you would’ve spent actually finding the right answer, plus however long it takes you to notice it was wrong.
The bug that actually taught me something
Here’s the thing nobody tells you about RAG demos: they work great on the first three questions you try, because you subconsciously pick questions the naive version handles fine. Then you ask a real one and it falls over.
Mine fell over on: “how do I rotate the secret?”
With header-only chunking, that question retrieved the “Prerequisites” section as its top match instead of the actual command, which was buried three paragraphs into a much longer “How to run this” section. Both sections mention secrets. “Prerequisites” is short and generic, so it gets a denser, more “on-topic-on-average” embedding — the actual command was diluted by four other paragraphs of unrelated setup instructions sharing its vector.
I confirmed it wasn’t a fluke by actually scoring it: 0.32 for the wrong section, and the real command wasn’t even in the top few results. After adding the paragraph-level sub-chunking and bumping the default retrieval count from 4 to 6, the same question surfaced the exact command as the top-scored chunk — 0.49 instead of 0.32 — and the generated answer quoted it correctly.
That’s not a huge fix. It’s a few dozen lines. But it’s the difference between a demo that works when you already know the answer and a tool that works when you don’t, and I wouldn’t have found it without actually trying to break it with a real question instead of a curated one.
Extending it: from “answer my question” to “explain this alert”
The tool started as pure Q&A. The thing I added most recently is different in kind, not just degree: instead of a human typing a question, it now takes a real Prometheus Alertmanager webhook payload and tells you which runbook section explains it.
rag route examples/sample-alertmanager-payload.json[firing] PolicyConstraintSyncFailed (severity=warning)
✓ confident — top match 0.6190.619 [RUNBOOK.md § 1. Spin it up (part 4/4)]
0.604 [RUNBOOK.md § Troubleshooting (part 2/10)]
0.445 [RUNBOOK.md § Troubleshooting (part 10/10)]
0.410 [RUNBOOK.md § 4. Try the GitOps loop yourself (part 1/2)]
The interesting decision here isn’t the retrieval — it’s what I didn’t do: this path never calls an LLM. An alert firing (especially a flapping one that re-fires every few minutes) is not the place to add LLM latency and per-call cost to every single webhook delivery. rag ask stays the path for when a human wants a synthesized paragraph. rag route answers a narrower question fast — "which section is this about" — with a confidence score instead of prose:
DEFAULT_CONFIDENCE_THRESHOLD = 0.35def route_alert(store, alert, k=4, threshold=DEFAULT_CONFIDENCE_THRESHOLD):
query_embedding = embed([alert.query_text()])[0]
matches = store.top_k(query_embedding, k=k)
confident = bool(matches) and matches[0].score >= threshold
return RouteResult(alert=alert, matches=matches, confident=confident)Below 0.35, it doesn’t guess — it says “no confident match, page a human” instead of pointing you at a plausible-looking wrong answer. I tested this both ways on purpose: a real on-topic alert scored 0.619 (confident), and a deliberately unrelated one (I made up a solar-panel-output alert, just to see) scored 0.165 and correctly came back not-confident. A wrong-but-confident suggestion in the middle of an incident is worse than an honest shrug.
There’s also a real webhook receiver behind this now, not just a CLI:
uvicorn rag.webhook:app --port 8000YAML
# alertmanager.yml
receivers:
- name: runbook-router
webhook_configs:
- url: http://<host>:8000/alerts/routeFrequently Asked Questions
Why use SQLite and numpy instead of a dedicated vector database like pgvector or Qdrant?
- Scale: A single repository’s documentation usually yields anywhere from 20 to a few hundred chunks.
- Zero Infrastructure: Brute-force numpy cosine similarity over a few hundred 384-dimensional vectors takes less than 2 milliseconds on standard CPU hardware. Adding a managed or containerized vector database introduces deployment friction without performance benefits at this scale.
Why bypass the LLM entirely for the Alertmanager route (rag route)?
- Deterministic Speed: Alertwebhooks require immediate feedback during an incident. Local vector lookup delivers section matches in milliseconds.
- Cost & Flapping: Flapping alerts can fire dozens of times in short succession. Running an LLM synthesis step on every payload introduces API latency and unnecessary infrastructure costs.
How does the ingestion process prevent code snippets from breaking during chunking?
- Fenced Block Awareness: The chunker tracks markdown code fence markers (```). If a section exceeds the 600-character threshold, the splitter packs paragraphs up to the boundary without ever slicing through active code blocks or single bash commands.
Can this tool hallucinate incorrect recovery steps during an outage?
- Strict Prompt Constraints: The system prompt forbids speculation, explicitly instructing the model to return a plain refusal if context is missing.
- Mandatory Inline Citations: Every statement is mapped to a explicit tag ([file § section (part N/M)]), allowing the reader to click directly into the source file to verify context.
How does local embedding performance hold up without GPU acceleration?
- Lightweight Architecture: The underlying model (all-MiniLM-L6-v2) runs efficiently on standard x86 and ARM CPUs. Ingesting 50–100 documentation chunks takes roughly 1 to 2 seconds total on a modern laptop CPU.
📢 Have questions or feedback? Drop a comment below or connect with me on Twitter/X@spysood!
Originally published on Medium.