01The tool, and the risk in it
I built a /sprint-documentation-review command that runs every sprint, unattended, across whatever stories shipped that sprint. It flags each one as "needs docs" or not, proposes the updates, and a human approves them. That approval step looks like the safety net. It isn't the one that counts.
02Why recall is the number that matters
You can only approve what the system puts in front of you. A story the classifier wrongly marks "no docs" never shows up in the review, so the human can't catch it. The gate stops bad proposals; it can't stop missing ones.
So recall is what actually protects the docs, the share of genuinely doc-worthy changes the classifier catches. A false positive is noise a human clears in a second. A false negative is invisible, a gap you'd otherwise find through a support ticket weeks later. This eval optimizes for recall, not accuracy, because the two errors have very different costs.
03What I built
- A reference set of 29 stories across two sprints, describing a category-standard messaging product. Fictional, so it's shareable.
- Hand-labeled ground truth: each story marked needs-docs yes/no against a product-truth definition, held in a separate answer key the classifier never sees.
- A runner that applies the command's real classification rules with a single blind agent, then scores the confusion matrix, precision, recall, and a diagnosis of every error.
The set leans on enhancements to existing features on purpose, the exact case a naive "new feature only" classifier drops.
04v1: a perfect score I didn't trust
v1 scored a perfect 100%, recall and precision both. Instead of shipping that, I treated it as suspect. A perfect score usually means the test is too easy.
It was. My story descriptions leaked the answers, phrases like "same UI" and "no user-facing change". The classifier was reading labels, not judging ambiguous stories. A 100% that comes from an easy test is exactly the "green doesn't mean correct" failure mode the tool is supposed to guard against.
05v2: hardening the test until it could fail
v2 wasn't a change to the classifier. It was a change to the test. I rewrote all 29 stories in raw Jira-ticket voice, where user impact has to be inferred: "migrate to BullMQ, payloads unchanged" instead of "internal, no user change".
Recall held at 100%. That's the result worth having: the classifier genuinely infers doc-need from a bare ticket, it wasn't just matching tells. The harder test also surfaced the real weak spot, one false positive on a reliability change (a webhook retry), which is the cheap kind of error. The frontier moved from recall to precision.
100%
recall, held from v1 to v2 (16/16 doc-worthy stories caught) even after the test was rewritten to hide the answers.
100 → 94%
precision, v1 to v2. The one new error is an over-flag on a reliability change, the cheap kind to make.
06The honest limit on the number
This is a small set. 100% recall means zero misses on 16 doc-worthy stories, not a perfect classifier. By the rule of three, zero misses on 16 is still consistent with a true recall near 80%. That's why the two-run story matters more than the headline number, and why I'd rather state the limit than lean on the 100%.
07What I deliberately left as future work
- Stage 2, output quality. This eval covers routing, which stories need docs. The command also drafts the doc text, and judging that is a separate, softer eval with no single right answer. Sketched, not built.
- v3, scenario ambiguity. v2 hardened the phrasing; the scenarios are still one clean change per ticket. Real sprints bring compound and vague tickets, and that's where recall would finally bend.
Both are noted so the map is visible. One eval done well beats three half-built.