The diagnostic agent was reusing a cutoff tuned for full questions — but it sends short symptom fragments, which score worse, so everything was being thrown away. And why 0.36 is a working setting, not a solution.
The retrieval code was correct — and still useless. The real culprits were an eval suite grading itself too kindly and a silent truncation dropping right answers.
A latent bug that stayed invisible until a placeholder config changed from 1.0 to 0.30 — because two different retrieval modes were forced through one code path.
Bad data, silent drift, and lessons that never stick — and how to harden each.
A public notebook for building reliable ML systems.