[{"content":" The short version, for anyone: This project has an automatic \u0026ldquo;quality gate\u0026rdquo; that blocks any code change which makes the system worse. But the gate\u0026rsquo;s own measurements were bouncing around between runs — because one early step uses an AI model that phrases things slightly differently each time. So the gate couldn\u0026rsquo;t tell a real problem from its own randomness, and on one run it \u0026ldquo;failed\u0026rdquo; for no real reason. The fix: freeze that non-deterministic step during measurement, so every run measures the same thing. The principle: a gate whose own measurement isn\u0026rsquo;t reproducible can\u0026rsquo;t gate anything.\nFull finding below.\nSurfaced by: first full end-to-end run of the Layer 3 runner · Related: ADR-016, ADR-020\nWhat happened The first full run of the Layer 3 runner produced Layer 2 numbers that moved against the committed baseline. The headline: decline_rate dropped 1.000 → 0.500, which under ADR-020 is a hard invariant and would fail the gate. But it\u0026rsquo;s only 2 no-match entries scored, so 0.5 is literally one entry flipping — one no-match description stopped declining and leaked a candidate.\nLayer 1 reproduced almost exactly (hit rate 1.000, MRR 0.918 vs 0.9177). Layer 2 did not.\nWhy it moved — not a regression, a reproducibility gap The movement is not a code or corpus regression. The store was freshly re-indexed and Layer 1 reproduced perfectly, so retrieval is sound. The cause is that the runner ran the full live graph, including the Decompose node.\nDecompose is LLM-driven and non-deterministic. Its output — the symptom breakdown that everything downstream retrieves against — varies run to run. So each run measures against a different set of symptoms, and the metrics wobble accordingly.\nThe project already anticipated exactly this: a frozen symptoms file exists specifically to hold Decompose\u0026rsquo;s output fixed so measurement is reproducible. The Layer 2 baseline was built against frozen symptoms. The runner was not — it bypassed the freeze and called the live graph end to end. So current and baseline were not measuring the same thing.\nWhy this matters for the gate The entire point of the gate is telling a real regression from noise. A gate that runs live Decompose can never do that for Layer 2: every run drifts by an unknown amount from Decompose alone, so any metric movement is ambiguous by construction. decline_rate dropping below its hard invariant on this run is the proof — it looks like a gate failure, but it\u0026rsquo;s Decompose variance on a 2-entry denominator, not a broken system. A gate that fires on its own measurement noise is the false-alarm failure mode ADR-020 warns about. It would get muted.\nThe fix The runner must measure Layer 2 against frozen symptoms, not live Decompose — the same way the baseline was built. Run the graph from the frozen symptoms starting point rather than letting Decompose run fresh. This makes the runner reproducible and makes current-vs-baseline an apples-to-apples comparison.\nRemaining downstream LLM variance (Retrieve / Assess / Diagnose) is a deferred question — whether it\u0026rsquo;s small enough to gate on, or whether more of the chain needs pinning, is answered once frozen-Decompose runs are compared across several repeats. That repeat data is also what tightens the provisional thresholds in ADR-020.\n","date":"2026-07-26T00:00:00+02:00","permalink":"/p/live-vs-frozen-decompose/","title":"A Quality Gate Must Be Reproducible Before It Can Gate Anything"},{"content":" The short version, for anyone: Tests that had been passing suddenly started failing, right after a code change — so the code change looked guilty. It wasn\u0026rsquo;t. The real cause was a database of pre-computed data that had quietly gone stale: it was excluded from version control, so the usual \u0026ldquo;has anything changed?\u0026rdquo; check showed everything clean while the thing the system actually depends on was out of date. The takeaway: a clean version-control status tells you nothing about the state of files it doesn\u0026rsquo;t track — and this is exactly the failure the project\u0026rsquo;s automatic quality gate is built to prevent.\nFull finding below.\nSurfaced by: first Layer 2 runs while building the Layer 3 runner · Related: ADR-020 (Layer 3 gate policy)\nWhat happened Mid-session, Layer 2 entries that pass in the committed baseline started failing. Two examples: L2-006 went from a correct top-1 hit to 0 candidates, declined, retrieved nothing; L2-001 went from the correct incident in its candidate set to 3 candidates, all wrong.\nOn the surface this looked like a regression — the system had gotten worse — and it appeared right after a code refactor, so the refactor was the obvious suspect.\nWhy it was NOT the obvious cause The refactor was innocent. It only moved a loop (run_suite extracted from main); it touched no retrieval or graph code. Isolation confirmed this:\nollama list — the embedder was loaded and fine. git status / git log — corpus untouched since the baseline commit. No data change. A second entry also mis-retrieved, so it wasn\u0026rsquo;t one bad entry — it was systemic to retrieval. Comparing to the committed baseline showed the affected entries used to pass. So the baseline was right and current had drifted — pointing away from a bad baseline and toward something environmental. Root cause The vector store on disk was stale. The corpus had grown from 15 to 20 documents in an earlier commit, but the ChromaDB index at data/chromadb had never been rebuilt after that growth. So retrieval was searching the old 15-doc index while the suite and baseline expected the current 20-doc / 107-chunk corpus. Wrong incidents came back, or none at all.\nThe reason git status gave no warning: data/chromadb is gitignored — it\u0026rsquo;s a generated artifact, not tracked. So git reported a clean tree while the actual thing retrieval depends on was silently out of date. A clean git tree says nothing about the state of the index.\nThe fix Re-index so the store matches the current corpus (python -m src.embedding). After re-indexing, L2-001 immediately found the expected incident again, and the full suite reproduced documented behavior.\nWhy this is load-bearing for the quality gate This is exactly the failure the Layer 3 runner\u0026rsquo;s mandatory re-index step prevents. A gate that evaluated without re-indexing would compare a fresh baseline against a possibly-stale store and report regressions that aren\u0026rsquo;t real — the false-alarm failure mode ADR-020 warns about. The re-index isn\u0026rsquo;t hygiene; it\u0026rsquo;s a correctness precondition, now demonstrated rather than assumed.\nLessons Gitignored artifacts have no version signal. A clean tree can sit on top of a stale generated dependency. Never infer store freshness from git. Diagnose environmental vs. logic failures before concluding. The symptom framed the refactor as guilty; the cause was a stale artifact. Isolation, not assumption, found it. The baseline was the tool that cracked it. Run current, diff against committed baseline, find the flipped entry, isolate the cause — the same procedure the quality gate automates, run here by hand. ","date":"2026-07-26T00:00:00+02:00","permalink":"/p/stale-vector-store-regression/","title":"A Stale Vector Store Caused a Phantom Regression"},{"content":" The short version, for anyone: The system finds past incidents by comparing meaning, but sometimes two very different failures use the same words — \u0026ldquo;Cloudflare,\u0026rdquo; \u0026ldquo;edge,\u0026rdquo; \u0026ldquo;database\u0026rdquo; — and it confidently returns the wrong one. This is the most dangerous kind of error: not a crash, not an obvious miss, but a confident wrong answer that would send an engineer down the wrong path during a live outage. Rather than hide it or fake a better score by deleting the test, I\u0026rsquo;ve kept it visible and documented — because knowing exactly where a reliability system fails is the reliability work, and this particular fix is real design work, not a quick tuning tweak.\nFull finding below.\nWhere it shows: Layer 1 retrieval and Layer 2 diagnosis — the same failure, twice. Status: Understood, not fixed. Kept visible on purpose.\nThe short version The system sometimes matches on shared words rather than shared cause, and does it with high confidence. A query and an incident can use the same vocabulary while describing completely different failures. The embedding scores them as close, the system returns the wrong incident, and nothing about the score signals that it\u0026rsquo;s wrong. This is the failure the whole project exists to guard against: not a crash, not an obvious miss, but a confident wrong answer that would mislead an on-call engineer during a live outage.\nInstance 1 — Layer 1 retrieval: the DDoS probe A no-match probe: \u0026ldquo;DDoS attack overwhelmed our CDN edge nodes and caused a 12-hour outage.\u0026rdquo; This is meant to retrieve nothing — the corpus has no DDoS incident. Against the 15-document corpus it correctly declined. Against the 20-document corpus it now matches a Cloudflare incident at distance 0.236 — well inside the 0.30 threshold, a confident hit.\nBut that Cloudflare incident is a configuration-error incident: a database access-control change that cascaded. It has nothing to do with a DDoS, which is a volumetric attack. The match is on surface vocabulary — \u0026ldquo;Cloudflare,\u0026rdquo; \u0026ldquo;edge,\u0026rdquo; \u0026ldquo;outage\u0026rdquo; — not on the failure mechanism. The probe stays classified as a no-match; its continued matching is the finding, not something to reclassify away.\nInstance 2 — Layer 2 diagnosis: the XID-wraparound attractor In the Layer 2 baseline, three descriptions produced a diagnosis of Postgres transaction-ID (XID) wraparound. Two were correct (a real Sentry Postgres incident). The third, L2-007, was wrong — it\u0026rsquo;s Roblox\u0026rsquo;s service-registry cascade, nothing to do with Postgres. But it shares symptom vocabulary — read-only, cascade, database — with the Sentry incident, and the model reached for the specific, authoritative-sounding failure it had seen before.\nWhy these are the same failure Both are the system latching onto a specific, plausible, well-documented failure because the surface features match, while the actual mechanism does not. The danger in both is the confidence. A vague wrong answer is easy to distrust. \u0026ldquo;PostgreSQL XID wraparound\u0026rdquo; and a 0.236 distance both look authoritative. During an incident, that\u0026rsquo;s worse than silence — it\u0026rsquo;s a false lead delivered with conviction.\nWhy it is being kept visible rather than patched It is honest signal. The Layer 1 decline rate is 0.500 — three of four no-match probes decline correctly, and the DDoS probe is the one that doesn\u0026rsquo;t. Forcing that number to look better by deleting the probe would hide a real property of the system. The metric is more useful with the known failure inside it.\nThe fix is not local. This isn\u0026rsquo;t a threshold to nudge or a prompt line to add. It\u0026rsquo;s a limitation of matching on embedding similarity and symptom vocabulary. Addressing it properly means giving the system more to discriminate on — richer context at retrieval time, or a verification step that checks whether the mechanism actually fits before returning a confident answer. That\u0026rsquo;s real design work, recorded here as the direction rather than attempted as a patch.\nWhat would actually address it Retrieval: search richer text so the match rests on more than a few shared nouns. More context gives the embedding more to separate genuinely-similar incidents from merely-similarly-worded ones. Diagnosis: a mechanism-check before a candidate is returned with high confidence — does the cited incident\u0026rsquo;s actual failure mode match the symptoms, or only their vocabulary? This is a grounding step one level deeper than citation-checking: not \u0026ldquo;is this incident real and retrieved,\u0026rdquo; but \u0026ldquo;does this incident actually explain what was reported.\u0026rdquo; Both are deferred. Both are the honest fix. Neither is a tuning change.\n","date":"2026-07-25T00:00:00+02:00","permalink":"/p/vocabulary-not-mechanism/","title":"Confident Matches on Vocabulary, Not Mechanism"},{"content":" The short version, for anyone: Before you can detect whether a system is getting worse, you need a measured snapshot of how good it is now — a baseline. Building that baseline for the diagnostic agent immediately caught a real bug that 30 existing unit tests had missed: a safeguard meant to strip out \u0026ldquo;made-up\u0026rdquo; citations was letting them through, as long as at least one real citation rode alongside. One genuine reference could smuggle in any number of invented ones. The lesson: unit tests check the cases you thought of; measuring against real output catches the ones you didn\u0026rsquo;t.\nFull finding below.\nMeasured on: 15 descriptions through the full agent · Result: first measured Layer 2 baseline; one grounding bug found and fixed.\nWhy this matters Layer 2 had 30 unit tests proving the machinery worked, and one integration run judged by eye. It had no numbers. You cannot build a regression gate — which detects regressions by comparing against a baseline — without a baseline to compare to. This is that baseline. Building it also caught a real bug the unit tests had missed.\nThe bug the baseline caught The first run reported 2 grounding violations — candidates citing incident IDs that were never retrieved. That number is supposed to be impossible by construction, so the tripwire fired.\nWhy it slipped through The grounding filter kept a candidate if any cited ID was real:\ncited = {...} # every id the model cited if cited \u0026amp; valid_ids: # at least one is real? grounded.append(d) # keep the whole thing, evidence unchanged So a candidate citing aws-s3-2017-02-28, gitlab-2017-01-31 passed — aws-s3 is real, the intersection is non-empty — and carried the fabricated gitlab citation through untouched. The filter checked that at least one citation was grounded. It never checked that every citation was. One real id smuggled in any number of invented ones.\nThe fix real = cited \u0026amp; valid_ids if not real: continue # nothing grounds this, drop it d[\u0026#34;evidence\u0026#34;] = \u0026#34;, \u0026#34;.join(sorted(real)) # keep only the real citations grounded.append(d) A candidate now survives only if it has a real citation, and its evidence is rewritten to contain only real citations. Fabricated ids are stripped, not tolerated. Covered by a new test that feeds one real and one fake id and asserts only the real one remains.\nWhy the unit tests missed it The existing grounding test used candidates that were entirely fabricated — those were correctly dropped. The gap was the mixed case: one real citation plus one fake. No test exercised it, so nothing failed. The scoring run on real model output was the first thing to hit it. That\u0026rsquo;s the point of scoring against real output: unit tests check the cases you thought of, evaluation catches the ones you didn\u0026rsquo;t.\nThe baseline (after the fix) Metric Value top-1 accuracy 0.625 (5 of 8 with a primary cause) any-hit rate 0.615 noise rate 0.444 decline rate 1.000 (2 of 2 no-match entries) mean candidates 1.80 grounding violations 0 mean iterations 2.20 No targets were set in advance — the first clean run is the baseline, so the gate enforces \u0026ldquo;do not fall below this,\u0026rdquo; not an invented number. The one exception is grounding violations, which isn\u0026rsquo;t a target but an invariant: it must be zero.\nReading it What works: decline rate is perfect — the reliability behaviour (declining rather than inventing) holds. Mean candidates fell from 5 to 1.8; the list no longer pads. Grounding is clean and now provably so.\nWhat doesn\u0026rsquo;t: noise rate 0.444 is high, but concentrated, not spread — two specific descriptions (L2-003, L2-007) produce almost all the noise in the whole suite. Every other entry is clean or nearly so. So the problem is two descriptions the agent handles badly, not a diffuse quality issue — a located problem to work on rather than a general sense that quality is mediocre.\nWhat this unblocks Layer 2 now has a committed baseline across seven metrics and a clean grounding invariant. The regression gate has something to regress against — and two concrete, located problems to work on: the noise concentrated in L2-003 and L2-007, and the XID-wraparound attractor (documented separately).\n","date":"2026-07-24T00:00:00+02:00","permalink":"/p/layer2-baseline-grounding-filter/","title":"The Layer 2 Baseline, and a Grounding Filter That Leaked"},{"content":" The short version, for anyone: A project\u0026rsquo;s automated tests are its safety net — but only if they get run. This suite took over 15 minutes, which in practice means developers skip it, which means the safety net isn\u0026rsquo;t there. Every test was correct; the problem was speed. This is the story of cutting it from 15m 46s to 8.5 seconds — not by removing tests, but by noticing the same expensive work was being redone dozens of times. The theme: correct is not the same as usable.\nFull technical walkthrough below.\nResult: Fast suite 15m 46s → 8.5s · Also found: an 82-vs-83 chunk discrepancy open since early July.\nFraming: this was not a bug Every test passed. Every test was correct. Nothing produced a wrong answer. The defect was that the suite took 15 minutes and 46 seconds, which in practice means you stop running it. A test suite you avoid running provides no safety at all, however correct it is.\nSo this is a performance defect in the test harness, not a bug in the system — worth being precise about, because the fix is different in kind. Nothing was repaired. Work that was being done repeatedly was made to happen once.\nThe symptom 85 passed, 1 deselected in 946.89s (0:15:46) The project\u0026rsquo;s stated test architecture is pure-core / impure-shell: pure logic tested in milliseconds, slow integration tests run deliberately. The 29 agent tests already demonstrated this — fully mocked, 0.19 seconds. So the architecture was right and the suite still took a quarter of an hour. Something was not matching the design.\nMeasuring instead of guessing pytest tests/ -m \u0026#34;not integration\u0026#34; --durations=20 --durations prints the slowest tests. The output split cleanly into two groups, and the distinction between them turned out to be the whole story.\nGroup 1 — time in setup: four tests, ~90–110 seconds each, all in setup.\nGroup 2 — time in call: four tests, ~105–217 seconds each, in the test body.\nAnd once setup was done, the tests themselves ran in under two seconds. The tests were never slow. The setup was.\nCause 1: a fixture doing the same work eight times @pytest.fixture def indexed_chunks(temp_chroma, corpus_path, expected_chunks): docs = load_documents(corpus_path) chunks = chunk_documents(docs) index_chunks(chunks) return chunks No scope argument, so pytest defaults to function scope: the fixture runs fresh for every test that requests it. Embedding 82 chunks through nomic-embed-text on CPU costs roughly 100 seconds. Four tests used the fixture. Four identical embedding passes, ~400 seconds, for one corpus that never changed between them.\nFix — session scope, so the corpus is embedded once and shared. One complication: temp_chroma used pytest\u0026rsquo;s function-scoped monkeypatch, which a session-scoped fixture can\u0026rsquo;t consume, so the replacement instantiates pytest.MonkeyPatch() directly and undoes it manually.\nThe tradeoff, stated plainly: function scope exists for isolation — every test gets a clean store. Session scope trades that away for speed. Acceptable here because these tests read from the store rather than corrupting it — and because a suite nobody runs is worse than a small isolation risk. Result: 15m 46s → 11m 25s.\nCause 2: a test embedding 82 chunks to count to 82 def test_vector_count_matches_chunk_count(corpus_path, expected_chunks): docs = load_documents(corpus_path) chunks = chunk_documents(docs) chunks_with_vectors = embed_chunks(chunks) assert len(chunks_with_vectors) == expected_chunks 110 seconds. Read the assertion: it checks that embed_chunks returns one vector per chunk it was given. It never inspects a single vector. That property does not depend on corpus size — five chunks prove it as well as 82. Slicing to chunks[:5] took it from 110s → 4.67s, and the assertion got better: the real property is \u0026ldquo;same number out as in,\u0026rdquo; not \u0026ldquo;82.\u0026rdquo;\nCause 3: genuinely slow tests were not labelled Three tests could not be shrunk — test_store_and_search, test_search_with_filter, test_idempotency — because they genuinely need a fully populated store; that\u0026rsquo;s the thing under test. They needed a label, not an optimisation:\n[pytest] markers = slow: embeds the corpus, needs Ollama integration: full end-to-end run across components Three speeds instead of two:\npytest -m \u0026#34;not slow and not integration\u0026#34; # 8.5s - run constantly pytest -m \u0026#34;slow\u0026#34; # ~5 min - before committing pytest -m \u0026#34;integration\u0026#34; # ~100 min - deliberately The middle tier is the one that was missing. Previously \u0026ldquo;not integration\u0026rdquo; meant 15 minutes, so the only real choice was 15 minutes or nothing.\nResult Stage Time Start 15m 46s After session-scoped fixtures 11m 25s After rewriting the count test ~9m After marking slow tests 8.5s 78 tests run by default. 8 parked behind markers.\nThe side finding: 82 vs 83 conftest.py expects 82 chunks and the chunking test passes — so chunking really does produce 82. But the live store at data/chromadb held 83. This discrepancy had been open since early July. It\u0026rsquo;s now explained: the corpus is correct, and the live store contained one orphan chunk from an earlier ingest — a document later changed or removed without the store being rebuilt. A clean re-index resolves it, worth doing before further calibration, since a stale chunk can surface in retrieval and quietly distort a measurement.\nWhat this is a story about Correct is not the same as usable. Every test passed and asserted something true. The suite still failed at its actual job — being run often enough to catch regressions early. Measure before optimising: the setup-vs-call split pointed straight at two different causes needing two different fixes. Ask what a test is actually asserting: the 110-second count test verified a property that had nothing to do with corpus size. And some slowness is real — three tests genuinely need a populated store, so the fix was a marker, letting them run deliberately rather than be skipped by accident.\n","date":"2026-07-24T00:00:00+02:00","permalink":"/p/test-suite-too-slow/","title":"The Test Suite That Was Too Slow To Run"},{"content":" The short version, for anyone: The diagnostic layer breaks an incident into short symptom fragments and searches for each. But it was reusing a \u0026ldquo;closeness\u0026rdquo; cutoff that had been tuned for full, richly-worded questions — and short fragments always score as less close, so 25 of 27 symptoms found the right incident and then had it thrown away. Giving Layer 2 its own, looser cutoff fixed most of it. But the honest conclusion isn\u0026rsquo;t \u0026ldquo;0.36 is the right number\u0026rdquo; — it\u0026rsquo;s that a closeness score alone can\u0026rsquo;t cleanly separate signal from noise on short fragments, because in the data the two genuinely overlap. Naming that limit is the point.\nFull finding below.\nMeasured on: 15 incident descriptions, 27 frozen symptoms, 5 no-match symptoms.\nThe problem in one line Layer 2 was using Layer 1\u0026rsquo;s threshold. Layer 1\u0026rsquo;s threshold was tuned on complete questions. Layer 2 sends symptom fragments. Fragments score worse, so everything was thrown away.\nWhy fragments score worse A post-mortem chunk describes a whole incident — trigger, failure, cascade, recovery — many concepts in one paragraph. A complete question matches that richness. A fragment matches one small part of it, so the distance is worse even though it describes the same event. The whole description is closer to the document than any of its parts.\nInput type Typical distance Layer 1 complete questions 0.20 – 0.27 Layer 2 symptom fragments 0.32 – 0.41 The threshold both were using 0.30 The cutoff sits in the gap. Layer 1 clears it every time. Layer 2 never does.\nWhat was actually failing Verdict Count Meaning PASS 0 found, kept, visible to the agent THRESHOLD 25 found the right document, then discarded RANK 0 found but ranked too deep MISS 2 never found at all 25 of 27 symptoms found the correct document and had it thrown away. Retrieval was working. Filtering was calibrated for the wrong input — the correct document was often ranked first, at distances like 0.32–0.33, just above the 0.30 cutoff.\nThe sweep, and the decision Each threshold was tested against the same frozen symptoms. The usable range came out to 0.34–0.36: below it, entries get no evidence; above 0.38 junk starts leaking badly, and at 0.40 every no-match probe leaks — decline behaviour collapses entirely. 0.36 is the best point: 9 of 13 entries get evidence, junk still only 1 of 5. Decision: LAYER2_THRESHOLD = 0.36, while Layer 1 keeps 0.30.\nWhat this does not fix Stated plainly, because 0.36 is not a solution:\n4 of 13 descriptions still get nothing — the agent still fails on those. 4 symptoms return the wrong document — the agent reasons over a wrong incident, which during an outage is worse than returning nothing. 1 junk symptom leaks — a description with nothing matching gets treated as if it matched. Layer 2 goes from never working to working roughly two-thirds of the time, with some false leads.\nThe overlap problem The reason no threshold is clean: the distance bands interleave. A junk symptom (\u0026ldquo;air conditioning failed\u0026rdquo;, 0.362) can score worse than a real one, but another junk symptom (\u0026ldquo;machines shut themselves down\u0026rdquo;, 0.334) scores better than a real hit (\u0026ldquo;servers dropping everything\u0026rdquo;, 0.344). No cutoff separates them, because the separation doesn\u0026rsquo;t exist in the data. More samples would describe the overlap more precisely; they wouldn\u0026rsquo;t create a boundary.\nSo the honest conclusion is not \u0026ldquo;0.36 is the right number.\u0026rdquo; It\u0026rsquo;s: a distance score alone cannot separate signal from noise on short symptom fragments. 0.36 is a round number picked from where the table turns — a working setting that makes the component functional, to be revisited when the suite grows. Confidence in the exact number is low, deliberately so.\nWhat to try next The threshold fix treats the symptom; the cause is that fragments carry less signal than whole descriptions. The real direction: search the full description alongside each symptom and merge the results — matching rich text against rich chunks, the comparison the embedding model is actually good at. Evidence: the same content as complete descriptions scored 0.870 at threshold 0.30, while split into fragments it scores zero at the same threshold. That would widen the gap between signal and noise rather than moving a line through the middle of it.\n","date":"2026-07-24T00:00:00+02:00","permalink":"/p/layer2-threshold/","title":"Why Layer 2 Needed Its Own Threshold"},{"content":" The short version, for anyone: A search system was returning \u0026ldquo;nothing found\u0026rdquo; — while the correct answer sat in its database, ranked first. Nothing had crashed; every function did exactly what it was written to do. This is the story of discovering that the code was fine and the system was still broken — because the test suite had been unknowingly grading the system on easy questions, and a hidden limit was throwing away correct answers before anyone looked at them. It\u0026rsquo;s a good example of the kind of failure that doesn\u0026rsquo;t show up as an error message — the most dangerous kind.\nBelow is the full technical investigation. Skip it freely — the summary above is the point.\nLayer: 1 (retrieval) · Outcome: No defect found in application code. Two real problems found anyway.\nWhere it started Layer 2 was blocked. A retrieval probe returned zero results against a store holding 83 chunks. No error, no exception — just an empty list.\nThe working theory was environmental: the embedding model wasn\u0026rsquo;t loaded in Ollama. That theory mattered, because Layer 2\u0026rsquo;s \u0026ldquo;the agent declines honestly\u0026rdquo; framing depended on knowing whether the empty result was a genuine no-match or a broken pipe.\nFirst command of the day killed it. ollama list showed nomic-embed-text, 274 MB, installed four weeks earlier. The model was fine. So the bug was real, and somewhere in code I\u0026rsquo;d written.\nThe elimination Rather than guess, I listed every link in the chain and tested each in order. Six suspects.\nSuspect Result Model not installed Present Wrong database path App points at data/chromadb — correct Store empty 83 chunks Wrong distance measure hnsw:space: cosine, set explicitly Embedding prefix mismatch search_document: / search_query: correctly paired Threshold comparison inverted distance \u0026lt;= threshold — right direction Every single one came back clean. Every function I checked did exactly what it was written to do. That was the first real finding: the system was correct and still useless.\nThree false leads, all self-inflicted The diagnostic script lied to me three times, and each time for the same reason.\nIt hardcoded the database path. Pointed at data/chroma instead of data/chromadb. ChromaDB doesn\u0026rsquo;t error on an empty folder — it silently creates a blank database. So the script made an empty store and correctly reported it was empty. Cost: half an hour convinced the corpus had vanished.\nIt reimplemented the embedding call. Called Ollama directly with plain text, no prefix. The corpus was embedded with search_document:. So it was comparing unlabelled queries against labelled documents — a mismatched measurement that produced a real-looking number I then reasoned from.\nIt used a candidate depth the system doesn\u0026rsquo;t use. I set 15 to see where documents ranked. The system uses 5. That made the diagnostic disagree with the sweep, and reconciling the two is what exposed the actual bug.\nThe lesson generalises: a diagnostic that restates what the system defines will eventually disagree with it, and it will disagree quietly. Every fix was the same — import the real code instead of copying it.\nWhat was actually wrong Two things, neither a broken function.\nThe evaluation suite was easier than reality The 31-query suite was written after reading and normalising all 15 post-mortems. So it reuses the documents\u0026rsquo; own vocabulary without meaning to. It scored the system 1.000 while plain-English questions were failing.\nTo measure this instead of asserting it, I built a paired suite: same entries, same expected_doc_id, same difficulty, only the query text rewritten in plain English. No-match probes and filter queries held constant as a control.\nSuite Hit rate @ 0.30 MRR Original wording 1.000 0.949 Plain English 0.826 0.783 The control held — decline rate identical across both suites at every threshold. So the gap came from wording and nothing else.\nThe clearest single illustration:\nQuery Distance \u0026ldquo;BGP route leak\u0026rdquo; — the document\u0026rsquo;s own words 0.1996 \u0026ldquo;database ran out of connections\u0026rdquo; — same kind of event, plain words 0.3035 The distance was largely measuring word overlap. The threshold was acting as a jargon filter.\ntop_k was discarding correct answers before checking them The plain-English curve flattened at 0.913 and never moved, even at threshold 0.50 where the filter does nothing. That meant some failures weren\u0026rsquo;t threshold failures at all.\nA per-query diagnostic found the reason: one query\u0026rsquo;s correct document sat at rank 6, distance 0.2888 — comfortably inside the 0.30 threshold, never looked at, because top_k=5 truncated the list first.\nThe candidate count was overriding the relevance rule. Raising it to 10:\nBefore After Hit rate @ 0.30 0.826 0.870 Ceiling 0.913 0.957 Decline rate 0.600 0.600 Real answers gained, no noise admitted. The same number turned out to be hardcoded in four places — retrieve(), run_sweep, score_filter_query, and the agent\u0026rsquo;s retrieve node at 3. It\u0026rsquo;s now one constant, DEFAULT_TOP_K.\nWorth noting the agent was retrieving three candidates per symptom while Layer 1 used five. The component doing the harder work had the least evidence.\nThe decision I didn\u0026rsquo;t make Two queries still fail at 0.30, missing by 0.0055 and 0.0092. Moving the threshold to 0.35 would recover both. I left it alone.\nThe distance bands overlap — a junk probe scored 0.215, closer than four of five real queries. There is no value that separates good from bad, so any number is a tradeoff, and one tuned to clear 23 specific queries is fitting the sample rather than the problem. 0.35 would also drop decline rate from 0.600 to 0.200 — triple the noise to recover two borderline cases.\nAnd the failure modes aren\u0026rsquo;t equal. Returning nothing costs an engineer time. Returning the wrong past incident during a live outage sends them after the wrong root cause. Strict is the right side to fail on.\nWhat this is actually a story about Not a bug hunt. Every function was correct. It\u0026rsquo;s about a system that stays up, throws no errors, and quietly returns nothing while holding the answer. During a live outage it would tell an engineer that nothing similar has ever happened — with the matching post-mortem in hand, ranked first.\nAnd it\u0026rsquo;s about the evaluation framework marking its own homework. The suite said 1.000. The system was at 0.826 for anyone who hadn\u0026rsquo;t read the corpus. The measurement was wrong in the flattering direction, which is the direction you don\u0026rsquo;t check.\nThe fix for that isn\u0026rsquo;t a better threshold. It\u0026rsquo;s a regression gate — re-running these measurements on every corpus change and reporting when the numbers move. You don\u0026rsquo;t calibrate once; you build the thing that notices when calibration has gone stale.\n","date":"2026-07-23T00:00:00+02:00","permalink":"/p/chasing-a-bug-that-didnt-exist/","title":"Chasing a Bug That Didn't Exist"},{"content":" The short version, for anyone: The system had two different jobs that happened to share one piece of code: ranking results by closeness, and fetching an exact set of records that match a filter. A cutoff meant only for the first job was silently being applied to the second — but it stayed harmless as long as a temporary setting was loose enough to never cut anything. The moment that setting was tightened to a realistic value, filter accuracy collapsed from 100% to 33%. The bug had been there since the first line of code; changing one config value exposed it. The lesson: tests passing at a permissive setting are not proof of correctness.\nFull bug report below.\nID BUG-001 Component retrieve() — src/embedding.py Severity Major · Priority High Status Closed — fixed and verified Related TP-001, ADR-013 What happened RELEVANCE_THRESHOLD was lowered from 1.0 to 0.30. Semantic queries improved. Filter queries collapsed. Filter precision, recall, and exact-match all dropped from 1.000 to 0.333 — one of three queries passing. The same value across three runs. Not noise. A logic bug.\nThe defect was already there. At 1.0, nothing ever got cut, so the bad path never ran. A config value changed and the latent bug surfaced.\nEvidence Metric t = 1.0 t = 0.30 Delta Hit rate@5 1.000 1.000 — MRR 0.949 0.949 — Section accuracy 0.435 0.435 — Decline rate 0.000 0.600 +0.600 Filter precision / recall / exact 1.000 0.333 −0.667 Regression is isolated. Everything else held.\nRoot cause retrieve() applied the cosine-distance threshold to every query — including metadata-filtered ones:\n# Keep only results close enough to be relevant relevant = [r for r in results if r[\u0026#34;distance\u0026#34;] \u0026lt;= threshold] # applied unconditionally Two things are wrong. The list comprehension has no condition — every caller gets the distance cutoff, so semantic ranking and metadata set-retrieval share one path. And the docstring states it as a flat rule, with no sign that a metadata query might need different treatment. The defect is in the design, not a slip in the code.\nA filter query like \u0026ldquo;minor-severity incidents\u0026rdquo; is a set question. The right answer is every document matching the filter — it doesn\u0026rsquo;t matter how semantically close the query phrase sits to the chunk text; the metadata already decided. At 1.0, nothing was ever discarded, so the wrong logic produced correct output by accident. At 0.30, documents that matched the filter correctly but sat far in cosine distance got silently dropped. Filter scoring is exact set-match — lose one document, lose the query.\nThe real defect: two retrieval modes — semantic ranking and metadata set-retrieval — forced through one code path, with a parameter meant for one mode leaking into the other.\nFix retrieve() now accepts threshold=None — no distance cutoff, return all metadata-matched results, ranked. The call site for filter scoring passes threshold=None; semantic queries keep the real threshold. The two modes are now distinguished at the call site.\nFixed at the source — retrieve(), not the eval harness — so every downstream caller inherits it, including the Layer 2 diagnostic agent.\nWhat was missed score_filter_query passes top_k=5. If a metadata filter matches more than 5 documents, the vector store returns only the top 5 by distance, and set-match scoring counts the rest as missing. Harmless now — no filter in the 15-document corpus exceeds 5 — but the same defect class: a semantic parameter constraining a metadata query. Tracked separately, to fix before the corpus grows.\nLessons learned 1. Passing tests at a permissive setting are not evidence of correctness. The bug existed from the first line of code. RELEVANCE_THRESHOLD = 1.0 meant the faulty branch never discarded anything. The placeholder was even flagged in the source (# loose placeholder). Knowing a setting is temporary is not the same as testing what happens when it changes.\n2. Partial metric reporting hides regressions. The threshold sweep tracked 3 of 5 metrics — hit rate, MRR, decline rate. It \u0026ldquo;confirmed\u0026rdquo; 0.30 as optimal while silently breaking a metric it did not watch. Caught only on the confirming run that reported the full set. Fix: full metrics on every run, enforced by exit criteria.\n3. Determinism separates bugs from noise. The first thought was jitter. Three identical runs at 0.333 reclassified it as logic, worth investigating.\n","date":"2026-06-29T00:00:00+02:00","permalink":"/p/bug-001-distance-threshold/","title":"BUG-001 — A Threshold Applied Where It Didn't Belong"},{"content":"Most ML portfolios show a model that runs once. The interesting engineering is in the part that comes after: keeping it running. In my experience, ML systems fail at three predictable points.\n1. Bad data gets in The model trains or scores on corrupt input and nobody notices until the numbers are wrong three steps downstream. The fix is boring and essential: validate data before it reaches the model — missing values, duplicates, malformed headers, schema drift.\n2. The model drifts in production A model that passed every test on launch day quietly degrades as the world changes underneath it. The first sign shouldn\u0026rsquo;t be an angry user — it should be a metric. That means monitoring distribution shift (PSI, Wasserstein) and gating deployments on it.\n3. The same failure repeats An outage happens, someone writes a post-mortem, it gets buried in a wiki, and six months later the same thing happens again. Turning past failures into searchable, deployment-gating feedback is the third layer — and the one almost nobody builds.\nOne principle — reliability — across three layers: data, model, system. That\u0026rsquo;s the thread running through everything on my projects page.\n","date":"2026-02-12T00:00:00+01:00","permalink":"/p/the-three-layers-where-ml-systems-fail/","title":"The three layers where ML systems fail"},{"content":"This is where I keep my notes on building ML systems that stay working.\nI\u0026rsquo;m a slow, deep learner — I\u0026rsquo;d rather understand one thing from first principles than skim ten. So instead of a static résumé, I keep a working notebook: what I\u0026rsquo;m building, what broke, and what I learned fixing it.\nIf you\u0026rsquo;re a recruiter or engineer in Berlin looking at my projects, this blog is the \u0026ldquo;why\u0026rdquo; behind them. If you\u0026rsquo;re learning MLOps yourself, maybe some of it saves you a wrong turn.\nMore soon.\n","date":"2026-02-10T00:00:00+01:00","permalink":"/p/why-this-site-exists/","title":"Why this site exists"},{"content":"Before I moved toward machine learning, my first real role was a front-end development internship at Mandala Infosys in Kathmandu, from April to July 2024. On paper it was an HTML-and-CSS job. In practice it was a hybrid — half building, half translating between what clients wanted and what the technical team could build — and the translating half is where I learned the most.\nThe build half The straightforward part was writing the front end. I built responsive interfaces in HTML, CSS, and JavaScript: semantic navigation components, landing-page layouts with formatted and optimised imagery, and form pages. It was a normal agile setup — sprint planning, code reviews, iterating on feedback — and it\u0026rsquo;s where I got my first real taste of shipping something other people would actually use, on a deadline, with someone reviewing my work.\nThat was valuable. But it wasn\u0026rsquo;t the part that changed how I think.\nThe half that mattered The more unusual side of the role put me between the client and the developers. I\u0026rsquo;d sit with customers to elicit what they actually wanted from a website — which, early on, I learned is almost never what they first say they want. So I started building small reference builds: quick, concrete mock-ups I could demo, because a vague request turns specific fast the moment someone can point at a real screen and say \u0026ldquo;not that — this.\u0026rdquo;\nFrom there I\u0026rsquo;d draft solution options for the technical team to weigh, and this is where the actual lesson lived: feature prioritisation. Clients want everything. Time and budget don\u0026rsquo;t allow everything. My job became walking non-technical people through the opportunity cost of each choice — \u0026ldquo;if we build this, here\u0026rsquo;s what it costs, and here\u0026rsquo;s what it pushes out of scope\u0026rdquo; — so they could make an informed trade-off instead of a wish list.\nSaying \u0026ldquo;yes, and here\u0026rsquo;s what that costs you\u0026rdquo; turned out to be far more useful than saying \u0026ldquo;yes\u0026rdquo; to everything.\nWhy this connects to reliability At the time I didn\u0026rsquo;t see the thread. I do now. The work I care about today — ML reliability, keeping systems trustworthy in production — is also mostly about honest trade-offs. When I decide not to auto-retrain a drifted model, or keep a known failure visible instead of hiding it behind a nicer metric, or refuse to cite a number I haven\u0026rsquo;t measured — that\u0026rsquo;s the same instinct I first practised in a small office in Kathmandu, explaining to a client why we shouldn\u0026rsquo;t build the thing they asked for.\nBuild the thing. But stay honest about what every decision trades away. That\u0026rsquo;s the non-code half of engineering, and it\u0026rsquo;s the half I\u0026rsquo;d bet on.\n","date":"2024-08-01T00:00:00+02:00","permalink":"/p/front-end-internship-lessons/","title":"The Non-Code Half of Engineering: Lessons from a Front-End Internship"}]