Finding on Nikhil Adhikari

A Quality Gate Must Be Reproducible Before It Can Gate Anything

Sun, 26 Jul 2026 00:00:00 +0200

The short version, for anyone: This project has an automatic “quality gate” that blocks any code change which makes the system worse. But the gate’s own measurements were bouncing around between runs — because one early step uses an AI model that phrases things slightly differently each time. So the gate couldn’t tell a real problem from its own randomness, and on one run it “failed” for no real reason. The fix: freeze that non-deterministic step during measurement, so every run measures the same thing. The principle: a gate whose own measurement isn’t reproducible can’t gate anything.

Full finding below.

Surfaced by: first full end-to-end run of the Layer 3 runner · Related: ADR-016, ADR-020

What happened

The first full run of the Layer 3 runner produced Layer 2 numbers that moved against the committed baseline. The headline: decline_rate dropped 1.000 → 0.500, which under ADR-020 is a hard invariant and would fail the gate. But it’s only 2 no-match entries scored, so 0.5 is literally one entry flipping — one no-match description stopped declining and leaked a candidate.

Layer 1 reproduced almost exactly (hit rate 1.000, MRR 0.918 vs 0.9177). Layer 2 did not.

Why it moved — not a regression, a reproducibility gap

The movement is not a code or corpus regression. The store was freshly re-indexed and Layer 1 reproduced perfectly, so retrieval is sound. The cause is that the runner ran the full live graph, including the Decompose node.

Decompose is LLM-driven and non-deterministic. Its output — the symptom breakdown that everything downstream retrieves against — varies run to run. So each run measures against a different set of symptoms, and the metrics wobble accordingly.

The project already anticipated exactly this: a frozen symptoms file exists specifically to hold Decompose’s output fixed so measurement is reproducible. The Layer 2 baseline was built against frozen symptoms. The runner was not — it bypassed the freeze and called the live graph end to end. So current and baseline were not measuring the same thing.

Why this matters for the gate

The entire point of the gate is telling a real regression from noise. A gate that runs live Decompose can never do that for Layer 2: every run drifts by an unknown amount from Decompose alone, so any metric movement is ambiguous by construction. decline_rate dropping below its hard invariant on this run is the proof — it looks like a gate failure, but it’s Decompose variance on a 2-entry denominator, not a broken system. A gate that fires on its own measurement noise is the false-alarm failure mode ADR-020 warns about. It would get muted.

The fix

The runner must measure Layer 2 against frozen symptoms, not live Decompose — the same way the baseline was built. Run the graph from the frozen symptoms starting point rather than letting Decompose run fresh. This makes the runner reproducible and makes current-vs-baseline an apples-to-apples comparison.

Remaining downstream LLM variance (Retrieve / Assess / Diagnose) is a deferred question — whether it’s small enough to gate on, or whether more of the chain needs pinning, is answered once frozen-Decompose runs are compared across several repeats. That repeat data is also what tightens the provisional thresholds in ADR-020.

A Stale Vector Store Caused a Phantom Regression

Sun, 26 Jul 2026 00:00:00 +0200

The short version, for anyone: Tests that had been passing suddenly started failing, right after a code change — so the code change looked guilty. It wasn’t. The real cause was a database of pre-computed data that had quietly gone stale: it was excluded from version control, so the usual “has anything changed?” check showed everything clean while the thing the system actually depends on was out of date. The takeaway: a clean version-control status tells you nothing about the state of files it doesn’t track — and this is exactly the failure the project’s automatic quality gate is built to prevent.

Full finding below.

Surfaced by: first Layer 2 runs while building the Layer 3 runner · Related: ADR-020 (Layer 3 gate policy)

What happened

Mid-session, Layer 2 entries that pass in the committed baseline started failing. Two examples: L2-006 went from a correct top-1 hit to 0 candidates, declined, retrieved nothing; L2-001 went from the correct incident in its candidate set to 3 candidates, all wrong.

On the surface this looked like a regression — the system had gotten worse — and it appeared right after a code refactor, so the refactor was the obvious suspect.

Why it was NOT the obvious cause

The refactor was innocent. It only moved a loop (run_suite extracted from main); it touched no retrieval or graph code. Isolation confirmed this:

ollama list — the embedder was loaded and fine.
git status / git log — corpus untouched since the baseline commit. No data change.
A second entry also mis-retrieved, so it wasn’t one bad entry — it was systemic to retrieval.
Comparing to the committed baseline showed the affected entries used to pass. So the baseline was right and current had drifted — pointing away from a bad baseline and toward something environmental.

Root cause

The vector store on disk was stale. The corpus had grown from 15 to 20 documents in an earlier commit, but the ChromaDB index at data/chromadb had never been rebuilt after that growth. So retrieval was searching the old 15-doc index while the suite and baseline expected the current 20-doc / 107-chunk corpus. Wrong incidents came back, or none at all.

The reason git status gave no warning: data/chromadb is gitignored — it’s a generated artifact, not tracked. So git reported a clean tree while the actual thing retrieval depends on was silently out of date. A clean git tree says nothing about the state of the index.

The fix

Re-index so the store matches the current corpus (python -m src.embedding). After re-indexing, L2-001 immediately found the expected incident again, and the full suite reproduced documented behavior.

Why this is load-bearing for the quality gate

This is exactly the failure the Layer 3 runner’s mandatory re-index step prevents. A gate that evaluated without re-indexing would compare a fresh baseline against a possibly-stale store and report regressions that aren’t real — the false-alarm failure mode ADR-020 warns about. The re-index isn’t hygiene; it’s a correctness precondition, now demonstrated rather than assumed.

Lessons

Gitignored artifacts have no version signal. A clean tree can sit on top of a stale generated dependency. Never infer store freshness from git.
Diagnose environmental vs. logic failures before concluding. The symptom framed the refactor as guilty; the cause was a stale artifact. Isolation, not assumption, found it.
The baseline was the tool that cracked it. Run current, diff against committed baseline, find the flipped entry, isolate the cause — the same procedure the quality gate automates, run here by hand.

Confident Matches on Vocabulary, Not Mechanism

Sat, 25 Jul 2026 00:00:00 +0200

The short version, for anyone: The system finds past incidents by comparing meaning, but sometimes two very different failures use the same words — “Cloudflare,” “edge,” “database” — and it confidently returns the wrong one. This is the most dangerous kind of error: not a crash, not an obvious miss, but a confident wrong answer that would send an engineer down the wrong path during a live outage. Rather than hide it or fake a better score by deleting the test, I’ve kept it visible and documented — because knowing exactly where a reliability system fails is the reliability work, and this particular fix is real design work, not a quick tuning tweak.

Full finding below.

Where it shows: Layer 1 retrieval and Layer 2 diagnosis — the same failure, twice. Status: Understood, not fixed. Kept visible on purpose.

The short version

The system sometimes matches on shared words rather than shared cause, and does it with high confidence. A query and an incident can use the same vocabulary while describing completely different failures. The embedding scores them as close, the system returns the wrong incident, and nothing about the score signals that it’s wrong. This is the failure the whole project exists to guard against: not a crash, not an obvious miss, but a confident wrong answer that would mislead an on-call engineer during a live outage.

Instance 1 — Layer 1 retrieval: the DDoS probe

A no-match probe: “DDoS attack overwhelmed our CDN edge nodes and caused a 12-hour outage.” This is meant to retrieve nothing — the corpus has no DDoS incident. Against the 15-document corpus it correctly declined. Against the 20-document corpus it now matches a Cloudflare incident at distance 0.236 — well inside the 0.30 threshold, a confident hit.

But that Cloudflare incident is a configuration-error incident: a database access-control change that cascaded. It has nothing to do with a DDoS, which is a volumetric attack. The match is on surface vocabulary — “Cloudflare,” “edge,” “outage” — not on the failure mechanism. The probe stays classified as a no-match; its continued matching is the finding, not something to reclassify away.

Instance 2 — Layer 2 diagnosis: the XID-wraparound attractor

In the Layer 2 baseline, three descriptions produced a diagnosis of Postgres transaction-ID (XID) wraparound. Two were correct (a real Sentry Postgres incident). The third, L2-007, was wrong — it’s Roblox’s service-registry cascade, nothing to do with Postgres. But it shares symptom vocabulary — read-only, cascade, database — with the Sentry incident, and the model reached for the specific, authoritative-sounding failure it had seen before.

Why these are the same failure

Both are the system latching onto a specific, plausible, well-documented failure because the surface features match, while the actual mechanism does not. The danger in both is the confidence. A vague wrong answer is easy to distrust. “PostgreSQL XID wraparound” and a 0.236 distance both look authoritative. During an incident, that’s worse than silence — it’s a false lead delivered with conviction.

Why it is being kept visible rather than patched

It is honest signal. The Layer 1 decline rate is 0.500 — three of four no-match probes decline correctly, and the DDoS probe is the one that doesn’t. Forcing that number to look better by deleting the probe would hide a real property of the system. The metric is more useful with the known failure inside it.

The fix is not local. This isn’t a threshold to nudge or a prompt line to add. It’s a limitation of matching on embedding similarity and symptom vocabulary. Addressing it properly means giving the system more to discriminate on — richer context at retrieval time, or a verification step that checks whether the mechanism actually fits before returning a confident answer. That’s real design work, recorded here as the direction rather than attempted as a patch.

What would actually address it

Retrieval: search richer text so the match rests on more than a few shared nouns. More context gives the embedding more to separate genuinely-similar incidents from merely-similarly-worded ones.
Diagnosis: a mechanism-check before a candidate is returned with high confidence — does the cited incident’s actual failure mode match the symptoms, or only their vocabulary? This is a grounding step one level deeper than citation-checking: not “is this incident real and retrieved,” but “does this incident actually explain what was reported.”

Both are deferred. Both are the honest fix. Neither is a tuning change.

The Layer 2 Baseline, and a Grounding Filter That Leaked

Fri, 24 Jul 2026 00:00:00 +0200

The short version, for anyone: Before you can detect whether a system is getting worse, you need a measured snapshot of how good it is now — a baseline. Building that baseline for the diagnostic agent immediately caught a real bug that 30 existing unit tests had missed: a safeguard meant to strip out “made-up” citations was letting them through, as long as at least one real citation rode alongside. One genuine reference could smuggle in any number of invented ones. The lesson: unit tests check the cases you thought of; measuring against real output catches the ones you didn’t.

Full finding below.

Measured on: 15 descriptions through the full agent · Result: first measured Layer 2 baseline; one grounding bug found and fixed.

Why this matters

Layer 2 had 30 unit tests proving the machinery worked, and one integration run judged by eye. It had no numbers. You cannot build a regression gate — which detects regressions by comparing against a baseline — without a baseline to compare to. This is that baseline. Building it also caught a real bug the unit tests had missed.

The bug the baseline caught

The first run reported 2 grounding violations — candidates citing incident IDs that were never retrieved. That number is supposed to be impossible by construction, so the tripwire fired.

Why it slipped through

The grounding filter kept a candidate if any cited ID was real:

cited = {...} # every id the model cited
if cited & valid_ids: # at least one is real?
 grounded.append(d) # keep the whole thing, evidence unchanged

So a candidate citing aws-s3-2017-02-28, gitlab-2017-01-31 passed — aws-s3 is real, the intersection is non-empty — and carried the fabricated gitlab citation through untouched. The filter checked that at least one citation was grounded. It never checked that every citation was. One real id smuggled in any number of invented ones.

The fix

real = cited & valid_ids
if not real:
 continue # nothing grounds this, drop it
d["evidence"] = ", ".join(sorted(real)) # keep only the real citations
grounded.append(d)

A candidate now survives only if it has a real citation, and its evidence is rewritten to contain only real citations. Fabricated ids are stripped, not tolerated. Covered by a new test that feeds one real and one fake id and asserts only the real one remains.

Why the unit tests missed it

The existing grounding test used candidates that were entirely fabricated — those were correctly dropped. The gap was the mixed case: one real citation plus one fake. No test exercised it, so nothing failed. The scoring run on real model output was the first thing to hit it. That’s the point of scoring against real output: unit tests check the cases you thought of, evaluation catches the ones you didn’t.

The baseline (after the fix)

Metric	Value
top-1 accuracy	0.625 (5 of 8 with a primary cause)
any-hit rate	0.615
noise rate	0.444
decline rate	1.000 (2 of 2 no-match entries)
mean candidates	1.80
grounding violations	0
mean iterations	2.20

No targets were set in advance — the first clean run is the baseline, so the gate enforces “do not fall below this,” not an invented number. The one exception is grounding violations, which isn’t a target but an invariant: it must be zero.

Reading it

What works: decline rate is perfect — the reliability behaviour (declining rather than inventing) holds. Mean candidates fell from 5 to 1.8; the list no longer pads. Grounding is clean and now provably so.

What doesn’t: noise rate 0.444 is high, but concentrated, not spread — two specific descriptions (L2-003, L2-007) produce almost all the noise in the whole suite. Every other entry is clean or nearly so. So the problem is two descriptions the agent handles badly, not a diffuse quality issue — a located problem to work on rather than a general sense that quality is mediocre.

What this unblocks

Layer 2 now has a committed baseline across seven metrics and a clean grounding invariant. The regression gate has something to regress against — and two concrete, located problems to work on: the noise concentrated in L2-003 and L2-007, and the XID-wraparound attractor (documented separately).

Why Layer 2 Needed Its Own Threshold

Fri, 24 Jul 2026 00:00:00 +0200

The short version, for anyone: The diagnostic layer breaks an incident into short symptom fragments and searches for each. But it was reusing a “closeness” cutoff that had been tuned for full, richly-worded questions — and short fragments always score as less close, so 25 of 27 symptoms found the right incident and then had it thrown away. Giving Layer 2 its own, looser cutoff fixed most of it. But the honest conclusion isn’t “0.36 is the right number” — it’s that a closeness score alone can’t cleanly separate signal from noise on short fragments, because in the data the two genuinely overlap. Naming that limit is the point.

Full finding below.

Measured on: 15 incident descriptions, 27 frozen symptoms, 5 no-match symptoms.

The problem in one line

Layer 2 was using Layer 1’s threshold. Layer 1’s threshold was tuned on complete questions. Layer 2 sends symptom fragments. Fragments score worse, so everything was thrown away.

Why fragments score worse

A post-mortem chunk describes a whole incident — trigger, failure, cascade, recovery — many concepts in one paragraph. A complete question matches that richness. A fragment matches one small part of it, so the distance is worse even though it describes the same event. The whole description is closer to the document than any of its parts.

Input type	Typical distance
Layer 1 complete questions	0.20 – 0.27
Layer 2 symptom fragments	0.32 – 0.41
The threshold both were using	0.30

The cutoff sits in the gap. Layer 1 clears it every time. Layer 2 never does.

What was actually failing

Verdict	Count	Meaning
PASS	0	found, kept, visible to the agent
THRESHOLD	25	found the right document, then discarded
RANK	0	found but ranked too deep
MISS	2	never found at all

25 of 27 symptoms found the correct document and had it thrown away. Retrieval was working. Filtering was calibrated for the wrong input — the correct document was often ranked first, at distances like 0.32–0.33, just above the 0.30 cutoff.

The sweep, and the decision

Each threshold was tested against the same frozen symptoms. The usable range came out to 0.34–0.36: below it, entries get no evidence; above 0.38 junk starts leaking badly, and at 0.40 every no-match probe leaks — decline behaviour collapses entirely. 0.36 is the best point: 9 of 13 entries get evidence, junk still only 1 of 5. Decision: LAYER2_THRESHOLD = 0.36, while Layer 1 keeps 0.30.

What this does not fix

Stated plainly, because 0.36 is not a solution:

4 of 13 descriptions still get nothing — the agent still fails on those.
4 symptoms return the wrong document — the agent reasons over a wrong incident, which during an outage is worse than returning nothing.
1 junk symptom leaks — a description with nothing matching gets treated as if it matched.

Layer 2 goes from never working to working roughly two-thirds of the time, with some false leads.

The overlap problem

The reason no threshold is clean: the distance bands interleave. A junk symptom (“air conditioning failed”, 0.362) can score worse than a real one, but another junk symptom (“machines shut themselves down”, 0.334) scores better than a real hit (“servers dropping everything”, 0.344). No cutoff separates them, because the separation doesn’t exist in the data. More samples would describe the overlap more precisely; they wouldn’t create a boundary.

So the honest conclusion is not “0.36 is the right number.” It’s: a distance score alone cannot separate signal from noise on short symptom fragments. 0.36 is a round number picked from where the table turns — a working setting that makes the component functional, to be revisited when the suite grows. Confidence in the exact number is low, deliberately so.

What to try next

The threshold fix treats the symptom; the cause is that fragments carry less signal than whole descriptions. The real direction: search the full description alongside each symptom and merge the results — matching rich text against rich chunks, the comparison the embedding model is actually good at. Evidence: the same content as complete descriptions scored 0.870 at threshold 0.30, while split into fragments it scores zero at the same threshold. That would widen the gap between signal and noise rather than moving a line through the middle of it.