DETERMINISTIC VERIFIER · CHECKPOINT 6c887a1 · VERIFIED 2026-07-12

Baseline against working copy.
The diff is the verdict.

Edit the controlled document below and the lab re-compares it against the pinned baseline deterministically — no model call, no judged score. Left is what checkpoint 6c887a1 indexed; right is your working copy; the panel underneath is the arithmetic between them.

11,309S1 documents · verified
130questions · verified
68passing tests · verified
@@ s1/confluence/runbook-auth.md · demo edit, never enters the registry @@chunker + embedder pinned at checkpoint · comparison exact, replayable

Baseline — as indexed @ 6c887a1

14Access tokens are issued by the gateway and
15validated per request.
16−Session tokens expire after 24 hours
17and are rotated on privilege change.
18Incident owners page the auth on-call before
19rotating the signing key.

1 of 11,309 documents · sealed under a fixed manifest hash

Working copy — your edit

14Access tokens are issued by the gateway and
15validated per request.
16+Session tokens expire after 30 days
17and are rotated on privilege change.
18Incident owners page the auth on-call before
19rotating the signing key.

re-indexed on edit · manifest hash recomputed, lineage recorded

Deterministic comparison — 12 checksdemo · deterministic
chunk_count±0chunk boundaries unchanged
anchor_targets±0every baseline anchor term survives
dedup_signaturecollisionoverlaps a labeled sibling chunk
retrievability_probe-1probe term dropped: hours
token_budget±0estimated chunk budget unchanged
7 further checks±0heading_integrity, manifest_hash_lineage, …

2/12 checks degraded — REGRESSION.

Verdict vocabulary: improvement / regression / tradeoff / tie. Every delta above is the runtime-deterministic output of this demo comparison — arithmetic on the edit above, never a claim about the project.

RESULTS NEVER PRODUCED

C3 closed without metrics: the evaluation timebox expired at dependency preflight, so no retrieval table, answer-quality score, judged result, or fallback metric was ever produced — and none appears here. Dataset integrity is what this page can prove, so that is all it shows.

CLAIM-REGISTRY.JSON

Verified stays verified.
Blocked stays blocked.

Every claim on this page traces to one line of the registry below: a status, a source file, and a stated boundary on what that evidence does and does not cover.

ClaimStatusSourceBoundary
c2.documents
11,309 S1 documents		verifieddata/MANIFEST.json at local evidence checkpoint 6c887a1Dataset and adapter integrity only; not retrieval or answer quality.
c2.questions
130 S1-answerable questions		verifieddata/MANIFEST.json at local evidence checkpoint 6c887a1Question adaptation and integrity only; no completed C3 evaluation.
c2.tests
68 passing tests		verifiedLocal CI-equivalent verification at checkpoint 6c887a1 on 2026-07-12Model-free tests and deterministic wrappers; not answer-quality evidence.
c3.results
C3 closed without metrics		blocked · no results/case-studies/rag-quality-lab/c3-timebox/dependency-preflight.jsonNo retrieval table, answer-quality score, judged result, or fallback metric exists.

Dependency preflight, verbatim

The evaluation runner never started: preflight found the local environment missing the packages a real hybrid-retrieval pipeline needs, and installing them fell outside this task's offline timebox. No fallback pipeline was substituted.

Missing packages:chromadblangchain_chromalangchain_huggingfacelangchain_ollamasentence_transformerstorchtransformers

Already available:numpypandasrank_bm25

Scopefull EnterpriseRAG-Bench S1 retrieval A/B, 130 questions
Statusblocked

A public repository (baseline commit 0fc1433, GitHub) predates and does not yet contain the local checkpoint 6c887a1 above — a distinct claim of its own, not a substitute for it. Earlier drafts of this page carried different corpus and answer-quality figures; this rebuild retires every one of them rather than repeat them here.

SOURCE · RECEIPTS

Every number opens
the same file.

What was verified
C2 verifies the dataset, adapters, manifests, backend contract and model-free tests.
Evidence class
Evaluation-foundation evidence and the recorded C3 dependency preflight.
Boundary
C3 closed without retrieval or answer-quality metrics because its real dependencies were unavailable. There is no fallback comparison; the earlier public baseline remains a separate checkpoint.
Files, hashes and methods
/case-studies/rag-quality-lab/claim-registry.jsonLucisZhang/portfolio-site · acf05ae78859
sha256:95b3380728df1c2de4df7bb69a1c525c10f0e8818f79577ab336c1c5ebc544ba
/case-studies/rag-quality-lab/c3-timebox/dependency-preflight.jsonLucisZhang/rag-quality-lab · 6e3d6a2b040c
sha256:78ab99f3c9d40c1e96f272dad9992acda2035e5af7750c401e46dd70f563418e
/case-studies/rag-quality-lab/c3-timebox/README.mdLucisZhang/rag-quality-lab · 6e3d6a2b040c
sha256:beecb175221d299f3a5513a880244b74d37017a9d7a34d37014c87b2212c9047

claim-registry.json carries every verified and blocked claim on this page; dependency-preflight.json and its README are the record of why C3 closed without metrics. docs/evidence/digits-rag.md pins every number on this page to one of these files.

Published baseline — baseline commit 0fc1433, published ahead of the local evidence checkpoint above.

Architecture

I built the A/B and regression harness first, then everything the scale-up needed: corpus adapters, deterministic manifests, a judge-free retrieval runner, and the verifiers that gate a run.

  1. Compare

    Run naive vector and hybrid-rerank pipelines on the same controlled questions.

  2. Regress

    Compare one pipeline across knowledge-base versions and bucket question-level changes.

  3. Scale

    Adapt 11,309 synthetic enterprise documents and 130 grounded questions.

  4. Manifest

    Bind datasets, adapters, and outputs to deterministic hashes.

  5. Gate

    Fail on data drift, contract breaks, or measured regression.

Results & negatives

On the controlled 12-question set, a document-only update degraded four questions and every reported quality metric. The repository now carries the same versioned-data and verification lifecycle to 11,309 synthetic enterprise documents and 130 answerable questions.

  1. I changed documents only — no code, no configuration — and Pipeline B's faithfulness fell from 0.988 to 0.867 across the 12 controlled questions. Four questions got worse, eight held, none improved. The regression run is the only reason I know that happened.

  2. The hybrid-plus-rerank pipeline wins all five quality metrics and retrieves 4.6× slower at 50K documents: P50 128.44 ms against 28.13 ms, P99 212.43 ms against 39.65 ms. I keep the two tables side by side, because the quality table on its own makes the choice look obvious.

Limitations

  1. The reported quality and latency measurements belong to the controlled saved runs; they do not transfer to the 11,309-document enterprise corpus.

  2. At enterprise scale, the current evidence establishes data integrity and runnable evaluation infrastructure, not a new answer-quality result.

  3. The lab is a single-machine evaluation workbench, not a released package or a production serving system.