Judge audit and threats to validity
Benchmark pages that only list wins aren't trustworthy. This page reports how we audited our own grading for bias, what we found, and the validity threats that remain open. The audit exists because our pilot runs used an Anthropic judge grading an Anthropic subject — an arrangement a skeptical reader should not accept on faith, and neither did we.
Method
Every graded answer from the canonical T1 run was exported into a blinded bundle: judges see only the rubric and the answer — never which arm (source-present / no-memory / memory-injected) produced it, never the run index — and item order is shuffled with a fixed seed. The arm key is sealed in a separate file and only rejoined after grading. Independent judges from three model families then re-graded the same bundles:
- Anthropic — the original judge (Haiku 4.5) and a stronger re-grade (Opus 4.8, full 120 items)
- Google — Gemini via Antigravity, headless, schema-enforced JSON output, stratified 24-item sample
- OpenAI — GPT-5.5 via Codex, single batched call, same sample
- Human — arm-blind pass over the same sample by a non-author-model grader
During external grading, memory injection on the grading machine was disabled so recalled context containing the true benchmark values could not leak into any judge's context.
Agreement
Cohen's κ over paired per-rubric-line marks (κ > 0.8 is conventionally "almost perfect" agreement):
| Judge pair | Scope | Raw agreement | Cohen's κ |
|---|---|---|---|
| Haiku (original) vs Opus | 120 items / 375 marks | 96.5% | 0.920 |
| Haiku vs Gemini | 24-item blind sample | 96.0% | 0.905 |
| Haiku vs GPT-5.5 | same sample | 94.7% | 0.871 |
| Opus vs Gemini | same sample | 98.7% | 0.967 |
| Opus vs GPT-5.5 | same sample | 97.3% | 0.934 |
| Gemini vs GPT-5.5 | same sample | 98.7% | 0.967 |
| Human vs Haiku (original) | same sample | 97.3% | 0.936 |
| Human vs Opus | same sample | 100% | 1.000 |
| Human vs Gemini | same sample | 98.7% | 0.967 |
| Human vs GPT-5.5 | same sample | 97.3% | 0.934 |
The two fully independent model judges (Google, OpenAI) agree with each other most strongly — and the human grader, working blind, matched the strongest model judge exactly (75/75 marks identical to Opus). If the published numbers were an artifact of same-family judging, these are the rows where it would show.
The direction of every discrepancy
Every auditing judge graded the memory-injected arm at or above the published fidelity (published 0.942; Opus 1.000, Gemini 0.958, GPT-5.5 0.958, human 1.000 on the sample). The original Haiku judge was the harshest grader in the matrix — meaning the headline number is conservative, not inflated.
Threats we checked
- Same-family judge bias — measured above; not found, and the residual bias runs against us.
- Judge context contamination — memory injection disabled during grading (a leak vector unique to auditing a memory product with itself).
- Arm identifiability under blinding — the no-memory arm is partially self-identifying (its answers visibly don't know). This affects all judges equally and that arm shows the tightest cross-judge agreement (max Δ 0.04).
- Grader instruction ambiguity — our first human pass was discarded: the grader read the checklist as multiple-choice. Instructions were clarified and the pass redone (yielding the rows above). Reported here because silently dropping it would be the real sin.
Threats that remain open
- Self-authored tasks. The deepest bias risk: the same team built the system and wrote the exam. Planned fix: run on established third-party benchmarks (LongMemEval, LoCoMo) with published baselines.
- Baseline strength. Our naive-RAG baseline is our own construction; a tuned baseline and open-source competitor head-to-heads (as promised by the greyed chips on the console) are the credible comparison.
- Single subject-model family. The pilot ran one SUT family; the multi-model plan (≥3 labs) is specified in the protocol and not yet run.
- Small external samples. Cross-family agreement is measured on 24 stratified items (75 marks), not the full set.
Reproduction: the audit harness (judge_audit.py), blinded bundles, sealed keys, and every judge's raw marks live in the eval suite alongside the runs they audit.