A Judge Pays for Itself on One Kind of Sentence

Four note drafts went through seventeen independent judge passes this week, and two of them went live on this site. The judges raised fourteen blocking findings. Twelve were sentences that a dated record did not support: a figure no record measured, a host the session log contradicted, a commit date read as a landing. Two were cadence. None was a sentence the deterministic gates could have caught; the linter was clean on every pass. Each pass cost about 167,000 tokens and six minutes, and five of the ten failures were one-clause fixes that each bought a full pass. This is the first measurement of the control I built after the fail-open essay, and it says the control earns its keep on one kind of sentence.

A draft goes to a judge that did not write it, running in a fresh context with only the rubric, the draft and its provenance table. The judge first fails a known-dirty fixture, so a judge that would pass the fixture is caught before it passes something real. It returns a verdict and an anchored findings table; I fix blocking findings in prose and send the draft back to a new judge, because an edit after a verdict is text the verdict never read. The rule underneath, that the judge is never the author because self-review is self-attestation, was in the skill from the day I wrote it. What three judges taught me in September was narrower: a second check counts only if something that decides the answer differs from the first.

What the judges caught

Fourteen blocking findings across ten failed passes. Five were receipts: a figure, a host or a prior belief the record did not support. Four were sequence: a date or an order the record states differently. Two were cadence, one was a mechanism stated larger than its record, one was an adoption claim without an artifact, and one was attribution. All four below came from drafts that were later parked, and each would have embarrassed me in print.

• A note said a search that takes seconds alone took ten minutes under load. No measurement recorded seconds. The only timed solo runs were one to five minutes, and the contrast the paragraph turned on was invented.
• A first-hand receipt named Windows as the host of four runs. My own session log, in the same vault as the record I cited, recorded three of the four as scheduled cloud runs, not the desktop. The judge opened both.
• A note said a nightly check landed a day after an incident. The record dated its first commit a day after; it gave no merge date at all. I had written a commit date as a landing, in the same batch as a note arguing that merged is not running.
• A note said one routine failed that morning. The outcome record, which I had not cited, showed two runs and two notices.

Each of these is plausible, fluent, and wrong by a detail that only the record at its revision could settle. That is the class the judge exists for. In September, ten judge passes over two drafts came back with none clean, almost every failure a fabricated mechanism, with the deterministic gates green throughout, which is why the rubric now carries a tense tripwire and a causal-direction check. This week's ledger says the tripwire is where the yield is: the judges spent twelve of fourteen blocking findings on the relation between a sentence and a dated artifact, and none on anything the linter already covers.

What they disagreed about

The judges contradicted each other once, and the contradiction is in the rubric. The third judge on one note asked, as an advisory, that a practice be owned in the first person: with “I now write” as the suggested form, because the worth test wants the author's own before and after. The fourth judge on the same note failed, as blocking, the first-person acts in it: the fix branch, the prediction and the draft all belonged to scheduled routines in the fleet. The receipts rule wants the record's actor. Both judges cited the skill correctly. I resolved it as an owner decision recorded in the pull request: the fleet's routines act as my agents, concrete acts are stated at the record's size, and the lesson stays in first person.

Some disagreement is the expected cost of independent judges. Zheng and colleagues, in the paper that named LLM-as-a-judge, report that a strong model agrees with human preferences over 80 percent of the time, which in their words is “the same level of agreement between humans.” My own September baseline ran two independent checkers over the same 43 vault notes: they agreed on 32 verdicts and converged on the single FAIL, the same note, defect and repair. The pattern held this week in a weaker form. The blocking findings were reproducible in kind; the advisories varied with the judge.

The biases the literature names are the reason for the loop's structure, and one of them the loop does not control. Zheng and colleagues list position, verbosity and self-enhancement bias. Wang and colleagues showed that swapping the order of two candidate answers can flip the verdict, with one model beating a stronger one on 66 of 80 queries once the evaluator was gamed. Panickssery, Bowman and Feng found that evaluators score their own outputs above other models' outputs that human annotators rate as equal, and that when they fine-tuned models to recognise their own text, the bias grew in step with the recognition. A judge that never sees two candidates has no order to be biased by, and a judge that is not the author cannot favour its own draft. But every judge this week ran on the author's model, since I did not override the harness default, and self-preference is a property of the model, not the context. The fresh context removes the drafting rationale. Whether the shared model also inflated the pass rate is something this ledger cannot say.

What it cost

Seventeen passes consumed 2.84 million tokens and 104 minutes of judge time, fixture included: a mean of 167,000 tokens and six minutes per pass. The cost is per pass, not per finding. A pass that found three defects and a pass that found one wrong date cost the same, because a blocking fix is re-judged by a judge that did not author it, and a fresh context rereads the rubric, the draft and every record. Five of the ten failures were one-clause fixes. Scoping those re-judges to the changed claim would have saved about a third of the spend, on a projection rather than a measurement. The contract does not provide for it, and I have not changed the contract.

The fixture never discriminated. Seventeen of seventeen judges failed it, each with findings covering the four classes the answer key requires. That is the designed outcome, and it means the fixture has not yet told me anything a real pass did not. I keep it because the judge it would catch, one that passes a draft with invented figures and stacked aphorisms, is the one failure of this loop I could not see any other way, and because a fixture costs a fraction of a pass.

The loop also has a floor it cannot lower. Every first-hand receipt in the two shipped notes rests on a forge revision or on the Windows machine the fleet runs on, and the cloud session could reach neither. Each judge verified those mechanisms against the vault's dated records and marked the artifacts themselves could-not-run, on every pass, correctly. The notes shipped on record-level receipts with that status disclosed in every source field, a precedent two earlier notes had set. It is also the standard the notes argue against: from the cloud, the loop confirmed that each story matched its record, and the artifacts behind the records stayed unopened.

What I would change

1. Price a re-judge by the size of the fix. A one-clause fix should get a scoped pass over the changed claim and its provenance line, by a fresh judge, with the full pass reserved for a rewrite. The contract's reason for a full pass, that the judge is never the author, survives the scoping.
2. Put the actor in the provenance table. Every first-person verb in a draft gets a row naming who the record says did it. The attribution question is then answered before the judge opens the record, and the rubric tension above cannot re-fire.
3. Record owner decisions as standing rules, not per-PR dispositions. The decision that routines act as my agents was taken once this week and will be re-asked on every forge-sourced note until it lives in the skill.
4. Measure judge agreement on a fixed draft, periodically, the way the September baseline did. Successive judges over a changing draft cannot give an agreement rate; two judges over the same text can.

None of the four is landed. The ledger is in my notes, the contract is as it was, and the two notes are live under the old rules. I now know what the loop costs and what it catches, and have not yet decided whether the first is worth the second at this cadence. For two notes a week it is. For the four-draft batch I started with it was not: thirteen passes to bring four drafts to a verdict, before any decision about shipping.

A judge pays for itself on one kind of sentence, the plausible one no record supports. Price every pass against that sentence, and nothing else.