The workflow, end to end

How it works

Seven entry points, one audit round in eleven steps, and every option you can turn. Everything below is shown with real output — the artifacts are from recorded test runs kept in the repository, so you can open the originals and check that this page is not flattering itself.

Where you can come in

Seven entry points

Typing /grade-audit with no argument reads your workspace off disk and tells you which of these you want. You do not have to remember them.

Command	What it does	Real data?
`demo`	Grades three fictional students carrying seven planted defects, unseals the answer key, and scores its own catch rate. Twelve to eighteen minutes, measured twice.	No — synthetic
`check-mine`	The reverse audit. You wrote the evaluations; the harness fact-checks your quotes, numbers, arithmetic, attributions, and praise. Nothing is applied. The AI grades nothing at all.	Your writing
`calibrate`	You grade 3–5 submissions, the harness grades the same ones blind, and `agreement.py` reports κ, ρ, and mean absolute difference. Deterministic Python, so the agreement figures are computed rather than asserted by a model.	A small pilot
`setup`	A plain-language interview — seven questions and four follow-ups — that writes your whole workspace: the rulebook, the task file, the output template, the directory tree. No template-filling by hand.	Sets up for real
`run`	One audit round. The eleven steps below.	Yes
`status`	Reads the files and reports where you are: submissions found, their state, the round metrics table, what was written back, and the single next step.	Read-only
`bank <lesson>`	Write-back between rounds. Drafts a dated one-line precedent, decides whether it belongs to this task or to your standing rules, shows you the diff, and appends only on approval.	Read-only until you approve

The round — 1 of 3

One audit round, in eleven steps

01Preflight — confirm the rubric was fixed before grading, the sources of truth exist, and every submission is present.

02Lead normalizes — one pass to make the drafts comparable in shape, so auditors are not flagging formatting differences as substance.

03Fan out — one auditor subagent per submission, each with a fresh context containing only its own. Figure 4.

04Build the deduction matrix — cluster findings into defect families and price them across every submission at once. Figure 5.

05Lead consistency pass — the only reader that sees every submission. Arbitrates the matrix and catches what blindness structurally hides.

5bExpected-average check — if you set a target, report the gap and the honest ways to close it. Applies nothing.

06Two things go to you, together — the findings table and the questions the harness cannot answer. Rulings are requested, never assumed.

07Apply what you approved — then re-audit each touched submission whole, because a fix is new text and new text is where findings come from.

08Rebuild the matrix and diff it — how many cells moved is the number that tells you whether outcomes are still in motion.

09Convergence check — may you stop, and separately, should you keep trying? Two different questions. Figure 7.

10Write-back — every lesson becomes a durable rule. The round is not closed without it.

11Round report — append one row of metrics, so the trajectory across rounds is visible rather than remembered.

Figure 3. Step 10 is the step that makes the rest worth repeating. A caught failure that is not written back will be caught again next term, at full price.

The round — 3 of 3

The deduction matrix

Every finding is clustered into a defect family and priced across every submission at once. (The column labels below are the codes the tool assigns when you grade blind — one per student, reshuffled every run, so nobody's name is attached to the work while the judgment is forming.) One binding rule governs it: one charge per defect family per student — never per instance. Charging every instance instead of one charge per family moved the mean absolute error from 3.6 to 11.4 points on a /90 basis in testing, more than every other cause combined. That one rule is worth more accuracy than everything else in the method.

Defect family	unit‑a	unit‑b	unit‑c	Ruling
Test statistic reported without degrees of freedom	—	n/a	−1	Flagged. unit‑a reports `F = 594.8, p < 0.001`, equally without df, and lost nothing. unit‑b used a permutation test, so it is not comparable.
Included figure never discussed in the text	−1	—	—	Singleton. Consistent.
Quotation marks around a paraphrase	−0	—	—	Defect in the evaluation, not the submission. Fix the letter; charge nothing.

Figure 5. A row reading — | n/a | −1 is self-evidently a question, which is the entire point of building the grid. No blind auditor could have raised it; the lead pass did, and it reached the human as a ruling request rather than a silent decision.

Verbatim, from a recorded run

“Test statistic without degrees of freedom deducted in one unit and waived in another on identical evidence… The rubric is constant across students; a waiver applies to all comparable units or none.”

— docs/test-runs/2026-08-24-demo-fixture/lead-consistency-report.md

The same structure catches something else worth having. When three students make the identical mistake, that is usually not three students being careless — it is a bug in the assignment. In one real run the cross-student view found that three submissions had inherited the same two errors from the course's own scaffold code. None was charged, under the standing rule that you never deduct for following an instruction you gave them. A blind per-student reader would have charged all three or none, and been confidently wrong either way.

Real output

What an auditor actually reports

Below is a genuine auditor report from the synthetic demo fixture, unedited. Note the last two lines: a tally of what was checked and found clean, and a declaration of every file read. A finding you can check is only half of it — the other half is knowing what was looked at.

# Auditor report — student-a — round 1 4 findings (2 blockers, 1 minor, 1 note) 1. [BLOCKER] — “We removed rows with missing values since they were under three percent of the data.” is presented in quotation marks but is a paraphrase — report.md reads “We removed the rows with missing values because they were fewer than three percent of the data.” fix: replace with the verbatim sentence. 2. [BLOCKER] — “Every figure is labeled and captioned.” is a falsified praise universal — Figure 3 has alt text but no caption line, unlike Figures 1 and 2. fix: “Figures 1 and 2 are labeled and captioned; Figure 3 carries a label but no caption.” Verified clean: quotes 0/1 verbatim (the one quoted string failed); numbers 8/8; universals 3 found, 1 falsified, 1 unprovable, 1 verified; attribution 3 checked, credit correct; behavior claims 2/2; world-claims 0; consistency OK (19/20 arithmetic checks); policy OK. FILES READ: agents/unit-auditor.md; demo-workspace CLAUDE.md, assignment.md, rubric.md; student-a draft-evaluation.md, ledger.md, report.md, session-log.md

Source, unedited: docs/test-runs/2026-08-24-demo-fixture/student-a-report.md. Two further recorded runs live alongside it, including the release audit that produced 63 findings and 18 blockers and stopped a release — along with the one finding that was checked and rejected as wrong.

Options — 1 of 4

Strictness, and why it never becomes a way to stop looking

Three presets seed a price schedule. The obvious way to implement a strictness dial is to make the auditors less sensitive at lower settings, and it is wrong: that hides defects rather than pricing them, and it takes the feedback away from the student precisely where you had decided not to charge for it.

lenient

−0

The missing degrees of freedom cost nothing. Only blockers carry a price.

Still written in the letter, in full.

standard · default

−1

Blockers and minors are charged; notes are feedback only.

Still written in the letter, in full.

strict

−2

Everything named costs something, down to notes.

Still written in the letter, in full.

Figure 6. The same finding at all three settings. The reporting threshold never moves. A lower preset demotes a finding from charged to noted; it never removes it. Strictness governs what a defect costs, not whether anyone looked — and an unnamed issue is awarded full credit at every preset, which is the attribution guarantee.

Strictness is a real axis and the presets do not settle it for you. The same harness on the same submissions produced 96–100 under one rulebook and 63–84 under a stripped-back one. That spread is a fact about your criteria, not about the tool, and it is why setup runs a criteria audit before any grading happens.

Options — 2 of 4

Expected grade, and the cost of enforcing one

You may record an expected average. The harness will then report the gap — and apply nothing. Closing it is a policy decision with a real cost, so it is raised as a question rather than absorbed silently.

The honest route Re-price every defect family by one uniform multiplier and re-derive every grade. Attribution survives: each point still names its issue. The residual left by discrete rounding is reported, never hidden.

The exact route Move individual grades until the average lands. Hits the target exactly and breaks the rule that every point removed names an issue — the student can no longer be told what a point was for.

Advisory only Report the gap and change nothing. Often correct: a cohort can genuinely be excellent or genuinely weak, and forcing an average onto one that is misreports them.

Figure 7. The three routes are presented together, with their costs, as a ruling request. The harness never picks one.

A note on small rubrics

On a /6 or /8 scale the arithmetic gets sharp: rounding to whole points can waive most findings, banker's rounding can silently zero them, and a fixed tolerance of one point is seventeen percent of the whole scale. All three were real bugs, found and fixed. The reconciliation script now infers the granularity your schedule is actually written in and rounds ties away from zero.

Options — 3 of 4

When the loop stops — two different questions

A stopping rule tells you when you may stop. It does not tell you when to stop trying. Those need separate rules, because a loop that keeps finding things can be finding them in its own repairs.

Rule 1 — convergence

When you may stop

One pass comes back clean for the entire set at the same time, and the matrix is stable: no unarbitrated flag, no open ruling request, no cell moved this round.

Rule 2 — bounding

When to stop trying

Blockers have been zero for two consecutive rounds and most of this round's findings sit in text the previous round's own fixes introduced. That is a loop measuring its own edit rate.

Neither rule

The human closes the loop

Not the counter. One recorded run ended on the grader's judgement that the remaining changes were minor; round ten was never run, and that was correct.

round 1 6 / 10 / 7

round 2 1 / 10 / 5

round 3 1 / 1 / 7

round 4 clean

blockersminorsnotes

Figure 8. A real four-round trajectory on live work, ending in convergence and the first write-back this method has ever fired. Deduction-matrix cells moved: 0 in every round. The grades were settled after round one; the four rounds of work went into the prose and the evidence. Round 2's findings were largely defects that round 1's own fixes introduced — the pattern Rule 2 exists to catch. Round 4 ran on a smaller model and a narrowed scope, so it is a convergence event rather than a comparable measurement.

Options — 4 of 4

Models, cost, and coded submissions

One round over N submissions spawns N auditor subagents plus one consistency pass. Budget three to five rounds. Each role's model is set independently in your task file, and all three default to the mid-tier model because the cost driver is the auditors, not the reasoning.

Role	Calls per round	Default	Worth raising?
Auditors (one per submission)	N	`sonnet`	Last. This is where the money goes — one per submission, every round.
Lead consistency pass	1	`sonnet`	First. Hardest reasoning in the method, and only one call.
Fresh reader	1	`sonnet`	A weak proxy for the human outside reader, and labelled as one.

Separately, code-units.py replaces names on submission folders with random codes before grading. It copies byte-for-byte with hash verification and never rewrites file contents — an earlier version that did broke the method's own attribution check by folding a collaborator into the subject's code. The primary reason to use it is blind grading, which is a pedagogical argument you can make without a lawyer. Its privacy limits, which are real, are set out on Limits & privacy.

What it cannot do → How it compares → Back to the overview →

Seven entry points

One audit round, in eleven steps

Blind auditors, and the one reader who sees everything

The deduction matrix

What an auditor actually reports

Strictness, and why it never becomes a way to stop looking

Expected grade, and the cost of enforcing one

When the loop stops — two different questions

When you may stop

When to stop trying

The human closes the loop

Models, cost, and coded submissions