A Claude Code plugin, and the method it came from
Grade it like an audit
A grading tool whose purpose is feedback. It drafts evaluations and proposes grades — but never a number it simply thought of. Every point deducted names an issue and cites a source you can open, and you decide every one of them.
It was built on a real teaching workload and is published with its failures attached. The headline result is the one most tools would bury: across fourteen audit rounds and seventy independent reviews, no finding has ever changed a grade. Every catch improved the evidence or the wording. That is not a disappointing result. It is the design.
GitHub · henkaku-center/grade-it-like-an-audit →Start here if you are skeptical
The numbers, including the ones that do not flatter us
Most of what is written about AI grading is written by people selling it. Here is the whole measured record of this method, good and bad, from real runs on real work.
The failures, stated first
One run never converged. It went nine rounds and was stopped by a diff rather than by a clean pass, because the loop had begun finding defects mostly in its own repairs — it had stopped measuring the work's error rate and started measuring its own edit rate.
After all nine of those rounds — forty-five separate reviews — one human, reading the finished material cold, found a defect every one of them had passed. It was a claim about the world (“this will serve you well”), and no source can contradict a claim about the future. That is the permanent hole in the method, and it is why the last reader has to be a person.
A more recent run did converge, in four rounds, and fired the write-back step for the first time. Even there the clean round came from a smaller model on a deliberately narrowed scope, which is the ambiguous case rather than the triumphant one. All of it is written up in Limits & privacy, at the same length as the successes.
The reason to publish this way is not modesty. It is that a grading tool asks you to stake your students' feedback and your own professional judgment on it, and you cannot evaluate that bet against marketing copy. You can evaluate it against a failure log.
The obvious question
Does it decide grades? No. Does it propose them? Yes — like this.
It would be easier to claim this tool never touches grades. That claim would be false, and you would catch it in about thirty seconds of reading the source. What is true is more useful: a proposed grade here is arithmetic, not judgment. It is what is left after each deduction has named a specific issue and pointed at the file that shows it. Nothing is subtracted for a general impression, because a general impression has nothing to cite.
report.mdF = 594.8, p < 0.001 match the source.
Source: report.mdreport.mdreport.md, Results sectionNow the part that is actually novel. That draft above was itself audited, and the audit found two defects in it — neither of which touched the 19/20:
A misquote inside quotation marks, and a compliment that overreaches by one counterexample. Both would have gone to the student. Neither is a grading error, and that is the point: the machinery protects what the student reads, not what the student scores. Praise is audited exactly as strictly as criticism, because an unearned compliment is misinformation the student may act on.
Before it touches anything real
You are not asked to trust it up front
There are three rungs before a real submission is involved, in increasing order of exposure. Stopping at any of them is a legitimate outcome, including stopping at the first.
demo Grades three fictional students whose draft evaluations contain seven planted defects, then unseals the answer key and scores itself. One of the seven is designed so that no source can settle it — the harness flags it as unverifiable, and you learn firsthand where the method stops. Data exposure: none · fully synthetic · 12–18 minutes, measured twice
check-mine Inverts the whole arrangement. The AI grades nothing. You wrote the evaluations; it fact-checks them — quotations against the source, numbers against the data, arithmetic, attribution, and whether your praise survives a counterexample. Findings are shown to you and nothing is applied. Data exposure: your own writing · de-identify it first if you like
calibrate You grade three to five submissions first. The harness grades the same ones blind. A plain Python script — not a model — reports quadratic-weighted κ, Spearman ρ, and mean absolute difference, so you see how it compares to your judgment before it touches anything that counts. Data exposure: a small pilot · you measure it before it measures anything of yours
setup → run Real work, once your institution's position is clear and you have decided the tool has earned it. Data exposure: real · read Limits & privacy first
Where this stands
A support device, and the reasons that is not just a disclaimer
The most common objection to AI in grading is not that it scores badly. It is that careful, personal feedback gets replaced by fluent boilerplate, and the relationship between a teacher and a student quietly degrades. That objection is correct about most of the category, and this project is built as an answer to it rather than a denial of it.
Design commitment
You decide every outcome
Findings are proposals. Each one is shown to you with its evidence and applied only on your approval. The harness proposes; the human disposes.
Design commitment
Everything is a file you can read
Rubric, findings, deduction matrix, round metrics, precedents — all plain markdown on your machine. You can audit the auditor, and a session that dies mid-round loses nothing.
Design commitment
It documents its own reasoning
Guidance to faculty is consistent that you remain accountable for a grade and must be able to explain its basis. A run leaves exactly that record, as a by-product rather than an afterthought.
There is a version of this tool that would be more popular and much worse: one that reads a stack of submissions and returns a column of numbers. That tool would inherit every documented failure of single-pass LLM scoring — the same essay scored differently on different runs, leniency that tracks the writer's first language, agreement between models that is only moderate — and it would hand you no way to notice. This one is slower on purpose, and it hands you the evidence instead of the verdict.
On telling your students
Institutional AI guidance keeps arriving at the same two points: people being evaluated have an interest in knowing an automated system was involved, and the instructor remains accountable for explaining the basis of a decision. Setup now asks whether you intend to disclose, records your answer, and offers suggested wording you are meant to edit rather than paste. It never blocks you, and it does not lecture you — disclosure norms vary by institution, and some set them centrally.
What makes disclosure comfortable here is having something concrete to disclose: a human set the criteria, approved every judgment, and decided every grade, and an automated pass checked the quotations, figures, and arithmetic. On this workflow all of that is simply true.
Getting started
Install, then run the demo before anything else
New to Claude Code? It runs in the browser at
claude.ai/code or in a terminal. A skill is an
instruction set you invoke by typing /its-name; a CLAUDE.md is a plain
markdown file of standing instructions — in this method, your rulebook, which you own and
edit. That is nearly all the jargon there is.
After the demo, /grade-audit on its own always tells you where you are and what
is next. There are seven entry points in total, and
How it works walks through each one with real output.
Before real student work
Know where your institution stands before student work enters any AI tool — this one included. Student submissions and grades are typically education records under FERPA, or your local equivalent. Many institutions already hold arrangements that cover AI services and yours may be among them, but that coverage comes from your institution's agreements, not from this plugin or any provider's marketing. If you are unsure, ask your registrar, privacy office, or counsel first, and use the no-student-data rungs meanwhile.
We do our best on privacy and we cannot guarantee it. Responsibility for the law and policy of your region rests with you. The full position, including what the safeguards do not cover, is on Limits & privacy — please read it before your first real run rather than after.