Read this before your first real run
Limits & privacy
A page that only lists what a method does well is not much use to someone deciding whether to rely on it. This one lists what it cannot do, what it costs, and where it has failed.
Nothing on this page is legal advice. Where it describes a safeguard, it also describes what that safeguard does not cover — which is usually the part you needed to know.
The record
What the method has actually done, counted
Across two hardening runs: 14 audit rounds, 70 independent reviews, 0 outcome changes. Every finding was a grounding or phrasing defect — a misquote, a wrong number, an overclaim. The loop is excellent at protecting the prose and the evidence. On this record it has never once changed a grade.
One run converged in five rounds, and three of its catches were errors inside a previous round's fix. One run ran nine rounds and never converged; it was ended by a diff of what actually ships, not by a clean pass, after the loop began finding defects mostly in its own repairs. A third run, more recently, converged in four rounds and fired the write-back step for the first time — with the caveat that its clean round came from a smaller model on a deliberately narrowed scope, which is the ambiguous case rather than the triumphant one.
The defect the whole apparatus missed
After all nine rounds of the run that never converged — forty-five separate reviews — one human reading the finished material cold found a defect every one of them had passed. It was a claim about the world: “this will serve you well.” Nothing in any source can falsify a claim about the future, so source-checking waves it through every time.
This is the structural version of the problem, and it does not go away with more reviewers: a harness converges on the failure modes it was built to catch, and grows blind to the rest in proportion to how well it works. The better your loop gets, the more you need someone outside it. More subagents is not the fix. A person is.
Size your expectations accordingly. If you want the outcomes protected, that protection is your judgment plus the evidence discipline — not the round count.
The limitation to read twice
One student appearing in another student's feedback
Naming one student inside another's feedback is a never-event here: not a finding to be priced, but a condition that stops the run. The harness checks for it mechanically every round rather than only at delivery. But there are four ways one student can surface in another's letter, and only three of them leave something a machine can match.
unit-a. A label is a token, and a token can be searched for.--verify-quotes, which also catches quotes that
are sourced nowhere at all.Stated plainly, because it matters
The method forbids naming or characterizing another student in a student's feedback, and checks for it mechanically wherever a check is possible. But a comparative remark can disclose without naming anyone: “yours was the only submission that used X” tells the reader something about everyone else's work. We attempt to avoid these. That is not guaranteed.
This matters beyond good taste. Guidance to faculty commonly states that you may not discuss, reveal, or compare one student's performance with another's, and indirect identifiers can distinguish a person through linkage even when no direct identifier is present. A remark that discloses nothing by name can still disclose by elimination, and small classes make that easy — in a seminar of six, “the only one who…” identifies somebody.
We check what can be checked mechanically. The last read is yours.
Privacy
What coding submissions buys, and what it does not
code-units.py replaces student names on submission folders with random,
per-run codes, copies everything byte-for-byte with hash verification, and keeps the map outside
the workspace. It is worth doing. It is not anonymization, and the difference is not a
technicality.
unit-a/, unit-b/
— random codes,
reshuffled every run, so a code is never a profile you can follow across
assignments.
The headline limit
It codes the container, not the content. If a submission says “By Jane Doe” inside, it still does. On one real cohort, all 5 students' submissions carried identity findings inside their file contents — and 5,628 of the 5,629 individual findings sat in agent session transcripts rather than in the deliverables. The pass reports where identity remains, and does not rewrite it.
That is a decision rather than a missing feature, and the reason is not the obvious one. The substitution map already exists and is already correct: each student gets their own code and every other named person gets a separate token, so nobody is ever folded into the subject. (An early version did fold them together, which broke the method's own attribution check — a log reading “Sam suggested antithetic variates” came out as the student suggesting it. That bug is fixed.) The same substitution already runs on filenames. Pointing it at file contents would be a small change.
What stops it is that the evidence discipline depends on the source being unedited. Every quotation is checked verbatim against the file it came from. Rewrite the file and the auditor compares a rewritten quote against a rewritten source: the check still passes, and it can no longer catch a misquote of what the student actually wrote. False positives make that worse rather than better, because a surname that collides with a variable name is skimmable noise in a report and silent destruction in a rewrite.
It would also buy no compliance. You hold the roster either way, so the data stays personal under Recital 26. And the identity that matters most is often not a name at all — “my father's restaurant,” “as I said in office hours on Tuesday” — which no substitution reaches, for the same reason the fourth row of Figure 9 has no token to match. So the effective fixes are upstream: anonymous export from your LMS, or an assignment instruction to keep names in the LMS field and out of the files.
Two more honest edges. The deny rule the script prints for your settings file
blocks the Read tool and cat on the key directory; it does not stop
grep, less, or a Python one-liner, and the map sits under the same user
account as the session. It is a guardrail that makes casual access fail loudly, not a
sandbox. And auditor blindness is likewise by construction of context rather than by
enforcement: there is no per-submission filesystem sandbox, and an agent's “files I read”
line is itself a claim.
What is verifiable: every script this plugin ships reads and writes local files and contains no
network code at all. You can check that yourself with a grep for urllib,
requests, socket, and http. What leaves your machine is
whatever your Claude session sends under your own plan's terms — the plugin adds no
destination of its own.
By design
Eight things it cannot do
01
Claims about the world
Predictions and reception claims — “this will serve you well” — have no source that can falsify them. The harness cannot tick that box, and says so.
02
The criteria themselves
It defends every judgment against your rubric; it never questions the rubric. A criterion whose evidence was never collected is decoration.
03
Shared model blind spots
Independent reviewers drawn from the same model share failure modes. More of them will not help. A human outside the loop will.
04
Fairness across the cohort
Pricing, not detection, is where a run goes wrong. A run can be evidence-perfect and still unfair, because fairness is a property of the whole cohort, and no blind auditor can see the cohort.
05
Comparative disclosure
The fourth column of Figure 9. An identifying description with no name and no quote has no token to match.
06
Settling strictness
The same submissions scored 96–100 under one rulebook and 63–84 under another. That axis is yours, and the presets do not decide it.
07
Enforcing an average for free
Forcing an average onto a cohort that really is excellent, or really is weak, misreports them. The residual is always reported, never absorbed.
08
Guaranteeing isolation
Blindness comes from what an auditor's prompt contains. There is no filesystem sandbox behind it.
Two more that are less structural and worth knowing. Criteria that seem obvious can be read incompatibly: independent reviewers handed identical evidence once scored a single criterion across a twelve-point spread on a twenty-point scale — not disagreement about quality, but disagreement about what was even being read. And a self-test that passes proves less than it looks: three separate defects in one session sat behind green suites, because the tests exercised what the code did rather than what the documentation promised.
Fit
When not to use this
- As an outcome oracle. The harness proposes; you dispose. If you want a system to decide grades unsupervised, this is not it, on purpose.
- Before your institution has approved it — for real student work, full
stop. Until then: the synthetic demo, a de-identified
check-mine, or an anonymized pilot. - For work with no ground truth to check against. The whole discipline rests on being able to open a source and see whether a claim holds. Where nothing can be checked, this method has no advantage over careful reading.
- When one careful pass would do. A round over N submissions costs N auditor subagents plus a lead pass, three to five times over. For a handful of short submissions that is not a good trade.
Responsibility
Whose call this is
This project — the methodology, templates, plugin, and documentation — is provided “as is,” without warranty of any kind, express or implied. It is not legal advice, and nothing in it establishes compliance with FERPA, GDPR, or any other law, regulation, or institutional policy.
You are solely responsible for how you use it: for confirming that your use of any AI service on student work or other personal data complies with applicable law and with your institution's policies and agreements, and for the evaluative decisions you ship. To the maximum extent permitted by law, the authors accept no liability for inappropriate, unauthorized, or non-compliant use of this project or of any AI service used with it.
The CC BY 4.0 licence carries the governing warranty and liability terms; this restates them in plain words for this project's context.
We try hard on privacy and we cannot guarantee it. Approved-tool status at your institution is not the whole compliance story either. If you are unsure, the two questions worth asking your registrar, privacy office, or counsel — in writing — are: is this AI service approved for student educational records at our institution, and under which agreement? and what may I put into it: named records, de-identified records, or none?
The setup interview raises all of this whenever the material is student work, records your answer, and never blocks you. It asks because the cost of guessing wrong falls on your students and your institution. It defers to you because your institution's arrangements are yours to know, not this plugin's to police.