Who on our team should judge the AI test?

A drafting test asks one question over and over: is this value right for this document? The person who answers it fastest, and is believed, answers it every working day. Look for the person whose corrections the rest of the team accepts without an argument.

NIST's AI Risk Management Framework (January 2023) has a name for the role. Its appendix on who does what describes domain experts as people who know the industry or application area where an AI system is used and who interpret its outputs for the people testing it. It adds that, ideally, those who verify and validate a system are distinct from those who test it. For a document test, we read that as ruling out whoever built the drafts and anyone whose standing depends on a yes.

Two common picks miss. A manager sees escalations and totals, not the small rules the desk applies without thinking, and so tends to reward drafts that look tidy. An IT lead knows the fields but not whether a ship-to is right for this customer; their questions belong to a live run, when access and logging matter. Both belong at the readout. Neither should hold the pen.

Four things, all settled before the first draft exists:

• Blocked time. Scoring field by field goes wrong when it is squeezed between live orders, so plan short sittings with a set number of drafts in each. FDA's guidance on reading images in clinical trials (April 2018) notes that reader fatigue has been reported to lower the quality of interpretation, and asks trial planners to consider capping how many images a reader takes in one session.
• The old documents, and what your team did with them. Each document as it arrived, the record made from it, and anything that changed that record later, such as a credit memo or a reissued certificate. GAO's guide to assessing data reliability (GAO-20-283G, December 2019) traces both ways: from source documents to the data to find gaps, and from the data back to the source to find wrong entries. Your reviewer does both. Every line on the page should appear in the draft, and every value in the draft should trace to the page or to a record you sent.
• A short list of what counts as wrong. One page, agreed with the owner: a wrong value in a field the system uses, a dropped line, a value the draft can't point to, a problem it should have flagged. Then what doesn't count, such as spacing, abbreviations and free text that says the same thing differently. The FDA guidance warns of reader drift away from the criteria readers were trained on, which a list kept beside the sheet guards against.
• A note on what the drafts could see. Article 14 of the EU's AI Act asks that the people overseeing the systems it classes as high-risk be able to understand the system's capacities and limitations, interpret its output correctly, and decide to disregard or override it. A test on old documents isn't covered, but the habit fits. Tell your reviewer what the drafts were given, so a miss on a rule nobody sent counts against the instructions, not the reading.

One reviewer is enough when the answer sits on the page or in the record: a quantity, a PO number, a due date. A second earns their time when the job turns on judgment that two good people exercise differently, such as coding a bill to an expense account; when the work is split across shifts or branches; or when the expert's own work fills most of the sample.

Rules sometimes ask for two people where one miss is costly. For remote biometric identification, Article 14 of the AI Act bars acting on a match unless at least two people with the necessary competence, training and authority have separately verified and confirmed it. That rule is about faces, not invoices, and this isn't legal advice, but the reasoning carries over.

Disagreement is expected, not a failure. The FDA imaging guidance says readers of the same image may justifiably disagree, wants the plan for reconciling them written down in advance, and notes that how often adjudication was needed gives a sense of how subjective the outcome is. Choose one rule before scoring starts:

• The finished record stands unless both reviewers mark it wrong.
• A third person, named by the owner, decides the split fields only, seeing both marks and the source document.
• Splits are reported, not scored. Many of them mean the job carries more judgment than the criteria assumed, which is a finding in itself.

Before the sample is scored, have both reviewers mark the same few drafts alone and compare. Mary McHugh's 2012 article on interrater reliability in Biochemia Medica, available on NIH's PubMed Central, explains why raw agreement misleads: it ignores the chance that raters guessed, so it can overstate how much they agree. Cohen's kappa corrects for chance, and a kappa of zero means agreement no better than chance.

You don't need the statistic for 20 to 30 documents, only the habit behind it. The article expects raters to be trained to read data the same way, and treats agreement worse than chance as a reason to retrain them or redesign the instrument. Your instrument is the list of what counts as wrong, so rewrite any line the two split on. Practise on documents from outside the sample; the FDA guidance likewise keeps pilot reading studies off the images used in the trial itself.

Grading on a curve means going easy because the drafts beat expectations, going hard because a job feels at stake, comparing drafts with each other instead of with the record, or loosening up as the afternoon wears on. NIST's framework notes that people bring their cognitive biases into how AI is evaluated, and take different meanings from the same output. The fix lies in how the sheet is built, not in willpower.

• Score blind. Put the draft and your team's record for each document in one plain layout, labelled A and B in shuffled order by someone who isn't scoring. The reviewer marks both against the source document, and learns which was which only after the marks are in. The FDA guidance expects image readers to be blinded to a patient's treatment, because knowing it would be presumed to create bias. Blind scoring also shows where your own record was wrong, without anyone having to argue the point.
• Judge against the finished work, not taste. Each field is judged against the record, or the document where the record was wrong, not against how the reviewer would have worded it.
• Lock each mark. The FDA guidance encourages readers to lock their reads so none is altered. A mark entered is final, and a second thought goes on a new line with its reason.
• Slip in a few repeats. The same guidance suggests testing a reader's consistency with images mixed back into later sessions, far enough apart to limit recall. Put two or three documents back into a later sitting. A different mark the second time means the list needs another line.

Before scoring, the owner hears who the reviewers are, the list of what counts as wrong and the date of the readout. During scoring, nothing. A running tally invites the owner to lean on the reviewer, and the reviewer to lean on the line.

The readout is one page, read with the reviewer in the room: the call, yes, no or not yet decided, with its reason; where the drafts held up and where they didn't, field by field; the misses a person might have approved without noticing; how each split between reviewers was settled; and where blind scoring found your own record wrong.

GAO's guide gives a useful vocabulary for the call. Data can be sufficiently reliable for a purpose, not sufficiently reliable, or of undetermined reliability, and parts of it can pass while others don't, so a job can pass on quantities and fail on coding. Not yet decided is honest when the sample lacked a kind of document the job depends on; a few more of that kind, scored against the same list, settle it.

Our free AI proof of concept is built to be judged this way. It runs on copies of 20 to 30 documents your team already finished, sent after an NDA, with no access to your systems. The owner names your reviewer before anything starts, the success criteria are agreed first, and your reviewer puts each draft beside what your team actually did. Their call decides it. After a yes, the job runs live for about 30 days, and a person on your team approves each item before it posts anywhere.