What happens when the AI test says no?

A no is rarely one moment. It is a sheet: each of the 20 to 30 old documents, the draft made from it, what your team actually did with it, and your reviewer's mark on every field that matters. The call is no when the drafts fall short of the success criteria agreed before the test began. Read the sheet slowly before deciding what the no means; it is the most useful thing the test produced.

The comparison is what makes the call fair. NIST's AI Risk Management Framework (January 2023) observes that managing AI meant to augment or replace human work needs some kind of baseline to compare against. In a document job the baseline is already on file: the order your desk entered, the invoice your team posted, the certificate it issued. A no says the drafts didn't reach that baseline on documents where the right answer was known.

Sort the misses by kind before you count them. Every miss on the sheet belongs in one of four piles:

• Misread. The value was on the page and the draft got it wrong: a quantity, a date, a line skipped.
• Missing knowledge. The right answer depended on something the documents never said: a customer's standing instruction, a phone call, a rule the desk keeps in its head.
• Wrong answer key. The draft disagreed with your team, and on a second look your team's version was the one in error, or two people on the desk would have handled it differently.
• Guessed instead of flagged. The draft filled in a value it couldn't support rather than marking it for a person. This is the pile that matters most on a document job: a flagged gap costs your team a quick check, while a confident wrong value can post.

The piles tell you which kind of no you have, and each kind leads somewhere different.

A no about the job. Most misses sit in the missing-knowledge pile, spread across the whole sample. The deciding fact lives off the page, so better drafting can't reach it, and testing the same job again mostly repeats the result. The answer is a different job, usually the one beside it whose answer is written down.

A no about the documents chosen. The misses cluster: on one customer's layout, one batch of poor scans, one odd month, or documents whose finished version was missing or wrong. The sample didn't describe the work. If the cluster is typical of a normal month, it belongs in any second test; if not, it shouldn't have carried the result.

A no about the success criteria. The drafts were right in substance but failed a rule that didn't measure what the desk cares about: a free-text description scored for an exact word match when your team words it differently every time, or a pass mark your own team's work would miss if a second person on the desk scored it. NIST's framework asks that the effectiveness of the measures themselves be evaluated and documented (MEASURE 2.13), and its Playbook suggests checking metrics for usefulness and preferring descriptive approaches to overly complex ones. A criterion your own desk would fail measures opinion, not the job.

After a no it is tempting to decide the criteria were wrong. Sometimes they were. Here is the check: can you say what was wrong with them without looking at which drafts failed? If the reason only appears once the results are in, that is moving the line, and a yes earned by moving it tells you nothing about the next month's documents.

Public guidance fixes the criteria first for exactly this reason. The EU's AI Act requires the systems it classes as high-risk to be tested before they reach the market or go into service, against prior defined metrics and probabilistic thresholds suited to their purpose. GAO's AI accountability framework (2021) asks for performance metrics that are precise, consistent and reproducible, and lists the questions to put to them: what justifies the metrics chosen, and who developed them. The NIST Playbook suggests setting acceptable limits, such as how errors are distributed, with what to do when the system falls outside them, and recording why each metric was chosen, including those set aside.

A drafting test on old documents isn't what those rules govern, and none of this is legal advice. The habit still carries over: write the criteria down, record who set them and why, and change them only in writing, before the next test.

A test that ends in a no still leaves three things on your side of the table, and none of them depends on the people who built the drafts.

• Your reviewer's notes. Every mark on the sheet says what the right answer was and why the draft missed it. The NIST Playbook asks that both positive and negative feedback from the people using a system be assessed; your reviewer's reasons for rejecting drafts are the negative half, written by the person who knows the work best. Keep them with the sheet.
• The list of hard cases. The documents that broke the drafts: one order spread across three pages, an invoice billed against two POs, a form that changed mid-year. GAO's framework describes edge case testing as testing with input values at the extreme ends of what is expected. This list is your job's edge cases, chosen by your own desk, and any future test, by anyone, should have to get through them.
• A clearer picture of the work. Sorting the misses shows which rules live in one person's head, which customers or vendors need special handling and where your own answer key disagrees with itself. Those are worth writing down whatever happens with AI; otherwise a new hire learns them by making mistakes.

A second test is worth running when the no pointed at something you can fix and name before it starts:

• A different set of documents, when the misses clustered on a sample unlike a normal month. Choose fresh documents your team already finished. Rerunning the same 20 to 30 after the instructions were tuned to them measures how well the drafts learned those files, not the job.
• A different job, when the misses were mostly missing knowledge. Pick the neighbor whose answer is on file, such as the paperwork assembled after a decision rather than the decision.
• Rewritten criteria, when your reviewer can show the old ones would have failed your own team's work, and the new ones are agreed before any new draft exists.

Stop when the misses spread evenly across fields and documents with no cause anyone can name, when the job's answers depend on calls and judgment that never reach the page, or when your reviewer no longer trusts the drafts enough to check them fairly. A second test run in hope, with nothing changed, costs your reviewer another round of scoring to hear the same answer.

Not going ahead is a sound decision in its own right. NIST's framework names the options for responding to a risk as mitigating, transferring, avoiding or accepting it (MANAGE 1.3), and expects the response to be planned and documented. Write the decision down in a few lines: what was tested, what the sheet showed, who decided, and what would have to change before the job is worth another try. If you do test again, change one thing at a time, so a different result points to one cause.

On copies of old documents, a no is a sheet of marks and a decision. No customer saw a draft, nothing reached a ledger, nobody changed how they work, and nothing needs unwinding. The same discovery after a rollout arrives as orders shipped wrong, invoices already paid, approvers who stopped looking closely and a desk that has half forgotten its old routine.

NIST's framework asks organizations to examine and document the potential costs of AI errors, non-monetary ones included, against how much risk they will tolerate (MAP 3.2), and to keep processes for decommissioning and phasing out AI systems safely (GOVERN 1.7). It also notes that measuring risk early in a system's life may give different results from measuring it later. For a desk, that argues for finding the no early: on copies an error costs only your reviewer's time, and stopping needs no decommissioning.

Our offer is built around that. The free AI proof of concept runs on copies of 20 to 30 documents your team already finished, sent after an NDA, with no access to your systems. The owner names your reviewer, the success criteria are agreed before it starts, and your reviewer's call decides it. After a no, nothing connects, nothing posts and nothing is owed, and what the test taught you stays with you.