Which 20 to 30 documents should we pull for the AI test?

The set has to stand in for the job, so begin with everything the job received over a period, not with a folder of good examples. The PCAOB's audit sampling standard, AS 2315, says sample items should be selected so that the sample can be expected to be representative of the population, which means every item in the population should have an opportunity to be selected.

Where you draw from matters as much as how. The same standard notes that an auditor can't find omitted items by sampling the recorded ones; to test payables for unrecorded purchases, it suggests sampling the cash paid out afterward. A document job has the same blind spot. Pull from your ERP or filing system and you only see what was entered or filed, never the PO that sat in someone's inbox for a week or the request answered by phone and never logged. Pull from where documents land: the shared inbox, the scanner folder, the mail tray.

Choose a recent window and draw the set from what came in during it, instead of collecting favorites from across the year. California's tax agency describes this pick in its guide to audit sampling, form CDTFA-472: the most common judgment sample is a block period of time, such as a day, a week, a month or a quarter. A block keeps the mix honest. The customers who send something every day turn up about as often as they really do, and the one who orders rarely mostly doesn't.

Recent matters because the job drifts: item lists, prices, vendor layouts and people change, and an old set scores drafts against rules your team no longer follows. AS 2315 adds a point worth borrowing: in testing controls, an auditor should ideally use a method that could select items from anywhere in the period under audit. Inside your window, take from all of it, the month-end rush as well as the quiet days.

The cleanest examples make a weak set. They show the work at its easiest, and a draft that gets them right tells you little about the rest.

Every desk has documents that are awkward but normal: the customer who sends a spreadsheet instead of a PO, the vendor who bills freight on a separate invoice, the request that quotes last year's policy number. None of them is an emergency. They come every week, and they take most of the desk's thought.

Those belong in the set at about the rate they really arrive, because a live job would meet them at that rate. A block of recent work brings most of them in by itself. If one your team sees every week didn't land in your window, add an example and note that you added it.

The once-a-year disaster is a different matter. A few can join, marked as additions, so your reviewer sees how the drafts behave at the edge. A set built mostly from horror stories tests a job you don't actually have.

A draw from one window can still miss a large customer by chance, so check the set against the way your work divides. Auditors call this stratifying. AS 2315 lets an auditor separate items into relatively homogeneous groups, based on a characteristic tied to the objective, and then select an appropriate number from each group. The NIST/SEMATECH e-Handbook of Statistical Methods gives the reason in its section on sample sizes: stratifying the population reduces the variation within each group.

For a document job, the groups are whatever changes how the work gets done. Usually that is the sender, since each customer or vendor has its own layout and habits; sometimes it is the channel, such as a portal, a fax line or the post. List the senders in your window from most documents to fewest. Make sure each of the biggest appears, take the rest in rough proportion, and keep some from the long tail of senders who write rarely and in their own way.

Write the group beside each document on the list. When your reviewer scores the drafts, the misses can then be read group by group, and a job that works for your largest customers but not the others looks different from one that struggles everywhere.

Each document needs what your team finally did with it: the order as entered, the bill as posted, the certificate as issued, along with any later correction. That record is the answer key your reviewer scores against. A document without one can't be scored, so check for the record while you pull, not after the list is closed.

Decide now what happens if a record turns up missing later. AS 2315 meets the same snag: supporting documentation may be missing for an item already selected, and in a test of controls the auditor should consider why and ordinarily treat that item as a deviation. The honest version for your test is to keep the document on the list and report it as unscorable, never to swap in an easier one. Several missing records are a finding about the job in their own right.

Some documents can't go, whatever the test would gain from them. Leave out anything covered by an NDA you signed with a customer or vendor, anything a customer contract says stays in-house, and anything you would hesitate to send if that customer were watching. A customer's drawing or specification can be its trade secret, and under federal law information keeps that status only if its owner takes reasonable measures to keep it secret. Whether it travels is the owner's decision, not yours. This isn't legal advice: when you are unsure, leave the document out and ask your counsel.

Leaving a document out shouldn't leave a hole. If a whole group can't go, such as every order from your largest customer, we build look-alikes with you: documents that copy that group's layout, field positions and habits, filled with invented values, with the right answer written down for each. NIST's guide to de-identifying government datasets, SP 800-188 (2023), describes test data of this kind, resembling the original in structure and range of values, and suggests conspicuously non-natural names and addresses so test files aren't mistaken for real ones. It also lets test data carry extreme values on purpose, which is where the group's awkward habits go.

Mark every look-alike on the list: your reviewer judges them the same way, but that group's result describes how the drafts handle its layout, not that customer's real traffic.

Choose the documents, write one line on why each is there, and freeze the list before the first draft exists. Then nobody can drop a document because its draft went badly, or slip in an easy one because the score looked thin. California's audit sampling plan works this way: it records the plan and the criteria for judging the results, and it is completed with the taxpayer before the sample is selected. The NIST e-Handbook likewise has a sampling plan say what is measured, when, on which material, how and by whom, and has it verified before anyone carries it out.

Be clear about what 20 to 30 documents can tell you. The Census Bureau's quality standard for sample design doesn't apply to samples that won't be used to produce estimates or make inferences, such as those for operational tests and pilot studies, and a test set is that kind of sample. It shows, document by document, whether the drafts handle your real work, not an error rate you could quote. Inspection standards such as ISO 2859-1, on sampling for inspection by attributes, serve another purpose: pressing a producer, through the risk of rejected lots, to hold its process average to an agreed acceptance quality limit. And as the California guide says of samples picked by judgment, they aren't necessarily less accurate than statistical ones, but there is no objective way to evaluate their accuracy. Read the result as your reviewer's decision about one job.

Our free AI proof of concept runs on the set you choose: copies of 20 to 30 documents your team already finished, sent once the NDA is signed, with nothing touching a live system. The owner names your reviewer, the success criteria are agreed before anything starts, and your reviewer's call decides it.