Will the AI make up numbers on our orders or invoices?

NIST's profile for generative AI (AI 600-1, July 2024) counts among confabulations any output that diverges from the prompt or other input. On a document desk the input is the page, so an invented value is one that neither the page nor your records hold. That differs from a misread, a 3 on the page read as an 8, and from a bad lookup, the right page matched to the wrong customer. Count the three apart, because each has its own cause.

Invented values rarely look odd. They take the shape of the field they fill:

• A part number in your format. The right length and prefix, a believable revision letter, and on neither the PO nor your item list.
• A price borrowed from elsewhere. From the customer's last order or a similar line, set on a line the PO left unpriced.
• A quantity of one. On a line where the page gave none, because a line seems to need one.
• A stand-in date. The invoice date as the ship date, or today's date where the order named none.
• A reference that was never printed. A PO number on an invoice that carried none, shaped like your numbering.
• A line that balances the total. A freight or tax amount, or a nudged unit price, that makes the lines reach the printed total.

The same NIST profile calls confabulation a natural result of how generative models are designed: their output approximates the statistical distribution of what they were trained on, predicting the next word, or here the next value. Usually the likely value and the true one are the same. When the page says nothing, the likely value is all the model has.

OWASP's list of the main risks in applications built on language models files this under misinformation (LLM09:2025): false or misleading output that appears credible. On a desk, a wrong value passes a glance because it looks like every other value in the column.

So invented values cluster around gaps: a blank price, a missing date, an item described in words, a total that disagrees with its lines. A draft built to fill every field of your layout is pushed hardest there, and a test should push there too, with documents your team remembers as incomplete.

Checking a draft is slow when every value must be hunted down, and quick when each carries its origin. The W3C's provenance standard, PROV (2013), describes provenance as information about the things, activities and people involved in producing a piece of data, used to judge its quality, reliability or trustworthiness. Its data model calls one thing built from another a derivation, and a draft field is just that: a value built from a page.

Drug makers are already held to this. The FDA's guidance on data integrity in drug manufacturing (December 2018) asks that data be attributable, legible, contemporaneously recorded, original or a true copy, and accurate, shortened to ALCOA. It explains metadata with a small example: the number 23 means nothing without the unit mg beside it. A price on a draft means little more without the line it came from.

So every field needs a short note, checked as carefully as the value. The NIST profile warns that output can include confabulated citations offered to justify an answer, and one of its suggested actions, MS-2.5-003, is to verify the sources and citations in output during testing and ongoing monitoring. A note pointing to a line that says something else is a finding in its own right. The notes come in four kinds:

• From the page: page 2, line 4, with the words printed there.
• From your records: the item list or price list row it matched, and what it matched on.
• Worked out: the inputs and the step, such as invoice date plus net 30 terms, or quantity times unit price.
• Not found: an empty field and the reason, never a value.

Old documents give every field two things to hold a draft against: the page, and what your team entered from it. Your reviewer, the person on your team who knows this work best, sets the three side by side for each field, and the combination tells you what happened:

• Draft, page and entry agree. Nothing to do.
• The draft matches neither the page nor the entry. Check the page: if the value is on it but was read wrong, it is a misread; if it is nowhere, it was invented.
• The draft matches the entry, but the page doesn't show it. Your team knew something the page didn't say. If the draft's note names a record you sent and the record holds the value, fine. If it names nothing, count the field as invented even though it came out right: a guess that lands once misses on the next document.
• The draft matches the page, not the entry. Your team changed it on purpose, say for a substituted item. That is a rule to write down, not an invention.
• Page and entry are both blank, and the draft isn't. There is nothing it could have come from, so it was invented.

Then redo the arithmetic on every draft. New York's State Comptroller tells local officials who audit claims that the work demands more than a rubber stamp, that quantity, unit price and totals belong on the claim, and that claims paid should be mathematically correct. A draft whose lines reach its total only because one line moved has shown you where to look.

Put the rule in the success criteria before anything runs: a value with no source on the page, in your records or in a stated calculation counts as a miss, right or not. The criteria judge the habit, not the luck.

A test of 20 to 30 documents shows whether drafts invent values on your paperwork, and where. It can't promise a live run never will, and no tool can. So the control sits after the draft: a person approves each item before it posts anywhere. Public money works this way. Under 31 U.S.C. 3528, an official who certifies a federal voucher answers for the information in the certificate, the voucher and the supporting records, for the computation, and for repaying a payment made wrong by an inaccurate certificate. No such statute binds your approver, but the idea carries over: approval is where one named person stands behind every value.

To stop an invented value, the approver needs the draft, each field's note and the page on one screen. OWASP lists cross-verification and human oversight among its defenses, and names the failure they guard against, overreliance: trusting generated content without checking it. An approver shown only a tidy draft is set up to overrely; one shown a value with no source has a reason to stop. Record every change the approver makes. The FDA guidance defines an audit trail as a secure, time-stamped record from which someone can reconstruct how an electronic record was created, changed or deleted; a month of approvals with one shows which fields still draw invented values.

How a draft treats a missing value decides most of this. Our free AI proof of concept shows it on your own paperwork: copies of 20 to 30 documents your team already finished, sent after an NDA, with nothing touching a live system. The owner names your reviewer, the success criteria are agreed first, including how invented values count, and your reviewer's call decides it. If your reviewer says yes, the same job runs on new documents for about 30 days, and someone on your team approves each item before it posts.

Whoever you test with, us included, ask to see these answers on your documents, not hear them described:

• What appears when the page has no value? An empty field with a reason. A value, however likely, is a guess.
• Does it tell the reasons apart? Not on the page, unreadable, two values in conflict and taken from your records are four situations for whoever approves.
• Which fields have defaults? Today's date, a quantity of one, zero tax and the usual ship-to are common. Each should be listed, and marked wherever it was used.
• What does your system do with a blank? Find out whether an import would read an empty number as zero or fill in an empty date, and try it before any live run.
• What if your system requires a field the document lacks? The item should stop for a person, not get a value to pass the check.
• Will it ever change a printed value to make a total agree? It should show the disagreement instead.