How do I test PO automation on our own old orders before buying a tool?

Searches for PO automation often mean the buying side: raising purchase orders to suppliers. This note covers the other side, the customer POs your order desk turns into sales orders, whether they arrive as PDFs, scans, spreadsheets or lines typed into an email. Orders a customer already sends over EDI come in structured, so they aren't what you are testing.

Old orders make the fairest test you can run, because each one already has a right answer: the order your team entered. A demo runs on the vendor's own examples. NIST's AI Risk Management Framework (2023) asks for performance to be measured and shown under conditions similar to where the system will actually be used. For an order desk, nothing comes closer to those conditions than last month's inbox.

Take them from the last month or two, so the item list and prices still match. Then build the set on purpose instead of taking the first 30 in the folder:

• By customer. Your biggest accounts, since they are most of the volume, and a few small ones that order rarely and write POs their own way.
• By how they arrive. Clean PDFs, scans and faxes, spreadsheets, and orders typed into the body of an email, in roughly the mix your desk sees.
• By trouble. Orders that needed a phone call, a line with no price, an item number the customer invented, a unit your team had to convert, a changed or cancelled line, a rush note.

Your reviewer picks the set: the person on your team who knows the order desk best. Never the vendor. A sample chosen by the party being tested tends to flatter it. Don't tidy anything, either: a PO with a handwritten change in the margin is exactly the kind the test has to see.

Write one line on why each order is in the set. When a draft goes wrong later, that line tells you whether it failed on an easy order or on one you chose because it was hard.

For every PO, export or print the sales order as it was finally entered, header and every line. If the order changed afterward, because the customer sent a revised PO or the desk fixed a mistake, include the revision and use the final, corrected order as the answer. Mark the ones that changed: a draft that copies a mistake your team later fixed earns no credit for it.

Add what your team looks things up in: an export of the item list with its units and any customer part number cross-references, and the price lists or contracts for the accounts in the set, if price is part of the test.

Blank prices deserve a line of their own in the plan. A PO without a price is normal. Under the Uniform Commercial Code, a buyer and seller can form a contract without settling the price, and when nothing is said about it, the price is a reasonable one at the time for delivery. In practice your desk fills it in from the price list or the contract. A draft should do the same or mark the line for a person, and never make a number up.

An order that is right apart from one unit of measure is a wrong order. So score field by field on one sheet: a row per field per order, holding the value your team entered, the value in the draft, and one of three marks, matches, differs or marked for a person.

Header fields: customer account, ship-to, the customer's PO number, order date, requested date, ship method and notes. Line fields: your item number, quantity, unit, price and line date. Agree before you start what counts as a match, for example whether a ship-to written differently but pointing to the same address on file is right.

Keep two kinds of wrong apart. A field the draft got wrong but marked for a person gets caught in review. A field it got wrong and presented as right is the expensive kind, because nobody looks twice. NIST's profile for generative AI (2024) calls this confabulation: confidently stated but false content. On an order desk it looks like a real-looking item number that isn't the one the customer meant.

Dates need care too. The W3C notes that writing the month first is unique to the United States, while most of Europe writes the day first; the unambiguous form is ISO 8601's year-first YYYY-MM-DD. A customer abroad who writes 03/04 may mean the third of April, so score requested dates as dates, and check which reading your team used.

Your reviewer judges: the desk lead, a senior customer service rep, or the inside salesperson whose orders come back right. Not the vendor, and not whoever championed buying the tool. The NIST framework points the same way, asking for assessments by internal experts who didn't build the system, or by independent assessors, with domain experts consulted. On an order desk, the domain expert is already on your payroll. The owner names them before the test starts.

Write the success criteria before any draft exists, and don't change them once results arrive. The same framework asks for test sets, metrics and the tools used to be documented. Here that means the list of orders, the field list and the criteria, kept together and dated. Plain sentences your reviewer can check work best, for example:

• No line goes through with a wrong item number unless the draft marked it for a person.
• Every quantity is converted to your selling unit the way the desk converts it, or marked.
• On contract accounts the price matches the contract, and a PO price that differs is marked, never copied.
• The ship-to is always one already on the account, or the draft says it is new.
• Your reviewer would approve each draft with no more corrections than a capable new hire's order would need.