Should a distributor buy an order-entry tool, or test on old POs first?

Order-entry tools for distributors are not new. A Distribution Strategy Group report of September 1, 2026 describes the technology being built into the systems where orders already arrive. In it, one order-automation vendor said it processes more than 1.5 billion line items a year for more than 600 customers, and that it reports an accuracy figure across its whole automation platform. Another said its software turns POs and quote requests from emails, PDFs, spreadsheets, scans, voice notes and handwritten documents into structured orders for review, and that it works with more than 500 wholesale and distribution brands in the U.S.

Those are the vendors' own descriptions, and they answer a different question from yours. They say a tool works across a vendor's customer base. You need to know whether one tool works on your customers, your item file and your desk's rules, on terms you can live with if it doesn't. The same report framed the contest as how much work can be automated "accurately and with appropriate human oversight", and only your own orders can show either half of that.

So the real choice is the order of events: sign and find out on live orders, or score the tool first on orders your team already entered and sign knowing. The second costs some of your reviewer's time. The first can tie you to a contract before you know.

An accuracy figure means little without the test behind it. NIST's AI Risk Management Framework (2023) uses the ISO definition: how close results come to the true values, or to the values accepted as being true. It says accuracy measurements should always come with realistic test sets that represent expected use and details of how the test was run, and that results may be broken out by segment of the data.

At an order desk, the values accepted as true are the orders your team entered. A figure measured elsewhere was scored against other distributors' answers, item files and pricing rules. The same framework asks that the limits of how far results carry beyond the conditions a system was built under be written down. A platform-wide figure averages many desks, and yours could sit far from it.

Ask what the number counts, too. Scored field by field, one wrong unit on a ten-line order is a small miss; scored order by order, it is a wrong order. A figure may also leave out documents the tool declined to read. None of that is a trick. It is how measurement works, and why your own scoring sheet should outweigh anyone's headline, ours included.

A demo has its own job, showing what a tool does well, so the orders that hurt have no reason to be in it. The trade report named them: a wrong product, quantity, price or customer account can cause trouble in purchasing, fulfillment and billing.

Run a tool on POs your team already entered and you learn things no published figure can carry:

• Where it breaks, by customer and by format. Sort the scoring sheet by account and by how each PO arrived. If every miss comes from three branch customers who fax, you know what you are buying and what stays manual.
• Whether it knows your catalog. Your cross-references, superseded items and selling units live in your files, not the vendor's. The test shows how much setup it takes before the drafts read them the way your desk does.
• Where it starts before anyone teaches it. Some tools improve from your team's corrections; in the same report, one vendor described a feature that turns customer service rules and employee corrections into repeatable workflows. A sensible design, and it makes day one a fair question: how good are the drafts before any corrections, and whose hours pay for the teaching?
• How long a draft takes to check. A draft that is right but slow to verify still leaves the work on the desk.
• What it does when it isn't sure. Compare the lines it marked for a person with the questions your desk actually asked on those orders.

Federal agencies make this decision too, and the Office of Management and Budget's 2025 guidance on buying AI, memo M-25-22, works as a checklist for a private company. It strongly encourages performance requirements that let agencies assess vendor claims before award. For independent evaluations, agencies use data they define, kept out of the vendor's reach and as close as possible to what the system will see in use, and tests a vendor runs must be detailed enough to check or repeat.

Turned into questions for an order desk:

• On whose orders was your accuracy figure measured, and what does it count: fields, lines or whole orders?
• Will you draft 20 to 30 of our entered POs, picked by our reviewer, before we sign, and show us every draft rather than a summary?
• Can we keep the entered orders for a few of those POs on our side and score those drafts ourselves?
• Do our POs, or our team's corrections, train models your other customers use? The memo has agency contracts permanently bar vendors from using non-public agency data and outputs to further train commercially available AI without consent. Get your answer in writing.
• When the model changes, how will we know, and will you rerun our test set? The memo encourages agencies to require that a new version meet performance standards or be rolled back.
• If we leave, what comes with us? The part-number mappings, rules and corrections built on your orders, in a form another system can read. The memo counts knowledge transfer, data portability, clear licensing and transparent pricing as protections against lock-in.
• How is it priced, by order, page, line or seat, and what does a heavy month cost?
• What does it write into our ERP, and who approves an order before it posts?

A test isn't always worth the wait. Buying first can make sense when:

• What is left to key is narrow. Most volume already arrives by EDI, and the rest comes from a few large accounts sending the same clean PDF every week. A demo on a handful of those files tells you most of it.
• The vendor will run your test anyway. If it drafts your 20 to 30 old POs, picked by your reviewer and scored on your sheet, before you sign, that trial is the test.
• Walking away is cheap. A short term, little setup, and your mappings and rules exported if you leave.
• You have already seen it scored. Field-by-field results on your own orders, or on those of a distributor with your ERP and a similar customer mix, not only a reference call.

Test first when the opposite holds: many small accounts writing POs their own way, contract prices that change by account, more than one ERP after an acquisition, or a multi-year commitment. There a platform-wide figure says least about your result, and finding out on live orders costs most.

We are one more vendor in this decision, so every question above applies to us. What we offer is the test: a free AI proof of concept on one job, run on copies of 20 to 30 POs your team already entered, sent only after an NDA is signed. Nothing logs in to your ERP or your inbox, and no draft reaches a customer or a live system.

The owner names your reviewer, the person who knows the order desk best, and the success criteria go in writing with a price range for the live job before anything runs. Your reviewer scores our drafts against your team's orders and decides. After a yes, the job runs on new POs for about 30 days, with a person approving each order before it posts. If the drafts fall short, you learned it on old orders, and the same set is ready for the next tool on your list.