What is the difference between an AI proof of concept and a pilot, and which comes first?

Proof of concept is an engineering term. NASA's technology readiness levels grade how mature a technology is on a scale of nine, and a proof-of-concept model is often built at level 3, when active research and design begin. A fully working prototype arrives at level 6, level 7 demonstrates the working model in space, where it will actually be used, and level 9 comes only after it has proven itself on a successful mission.

Pilot is the word organizations use for a limited first run on real work. OMB's memo M-25-21 (April 2025), on how US federal agencies use AI, gives pilot programs for a proposed AI use rules of their own, and the first condition is that the program be of limited scale and duration. Production is the job running as normal, on all of its volume, for as long as it earns its place; NIST's framework calls this stage post-deployment.

The UK government's service manual uses other names for the same ladder. Its alpha phase is where a team tries out different solutions; alpha services should not be open to the public, and teams should expect to throw away the code and many of the ideas. Beta takes the best idea from alpha and starts building it for real, beginning with a private beta for a limited number of people. Then comes live. Read across, a proof of concept sits close to alpha, a pilot to private beta and production to live.

The proof of concept: can it do the work? On copies of documents your team already finished, does the AI produce drafts that match what the team did, and does it mark what it can't settle instead of guessing? The answer is a score on a known set, judged by your reviewer, the person on your team who knows this job best.

The pilot: does it hold up on the live flow? New documents turn up in formats the sample never included. Exceptions arrive at the worst point in the month. People have to fit checking into a full day. A pilot shows how often the approver corrects a draft, whether exceptions reach the right person in time, and whether the job keeps pace with the desk's real calendar.

Production: will it keep working? Inputs change. A customer redesigns its PO, or a vendor moves to a new billing system. GAO's AI accountability framework (2021) states its monitoring principle as ensuring reliability and relevance over time, and asks that the acceptable range of data drift be decided and documented, data drift meaning a change in what comes in compared with what the system was built on. NIST's framework expects post-deployment monitoring plans that take in feedback from users and cover appeal and override, incident response, change management and decommissioning.

Proof of concept. A small circle: the owner who approves it, the reviewer who scores it and the people building the drafts. Nobody else on the desk changes how they work, and no customer, vendor or system sees a thing.

Pilot. The circle widens to everyone whose work it touches: the desk team, the approver who checks each item, whoever owns the system the work lands in, and IT for any access. Someone senior signs off before it starts. US federal agencies work under a version of the same rule: M-25-21's conditions for a pilot include a certification by the agency's chief AI officer that it may go forward, tracked centrally, and, where possible, a way for people who may interact with the AI to opt in or out with enough notice to decide.

Production. A named owner for the process, someone who watches how it performs, and someone with the authority to stop it. NIST's framework asks for mechanisms, with responsibilities assigned and understood, to supersede, disengage or deactivate an AI system whose performance or outcomes don't match its intended use. Write down whose job that is before production starts, not after the first bad week.

A proof of concept runs on copies. Finished documents from the one job, each paired with what the team did with it, sent after an NDA and kept outside your systems; or look-alike documents built to the same layouts when the originals can't leave the building. Nothing writes back to anything. Federal guidance recognizes the pattern: for high-impact uses, M-25-21 requires pre-deployment testing that reflects expected real-world outcomes, and where an agency can't see the model, code or data, it calls for other methods, such as providing evaluation data to the vendor and obtaining results.

A pilot runs on live work, with people checking. New documents as they arrive, for one job, for a fixed period. Each draft goes to a person who approves, corrects or rejects it before anything posts or is sent. This is the first point at which access to a live system is agreed, and only for that job.

Production runs on the job's full volume. How much checking continues is decided from the pilot's record, not assumed. The UK government's AI Playbook gives this its fourth principle, meaningful human control at the right stage, which includes humans validating any high-risk decision the AI influences. For document work, decide which items stay with a person for good, such as credits, new accounts or anything above a limit you set.

Every stage ends in a decision, and the criteria for it are written before the stage starts. NIST's framework puts the decision plainly: a determination is made as to whether the system achieves its intended purposes and stated objectives, and whether its development or deployment should proceed. Stopping is a legitimate answer. The UK service manual says a team at the end of alpha could stop altogether, or go back and repeat the earlier work.

Typical criteria at each gate:

• Proof of concept to pilot. Your reviewer's call against the written success criteria: every field your desk would check either matches or is marked for a person, no draft asserts a value the documents don't support, and the hard cases in the sample were handled or handed over.
• Pilot to production. Every item went through a person before it posted. Corrections settled at a level the approver accepts, on live documents rather than the sample. Exceptions reached the right person on time. The off switch was named and tried. The pilot ran for its planned scale and length, and no longer.
• Ready for production. A monitoring plan exists: who checks the output, how often, and which change in incoming documents triggers a fresh look. GAO's framework makes monitoring frequency proportional to the impact of a wrong output, so a job that posts payments deserves closer watching than one that drafts status replies.

The proof of concept comes first. It asks almost nothing of the desk, puts nothing at risk and, if it fails, fails in private. Going straight to a pilot means the first mistakes land on live work, in front of the people you hoped would trust it.

Don't skip the pilot either. A sample, however well chosen, is a snapshot. The live flow brings formats and timing no sample predicted, and a pilot with a person checking every item is the safe place to meet them. A pilot with no end date is not a pilot: it is production without the checks production needs.

Run the proof of concept again when the job changes, not only when the AI does. A new ERP, a new kind of document or a second desk with habits of its own is a new test.

This is the order our offer follows. The free proof of concept runs on copies of one job's finished documents, and your reviewer decides it. After a yes, that job runs live for about 30 days with a person approving every item before it posts, which is the pilot. Production, with monthly support, comes only after that, and the next job starts again at its own proof of concept.