Does testing AI on our documents mean connecting it to our ERP?

A test needs three kinds of file, and none of them needs a login. The documents as they arrived: the POs, invoices or requests the job reads. What your team did with each one, printed or exported from the ERP: the order as entered, the invoice as coded, the reply as sent. And the reference lists the desk looks things up in, saved as spreadsheets: items and units, customers and ship-tos, vendors, the chart of accounts, price lists.

From those files the test answers the question that decides everything else: given what your team knew on the day, does the draft match what it did? It shows which fields the AI reads correctly, which lookups it gets right against your own lists, where it marks a document for a person instead of guessing, and which kinds of document it handles badly. Your reviewer, the person on your team who knows the job best, judges every draft against the real outcome.

That matters to IT too: a job that fails on copies never needs an account in your ERP, and nobody has to review a connection that will never exist.

An export is a snapshot. It holds the item list as it stood on the day someone pulled it, which is fair for old documents, since the test asks what the desk should have done with what it knew then. New work is judged against today's records, and some things show only when drafts meet the ERP itself:

• Today's data. An item added last week, a price that changed, a customer put on credit hold, a vendor with a new remit-to address.
• Duplicates. Whether this PO or invoice is already in the system, sent twice or sent again with a revision. A test catches a duplicate only when both copies happen to be in the sample.
• The ERP's own rules. Required fields, closed periods, credit limits, tolerances on price or quantity. A draft can read perfectly and still be refused at entry.
• Whether the access fits. Enough for the job and nothing more, proven on a real account in the real system rather than on paper.
• The working day. Monday's pile, month-end, and an approver fitting checks between everything else.
• The records. Whether the logs on both sides capture who did what, which nobody can check before the logs exist.

IT's first question is how much of the ERP the AI side could reach. NIST's catalog of security controls, SP 800-53, answers it with least privilege, control AC-6: allow only the access that users, or processes acting on behalf of users, need for their assigned tasks. An AI service is a process working for your team, so it gets the narrow slice its one job needs.

In an ERP, scope has several edges: which companies or entities, which modules, which records inside them, and which fields. An order job may need customers, ship-tos, items, units and prices, and nothing from payroll, the general ledger or vendor banking. The UK government's AI Playbook (2025) puts the same rule in one line: an AI model should only reach the data its user should be able to reach.

Write the scope as a list, not a description: these entities, these record types, these fields. A list can be checked against the account once it exists.

The connection should never borrow a person's login. NIST's account management control, AC-2, lists service accounts among the kinds an organization defines, beside individual, shared and temporary ones, and asks for each account to have specified roles and privileges, an approval before it is created, monitoring of its use and removal once it is no longer needed. On shared logins its enhancement names the problem in a phrase: the lack of accountability. Questions to settle before the account exists:

• Who owns it? Usually someone in IT, with the manager of the job named beside them.
• What role does it hold? One built for this job, never a copy of the desk lead's role.
• When may it run? An AC-2 enhancement allows usage conditions, such as certain days or hours, so activity at three in the morning stands out.
• Who holds the credentials? IT keeps them, they never travel by email, and they change on a schedule IT sets.
• How is it switched off? Disabling one account should cut the connection at once without touching anyone's own login.
• What does your ERP license say? Check how it treats an account that isn't a person before the account is made.

Reading and writing are separate grants with very different risks. A connection that reads can look up an item or check for a duplicate. One that writes can create records. The OWASP project's 2025 list of risks for applications built on large language models calls the danger excessive agency and traces it to excessive functionality, excessive permissions or excessive autonomy. Its example reads like an ERP: a tool that uses a products table may need only read access to it, not the power to insert, update or delete records. For document work that suggests four rules to ask for:

• Read only what the job looks up: items, customers, vendors, open orders, open invoices.
• Write only into a holding status. A draft lands unposted, on hold or awaiting approval, whatever your ERP calls it, so a person's approval is what posts it. OWASP's guidance says the same: a human approves high-impact actions before they are taken.
• Leave master data alone. No new vendors or customers, and no change to a price, a payment term or a bank account. Those stay with your people, under the checks you already run.
• Never delete and never post.

The same OWASP list puts prompt injection first, and its indirect form arrives in outside files: a document can carry instructions aimed at the AI, and they need not be visible to a person, only readable by the model. A vendor invoice or a customer PO is an outside file. With reading limited to the job, writing limited to a holding status and every draft waiting for a person, a planted instruction has very little to work with. Keep the grants narrow even when the vendor is trusted.

A connected run leaves records in two places. The ERP keeps its own: which account created a record, who changed it and who posted it. The AI side keeps the rest: which document came in, what was drafted, what the person changed and what they approved. NIST's control AU-3 lists what each record should establish: the type of event, when and where it happened, its source, its outcome, and who or what was involved. The 2024 joint guidance on deploying AI systems securely, from the NSA, CISA, the FBI and partner agencies in four countries, asks for logs covering inputs, outputs, intermediate states and errors. The UK playbook adds an AI-specific reason: log every prompt sent to a model and audit them over time to find out whether prompt injection is happening.

Logs hold data too. AU-3's own discussion warns that an audit trail can reveal personal information, especially when it records inputs. A log of the documents an AI read is a copy of those documents, so ask any vendor, us included, where its logs are kept, who can read them and when they are deleted, and get the answer in writing.

The paperwork is IT's to own. NIST's information exchange control, CA-3, has an organization approve each exchange between systems under an agreement, anything from an interconnection security agreement to a nondisclosure agreement, and record the interface, the security and privacy requirements and each side's duties. An NDA signed for a test covers copies of documents; a live connection is a different exchange. The deploying-AI guidance, written for organizations running AI that someone else built, says that when a team outside IT deploys one, it should work with IT to confirm the setup meets the organization's IT standards, and it suggests asking the developer for a threat model.

Our order of events leaves IT out of the test entirely. The proof of concept runs on copies and exports sent after an NDA, and nothing touches a live system. Nothing connects unless you say yes to the live run; then the same job runs live in your systems for about 30 days, and a person on your team approves each item before it posts anywhere. The questions on this page are the ones to settle with your IT before that month begins.