Should we start with an AI readiness assessment or test AI on one job?

A readiness assessment asks whether the company is set up to use AI well: its data, its systems, its people, its rules and the backing of its leaders. A test on one job asks something narrower. Can AI do this particular piece of work, on our own documents, to the standard our team already holds itself to?

Both are sensible first steps, and neither stands in for the other. A company can score well on readiness and still find its first job a poor fit, because the real rules live in one person's head. Another can score low, with scattered data and no AI policy, and still have a desk job that drafts well from its own past work.

A typical readiness assessment runs in three moves. A questionnaire, often with interviews, covers areas such as data quality, systems, skills, governance and leadership support. The answers become a maturity score for each area, placed on a scale of levels. Then the distance between where you are and where you want to be becomes a roadmap: what to fix, in what order, and which uses of AI to try first.

Public frameworks show the shape well. The UK government's Data Maturity Assessment framework is a self-assessment built from ten topics that cross six themes, with five levels running from Beginning through Emerging, Learning and Developing to Mastering. It deliberately gives no single score; the result is a matrix that shows where an organization is strong and where it is thin. NIST's AI Risk Management Framework describes the same step from score to plan: a Current Profile of how AI is managed today, a Target Profile of the outcomes the organization wants, and action plans for the gaps that comparing the two reveals.

US federal agencies were handed this exercise outright. OMB's April 2025 memo M-25-21 gave each CFO Act agency 180 days to write an AI strategy, and every strategy had to include an assessment of the agency's current AI maturity and a plan to reach its goals. The memo defines AI maturity as an organization's capacity to adopt AI successfully and responsibly across its operations, manage the risks and comply with the rules. That is a question about a whole organization, and an assessment is the right instrument for it.

A test on one job starts from the other end. You choose one piece of recurring work, such as keying customer POs or coding vendor invoices, and send copies of documents your team already finished, after an NDA, together with what the team did with each one. The AI drafts the work again, and your reviewer, the person on your team who knows that job best, sets every draft beside the real outcome.

What comes back is a scoring sheet, not a maturity level: for each document, which fields matched, which differed, and which the draft marked for a person rather than guess. Measured against success criteria written before the first draft existed, it ends in a yes or a no for that one job. No draft is sent or posted, and because the test reads copies, nothing in your systems moves while it runs.

Picking one job forces the specifics a broad review can leave open. Under MAP 1.1, NIST's framework asks that an AI system's intended purpose and the settings it will work in be understood and documented, including the specific users and what they expect. On a one-job test those stop being abstractions: they are the order desk, the people who key its orders and the reviewer who knows what a right order looks like.

The test has hard edges too. It says nothing about the next job, your data governance, your staff's skills or whether you need an AI policy. A pass on vendor invoices doesn't mean customer POs will pass, and a fail on one job doesn't mean AI has no place in the company.

Start with a readiness assessment when the question in front of you is about the whole company rather than one desk. A good one can also tell you that AI isn't the answer for some work. The UK government's AI Playbook makes "You use the right tool for the job" one of its ten principles, and NIST's framework asks that viable non-AI alternatives be taken into account alongside the AI option. In a back office that might be a better intake form, a rule in the ERP or a fixed handoff between two desks.

The usual signs that the assessment belongs first:

• You can't name one job yet. Several departments want AI, nobody agrees where to begin, and the candidates need comparing on the same terms before any of them is tested.
• Someone needs a company-wide plan. A board, a parent company, a lender or a large customer has asked how you will govern AI, and the answer has to cover more than one desk.
• The ground rules don't exist. Nobody has decided who approves a new use of AI, which data may leave the company or how a mistake gets reported. Settling those first makes every later test simpler.
• You are working toward a formal standard. ISO/IEC 42001:2023, the international standard for an AI management system, sets requirements for establishing, running and continually improving one, and it applies to an organization of any size or type. Measuring yourself against it is an organization-wide job by design.

Start with the test when you already know the job: a desk with a steady stream of documents, a backlog somebody watches, a person who can judge the work and a folder of finished examples. The owner's real question then is whether this job can be drafted well enough to hand over, and only that job's documents can answer it.

A test answers some readiness questions in the concrete, too. Can the documents be read at all? Is the finished outcome written down somewhere, or only remembered? Do two people on the desk do the job the same way? The scoring sheet shows each of these on real pages, which is often quicker than describing them in a survey.

NIST's framework doesn't say the assessment must come first, either. The AI Risk Management Framework is intended to be voluntary, and it says its actions are not a checklist and not necessarily an ordered set of steps. Its companion Playbook says the same of its suggestions. The order is yours to choose, and a smaller company whose AI plan really amounts to two or three desk jobs can reasonably begin with one of them.

Assessment, then test. The roadmap names the first jobs worth trying. A test on the top one checks the roadmap's first line against your own documents before anything is bought, and its result shows whether the plan's assumptions about that job held up.

Test, then assessment. A scored result on one job gives a later assessment something solid to stand on: a job that passed or failed, and why. The reviewer's notes, such as which documents couldn't be read or which rules lived only with one person, feed straight into the questions about data and know-how that an assessment asks.

Whichever comes first, three things should carry across: the list of candidate jobs, the person who owns each one, and success criteria written in plain sentences before anything runs. Ask any assessment to finish with named jobs and criteria, and ask any test to finish with a decision and a list of what it didn't cover.

We run the second kind: a free proof of concept on one job, on copies of finished documents, judged by your reviewer. If it passes, the job runs live with a person on your team approving each item before it posts or is sent. If an assessment has already named the job, bring its roadmap to the call.