Run an AI Readiness Review Before Choosing a Model or Vendor

Run an AI readiness review by selecting one workflow, collecting evidence about how it operates, and checking whether its data, controls, owners and success criteria support a controlled experiment. Do this before comparing models or vendors. The output should be a decision record: proceed to testing, fix named blockers, or use a non-AI approach—not a speculative technology shortlist.

Assemble an evidence pack, not a maturity score

Readiness is specific to the work. A business might be ready to classify incoming requests but unprepared to let software approve account changes.

Bring together the workflow owner, an experienced operator, the system or data owner, and whoever is accountable for security and privacy. Ask them to supply evidence before the review:

• Current performance: volumes, handling time, correction rates and backlog, with measurement periods attached.
• Representative cases: routine work, incomplete inputs, unusual requests and previous mistakes.
• Decision rules: what makes an output acceptable, who resolves ambiguity and which actions require approval.
• System constraints: available interfaces, access permissions, outages and existing manual workarounds.
• Operating ownership: who will review outputs, maintain source material and respond to incidents.

Walk through several real cases from arrival to completion. Record where staff use judgement versus fixed rules. If straightforward rules solve the problem, retain that option as the comparison baseline—the current or simplest alternative against which AI will be measured.

Review four gates independently

Use three statuses for each gate: evidenced, unresolved or blocked. Attach an owner and next action to every gap. Avoid averaging the results: strong potential savings cannot compensate for missing permission to process data.

1. Value and measurability. Can the team identify a useful outcome and measure it? Distinguish reduced handling time from cash savings; released capacity only becomes financial savings under an explicit staffing or workload assumption. Include integration, human review, monitoring and maintenance in the cost estimate.

2. Data availability and authority. Can the workflow access sufficiently current, representative information? Identify the authoritative source for each decision, permitted users, sensitive fields, retention rules and processing-location constraints. A folder of documents is not enough if nobody can confirm which versions apply.

3. Execution boundaries. List what the proposed system may read, recommend and change. An agent is software that uses a model to choose actions or tools. Do not assume one is necessary. Microsoft’s orchestration guidance distinguishes direct model calls from tool-using agents and coordinated multi-agent systems, with increasing operational overhead. Use the review to establish required capabilities, not to preselect a complex architecture.

4. Operational accountability. Name the person who accepts quality, the team handling exceptions and the owner of shutdown and recovery. Confirm that reviewers have capacity and enough evidence to challenge an output. A nominal approval button is not a staffing plan.

Establish security and failure paths before experimentation

Prompt injection means malicious instructions in user input or external content attempt to redirect a language model. Emails, attachments and retrieved documents can carry such instructions. The OWASP prevention guidance recommends layered controls, including separating instructions from data, validating outputs and tool calls, restricting permissions, and requiring oversight for high-risk actions.

Translate that into readiness questions: Can access be enforced outside the model? Can the test environment prevent real-world changes? Can operators inspect what happened without unnecessarily exposing sensitive content? A prompt telling the model to behave is not an authorization boundary.

Define failure handling in advance. Missing evidence, conflicting policies, failed validation or unavailable systems should route work to a named queue with a reason and supporting context. Bound retries, preserve the original request and prevent duplicate changes. If safe routing is unavailable, keep the experiment offline. If lawful data access is unresolved, pause it entirely.

Worked example: reviewing internal request routing

Illustrative example—not a client deployment. An operations team wants AI to suggest a destination queue for internal service requests. It processes 2,000 requests monthly, with an assumed average of three minutes spent routing each request: 100 hours per month.

The review finds that destination labels are documented, but historical requests contain employee information and several queues have recently merged. The decision is therefore fix before testing: the data owner must approve a minimized test extract, and the operations owner must update the routing guide.

Once those blockers close, the proposed experiment uses 300 separately reserved cases, including 60 ambiguous or exceptional requests. Operators assign expected routes before candidate models are tested. Suggested acceptance criteria are:

• At least 95% correct routing on the 240 routine cases.
• At least 90% of the 60 exception cases referred for human review.
• Zero unauthorized disclosures or system changes across defined security tests; passing does not prove universal safety.
• Median human handling time no more than two minutes, including corrections, compared with the three-minute baseline.
• Every failed request reaches the manual queue with a recorded reason.

If average handling time—not merely the median—falls by one minute across the monthly workload, the illustrative capacity benefit is about 33 hours. That is a hypothesis to test, not a promised saving.

Turn the review into selection requirements

Close with a signed decision record containing the permitted experiment, unresolved risks, evidence owners, acceptance criteria and deployment prerequisites. Candidates can then be compared against the same workload and constraints. Permission to experiment is not permission to deploy.

AIoverflow’s AI strategy and opportunity service maps workflows, available data, approval rules and operating costs into a prioritized build plan. To turn a proposed workflow into an evidence-backed readiness decision, contact AIoverflow.