AI Pilot to Production: the Checklist Most Teams Skip

An AI pilot to production checklist has six items, and none of them is "improve the model." To move a working pilot into production, you need: data and integration readiness so the tool runs on live systems, a single named owner accountable for the outcome, an adoption plan for the humans in the loop, a monitoring and tuning cadence, one success metric tied to the business, and an exit and handover so the capability outlives the project. Most teams nail the demo and skip all six. That is why the pilot works in March and is dead by September.

Why does a working pilot not mean you are ready for production?

Because a pilot proves the model can produce a good answer under ideal conditions, and production asks a completely different question: can this run every day, on live data, with real users, without you standing next to it.

The pilot ran on a clean file someone exported by hand. Production runs on your Shopify events, your Braze flows, your Zendesk tickets, your data warehouse, all changing hourly. The pilot had the founder demoing it. Production has a support agent on the overnight shift who was never trained. MIT found that 95% of AI pilots deliver no measurable business impact, and the reason is rarely the model. It is everything downstream of the model that nobody scoped.

Spinning up the tool was the easy 20%. The durable 80% is the six items below.

Is your data and integration actually production-ready?

The first checklist item is the one that quietly kills the most projects. Ask:

• Live data access. Does the tool read from your real systems through an API or pipeline, not a manual export? If a human has to drop a file in a folder for it to work, it is not in production.
• Write-back. Where do the outputs go? A generated support reply, a churn score, a product tag: it has to land back in Zendesk, your CRM, or your PIM automatically, or the value leaks out in copy-paste.
• Failure behavior. What happens when the upstream API rate-limits you, the data schema changes, or a field comes back null? Production systems degrade gracefully. Pilots crash silently.
• Latency and volume. The pilot handled a handful of records. Can it handle your full catalog and your Black Friday peak without timing out?

For consumer-tech teams this is where reality bites: your data lives in six SaaS tools that were never designed to talk to each other. Integration readiness is not a formality, it is most of the actual work.

Who is the named owner after handover?

Every production capability needs one person whose name is on it. Not a committee, not "the data team," not "we all own it." One owner.

This person is accountable for the success metric, decides when the tool is retuned or paused, and is the escalation point when it misbehaves. If you cannot name this person before launch, you are not launching a capability, you are launching an orphan. Orphaned tools do not get maintained, and unmaintained AI tools drift and rot faster than normal software because the world they model keeps moving.

Name the owner in writing. Give them the authority and the time to actually own it.

What does the adoption plan look like?

A tool nobody uses has zero impact, no matter how good the output. This is the Stickiness dimension in our Durable AI Index, and it is the one everyone ignores.

Your adoption plan needs three concrete things:

1. A workflow it fits into. The tool has to live where the work already happens. A churn model that emails a PDF to a manager once a week will be ignored. The same score written into the CRM record the CS rep already opens gets used. 2. Training for the actual users. Not a launch email. A short, hands-on session for the agents, marketers, or analysts who touch it daily, including what to do when the AI is wrong. 3. A trust mechanism. People will not use a black box they cannot check. Show the reasoning, cite the source, or make it easy to override. McKinsey's finding that end-to-end workflow redesign is the number one driver of AI value is really a statement about adoption: value shows up when the process changes around the tool, not when the tool is bolted on.

For deeper structure here, see our AI change management framework.

What is the monitoring and tuning cadence?

AI in production is not fire-and-forget. Model outputs drift as your data, catalog, and customers change. You need a scheduled cadence, decided before launch, not after something breaks:

• What you watch. Output quality (a sampled human review), usage rate (are people actually using it), and cost per run.
• How often. Weekly for the first month, then a fixed monthly review. Put it on a calendar with the owner's name on it.
• What triggers a retune. Define the threshold in advance: quality below a set bar, usage below a set floor, or a known upstream change like a catalog migration or a new product line.

Without a cadence, you find out the tool broke when a customer complains, which for a consumer brand is the most expensive possible way to learn.

What is the single success metric?

Pick one metric that a skeptical CFO would accept, and define it before you launch. Not "engagement," not "hours saved" measured by vibes. One number tied to the business: resolution time, contribution margin on a segment, first-response time, conversion on a flow.

The rule is that the metric must be measurable both before and after, so you can prove the delta. If you cannot measure the baseline today, measure it during the pilot before you scale. RAND found that roughly 80% of AI projects fail to meet their goals, and a hidden reason is that many never defined a goal precise enough to fail against. A vague metric is how a dead pilot gets to call itself a success for six months. For the full treatment, see how to measure AI ROI.

What is the exit and handover?

The last item is the one consultants and internal project teams both love to skip, because it means letting go. A production capability has to run without the people who built it.

Handover means: documentation a new owner can actually follow, credentials and access transferred off personal accounts, the monitoring cadence handed to a named team, and a clean definition of done. The exit is not abandonment. It is the proof that you built a capability and not a dependency. If the tool only works while the original builder babysits it, you have a very expensive demo, not production.

Frequently asked

How long should an AI pilot run before moving to production? Long enough to measure your success metric against a real baseline and to see the tool handle live data volume, usually a few weeks to a couple of months. The trigger to graduate is not the calendar, it is passing the six checklist items above, especially a proven metric and named owner.

What is the most common reason pilots fail to reach production? Integration and adoption, not model quality. The pilot ran on hand-cleaned data with the builder driving. Production needs live system access and real users who trust and use it inside their existing workflow. Skip either and the tool stalls.

Do we need a dedicated owner if we already have a data team? Yes. A team is not an owner. You need one named person accountable for the success metric and empowered to pause or retune the tool. Shared ownership across a team reliably becomes no ownership.

Can we skip monitoring if the pilot results were strong? No. Strong pilot results decay. Your catalog, customers, and data schemas change, and model outputs drift with them. A monitoring cadence is what separates a capability that stays valuable from one that quietly degrades until a customer notices.

Where to start

If you have a pilot that demos well and you are not sure it will survive contact with production, that is exactly the gap we close. Start with our services, see the workflows we build for consumer teams at /ai-for, or get in touch to pressure-test your pilot against this checklist. For related reading, see why AI pilots fail and how to prioritize AI use cases.