Most enterprise AI pilots die between demo day and production go-live. The model wasn’t wrong. The team wasn’t bad. The problem is that productionization was scheduled for “the next phase” instead of designed in from week one. By the time the demo lands, the pilot has none of the things production demands: real data access, evaluation harness, audit trail, runbook, ownership. And the next phase never comes, because the next phase is a full rebuild that nobody budgeted for.
I have watched this pattern fail dozens of times. I have also deployed against it. The framework below is what we use when a client wants a production AI system in a quarter, not an AI maturity model in a year.
What “production” actually means (because most pilots are confused about this)
An AI system is in production when five things are true at once, all measurable. Most teams call a system “production” when only two or three are true, which is why it breaks. The five:
- It runs against real production data, not a sandbox copy.
- Real users are using it for real work. Not test users on test tasks.
- An evaluation harness is running continuously and surfacing regressions.
- Risk, audit, and compliance have signed off the governance documentation.
- Operations can execute the runbook without the build team in the room.
If any of those five is missing, you have a pilot with production traffic, which is the worst of both worlds. Most “production” AI systems I get called to fix are stuck at four of five.
The 8-week framework
We open every AI engagement on the same cadence. The week numbers below assume a scoped, single-use-case pilot in a non-regulated environment. Regulated industries (finance, healthcare, government) add four to six weeks for governance handoffs. The technical work doesn’t change; the documentation work does.
Weeks 1 to 2: Outcome lock and architecture decision
The first two weeks aren’t about building. They’re about locking the outcome and making three architecture decisions that determine everything else.
Lock the outcome. “Reduce average handle time on tier-2 support tickets by 30% within 90 days of production go-live” is an outcome. “Pilot generative AI for customer service” is not. The outcome statement needs a business metric, a target, a population, and a timeframe. If we can’t agree on those four by end of week 1, we won’t get to production by week 8. Better to spend the rest of week 1 fighting about the outcome than to spend week 7 discovering you never had one.
Architecture decision 1: foundation model choice. Vendor-neutral. We profile data residency requirements, latency budget, cost-per-call ceiling, and existing cloud commitment, then pick a model. We don’t bring vendor preferences. We bring the math.
Architecture decision 2: retrieval pattern. If the use case requires retrieval (most do), we decide hybrid search architecture, vector DB, reranker, and evaluation methodology here. Not in week 6 when retrieval accuracy is already a problem.
Architecture decision 3: governance posture. Audit-trail format, access controls, human-in-the-loop checkpoints, escalation criteria. This is design input, not a documentation layer applied at the end.
End of week 2: we have a written Architecture Decision Record (ADR) that risk, compliance, and security have seen. If they have objections, we hear them now. Not at week 7, when the model is built and the demo is scheduled.
Weeks 3 to 5: Build with evaluation in parallel
Build the pilot system. Build the evaluation harness at the same time. Same week, same engineers. Not later.
For RAG systems we use RAGAS metrics (faithfulness, answer relevancy, context precision, context recall). For agent systems we use task-level metrics. We layer domain-specific metrics on top of either. Production targets we hit before user testing: faithfulness > 0.9, answer relevancy > 0.85, context precision > 0.8.
The build runs against a held-out evaluation set from week 3 forward. Every change runs the eval. Regressions caught here cost hours. Regressions caught in production cost months and trust, in that order.
Weeks 6 to 7: Closed beta with real users
By week 6 we have a system that meets evaluation targets and a small population of real users (not internal test users, actual customer service reps, actual underwriters, actual clinicians) using it for actual work.
This is where most pilots first meet messy reality. Edge cases the eval set missed. Adoption friction. Workflow integration issues. We fix in flight, with the evaluation harness running, so we know whether the fixes are improvements or regressions. The harness is what separates “we deployed it” from “we deployed it and we know it’s working.”
By end of week 7 we have measurable user-level data on the locked-in outcome metric. If it’s not on track, we tell you. Some pilots end here, and that’s a successful outcome. A pilot that proves the use case won’t work in production saved you twelve months of investment. Pretending otherwise is the consultant move. We don’t.
Week 8: Hardening and operational handoff
The system goes to broader production rollout. We finish the operational run-book, model drift detection, cost monitoring with circuit breakers, and the handoff documentation. Your team takes over operations. We move to advisory.
Production isn’t the end of the engagement. There’s typically a four to eight week stabilization phase where we’re on call as the system meets larger user populations. But ownership transfers at week 8. If a consultant won’t transfer ownership, they’re not selling you a system. They’re selling you a dependency.
Where the consensus is wrong
The standard advice is wrong: productionization is not a phase that comes after the model works. Most AI delivery guidance treats it as one. The certification courses teach it that way, the big-four playbooks teach it that way, and most internal AI teams default to it because it matches how they deployed non-AI software.
It doesn’t work for AI. The reason is specific: AI systems have failure modes that only appear under production conditions (real corpus messiness, real query distribution, real user expectations, real audit scrutiny). If you build the pilot in conditions that don’t surface those failure modes, you haven’t built a smaller version of the production system. You’ve built a different system. The “productionization phase” is a full rebuild dressed up as a phase, and that’s why most pilots die in the gap.
The fix isn’t more rigor in the productionization phase. It’s deleting the phase. Build the pilot in production conditions from week one. Evaluation, governance, real data, real users. Smaller scope, same shape.
What goes wrong (and how we fix it)
Four failure patterns show up most often when AI pilots stall: low user trust despite good eval scores, costs running 3x over budget, governance questions surfacing late, and wrong retrieval feeding a sound model. After deploying this framework dozens of times, they are predictable, and each has a known fix.
“The eval scores are great but users don’t trust the output.” Usually a UX problem, not a model problem. Fix: shorter answers, explicit uncertainty markers, “show your work” patterns (retrieved sources displayed inline). Trust is earned with transparency, not accuracy. A model that’s 95% accurate but opaque loses to a model that’s 85% accurate and cites its sources.
“Costs are 3x what we budgeted.” Almost always a caching problem. We instrument semantic caching for repeated queries (30 to 50% LLM call reduction in customer-service and knowledge-base workloads) and re-architect to a reranker-first pattern (rerank cheap, generate expensive). Cost-per-query usually drops 50 to 70% with these two changes alone.
“Risk and compliance have new questions every review cycle.” Symptom of governance built retroactively. The questions aren’t new; they always existed. They just weren’t asked until the demo. Fix can’t be applied late. We always build audit trail, authority limits, and runtime policy enforcement into the system from week one. Pilots that don’t are stuck and the fix is a rebuild.
“The model is fine but the data we’re feeding it is wrong.” The retrieval problem. About 40% of naive RAG systems retrieve the wrong documents. The pilot looked good because the demo data was curated. The production corpus is not. Fix: hybrid dense plus sparse retrieval with cross-encoder reranking, then re-evaluate. Almost always recovers production accuracy.
What the framework is not
It’s not a strategy document. It’s not maturity-model theater. It’s not a pitch for a multi-year transformation. It’s the eight-week engagement shape we use to deploy one production AI system for one outcome, and to leave you a repeatable internal pattern for the next ten.
If you have a scoped AI use case where the outcome is clear and the team has hit pilot-purgatory before, this is the framework. If you don’t have an outcome yet, start at AI Transformation. That’s where outcome-locking happens. If your specific bottleneck is retrieval, RAG Systems is the right service. If you need orchestration across multiple steps, AI Agents is.
If you’re hitting pilot purgatory right now
The diagnostic is usually fast. If your pilot is stuck, one of three things is true: you don’t have an evaluation harness, you didn’t build governance in, or your retrieval is wrong. Almost everything else is downstream of those three. Pick the one that feels most familiar, fix it first, and the rest of the framework usually unblocks.
If you want a second opinion, get in touch. A thirty-minute call is usually enough to diagnose which of the three you’re hitting.
Written by a Rockmere SAFe® SPCT and AWS ML Engineer who has deployed production AI in financial services, healthcare, insurance, and telecommunications environments.