From prototype to production: an LLMOps pipeline

An AI prototype usually comes together in a week. The distance between that prototype and something running in production with real users is where most projects die, and it is rarely the model's fault.

What changes when you do not train the model

The MLOps discipline was built around a pipeline that ends in model training and deployment. Most generative AI projects train nothing: they consume a model through an API.

That removes some stages and creates others. Periodic retraining disappears. Prompt management, context management, per-call cost control and a dependency on a vendor that updates the model without asking all appear.

What still holds from classic MLOps is the backbone: version everything that influences the result, automate the path to production, and monitor continuously afterwards. Google Cloud's MLOps guide describes that progression as maturity levels, and nearly all of it applies here.

Version whatever actually changes the answer

In an LLM application, the artifact is not only the code. The output changes if any of these change:

• the prompt, including system instructions;
• the model and its version;
• generation parameters;
• the indexed knowledge base and retrieval configuration;
• the available tool definitions.

If any of those lives outside version control, you cannot reproduce an incident. A prompt sitting in an environment variable edited through a dashboard is the most common mistake, and the most expensive on the day answers degrade without anyone having deployed.

Entanglement is the hard problem

A Microsoft case study on software engineering for machine learning identified a structural difference from traditional software: AI components are harder to treat as isolated modules, because they become entangled in non-obvious ways.

In practice that means the change that looks local is not. Tuning the prompt to improve one question type degrades another. Swapping the embedding model invalidates the whole index. Changing temperature changes the tool-invocation rate.

That is why the evaluation suite has to run across the full set of scenarios on every change, not only the scenario that motivated the change.

A minimal pipeline that is already serious

Four stages cover most of the risk:

1. Development. Prompts and configuration versioned alongside the code. Fast local evaluation before opening a PR.
2. Integration. Deterministic evaluation in CI, plus the golden set. A failure blocks the merge.
3. Progressive rollout. The new version behind a flag, released to a fraction of traffic, with quality and cost compared against the current version.
4. Observation. Logging input, output, prompt version, latency, cost and user feedback, with sampling for human review.

Stage 3 is the one most often missing in the projects I see, and it is what turns a large error into a small one.

Cost is a requirement, not a report

In traditional systems, cost shows up on the invoice at month end. In an LLM application, cost is a direct function of design decisions: context size, number of calls per interaction, model chosen per route.

Treat cost per interaction as an SLO. Define the acceptable ceiling, measure it in production per route, and alert when it is breached. Without that, optimisation only happens after the invoice shock, and then it gets done in a hurry without evaluation, degrading quality.

The fallback has to exist before you need it

Depending on an external API means outages outside your control. Define the behaviour in advance: cached response, alternative model, explicit degradation for the user, or a queue to process later.

Choosing that during the incident is like choosing the rollback plan during the rollback.

The same applies to rate limits. A vendor quota you share across features means one noisy feature can starve the rest, so decide upfront which route gets priority when the budget runs short.

What this means for your team

Start by moving the prompt out of anywhere that is not the repository. It is the cheapest change with the largest effect on reproducibility.

Then add progressive rollout. It needs no sophisticated platform: a config flag, a fraction of traffic and a dashboard comparing quality and cost across versions is enough.

And set the cost ceiling per interaction alongside the functional requirement, in the same conversation. That is cheaper than discovering three months later that the chosen architecture does not add up.