LLMOps is the practice of operating applications built on large language models in production. It covers evaluating prompt and model changes before release and monitoring quality and cost once users arrive. Most teams rent the model through an API, so the work centres on everything around it.

What is the difference between LLMOps and MLOps?

MLOps centres on training pipelines and retraining models you own. LLMOps centres on prompts and inference cost around a model you usually rent. Both need version control for anything that changes behaviour, and both need a fast way to roll back.

Do small teams need LLMOps?

Yes, in a light form. A pinned model version and an evaluation set of around 100 real cases cover most of the risk, and basic tracing with token counts covers the rest. Larger platforms can wait until traffic justifies them.

What are LLMOps tools?

Most fall into two groups: observability platforms that record traces and token costs, and evaluation frameworks that score outputs before release. LLM gateways and prompt registries are common additions. Your Git repository can serve as prompt management for a long time.

How do you monitor an LLM in production?

Record a trace for every request with the model version and token counts attached, and track error rates and latency per prompt version. Then score a daily sample of live outputs with your evaluation rubric, because most LLM failures still return a healthy HTTP 200.

Is LLM-as-a-judge reliable?

It is reliable enough to use with calibration. In the MT-Bench study, strong judge models agreed with human preferences over 80% of the time, close to how often humans agree with each other. The same study found position and verbosity bias, so check the judge against human grades and swap answer order in comparisons.

LLMOps: How to Run LLM Apps in Production Without Surprises

LLMOps is the set of practices for running large language model applications in production. It covers two jobs: proving a change is safe before it ships, and watching quality and cost once real users arrive. If you have an LLM feature live, you are already doing LLMOps. The only question is whether you are doing it on purpose.

The term sounds like vendor marketing, but the problem is real. A prompt edit or a model upgrade can change outputs for thousands of users without throwing a single exception. This guide covers what LLMOps means in practice and the minimum setup a small team needs to do it well.

What LLMOps is and how it differs from MLOps

In plain terms, LLMOps is operations work for software whose core component is a language model you probably did not train. You rent the model through an API and wrap it in prompts and retrieved documents.

That last detail changes the job. Classic MLOps, the discipline of training and redeploying machine learning models, centres on the training pipeline and its scheduled retraining. Most LLM teams never touch a training run. Their moving parts sit around the model.

	MLOps	LLMOps
Changes most often	Training data and model weights	Prompts and the model version you call
Main quality signal	Accuracy on a labelled test set	Graded outputs on an evaluation set, often scored by another model
Biggest cost driver	Training compute	Tokens at inference time
Typical failure	Data drift lowers accuracy	A prompt tweak breaks an edge case nobody tested

The overlap is bigger than vendors admit. Both need version control for everything that affects behaviour, and both need a way to roll back. If your team already runs MLOps, roughly half of LLMOps is the same muscle with new names on the dashboards. The other half is MLOps after it discovered the prompt, and then the invoice.

The LLMOps lifecycle: treat prompts like code

Every change that can alter model behaviour takes the same path as a normal software release:

Commit the change, whether a prompt edit or a model upgrade.
Run the evaluation set and compare scores against the current production version.
Release to a small slice of traffic and watch the live metrics.
Promote it, or roll back to the pinned previous version.

The habit that matters most is pinning. Call a dated model snapshot and record it next to the prompt version in every log line. An alias that points to the latest model means your application can change behaviour on a Tuesday afternoon with no deploy on your side.

Prompts deserve the same treatment. Keep them in the repository and review them in pull requests. Give each one an ID that shows up in traces. A prompt stored in a database field anyone can edit is a production change with no review step, and it will be edited at 5 p.m. on a Friday by someone who only fixed a typo.

This discipline is where most teams discover that their AI problem is a systems problem. We made that case in why AI is a systems engineering problem, and LLMOps is the operational half of that argument.

LLM evaluation before anything ships

An evaluation set is a fixed list of inputs, each with a description of what a good answer looks like. Run every candidate change against it before release. Without one, "the new prompt seems better" is a feeling, and feelings do not survive contact with production traffic.

Start with 50 to 200 real cases pulled from logs, weighted toward the inputs that have broken before. Score them in layers. Deterministic checks come first: does the output parse as JSON, and does it cite a document that exists? Those checks are cheap and never argue with you.

For qualities a regular expression cannot judge, such as helpfulness or tone, teams use a second model as a grader, a method known as LLM-as-a-judge. The research supports it with caveats. In the MT-Bench study by Zheng and colleagues, strong judges such as GPT-4 matched human preferences over 80% of the time, about the rate at which humans agree with each other. The same paper documented position bias and verbosity bias, where the judge favours whichever answer comes first or runs longest.

So calibrate the judge. Have a person grade a sample of the same cases and confirm the judge agrees before trusting its scores at scale. Swap the order of answers in pairwise comparisons. A judge that prefers long answers will happily reward a prompt that doubled the length of every reply, which is a regression wearing a medal.

Writing the grading rubric is itself prompt work. AI Prompt Engineering treats prompts as systems that are tested and maintained over time, which is the right mental model for a judge you will run thousands of times.

LLM monitoring and observability in production

Monitoring tells you something is wrong. Observability tells you why, by recording what the model saw and did on each request.

The unit of observability is the trace: one record per user request, with a span for each model call and tool call inside it. OpenTelemetry, the open standard most observability tools speak, publishes semantic conventions for generative AI so vendors name the same fields the same way. An inference span records the requested model in gen_ai.request.model and token counts in gen_ai.usage.input_tokens and gen_ai.usage.output_tokens. The conventions are still marked as in development, so expect some names to shift.

One detail in that spec is worth copying. Full prompt and response text are opt-in attributes, and the spec warns they may contain sensitive information. Log content deliberately, with retention limits and redaction, because a trace store full of customer messages is a data-protection liability.

Laptop showing analytics charts, the kind of dashboard LLMOps teams use for LLM monitoring — Photo by Lukas Blazek on Pexels

What to watch on the dashboard:

Error and timeout rates per model and per prompt version
Latency at the 95th percentile, including time to first token for streamed replies
Tokens per request, split into input and output
Finish reasons, since a spike in replies cut off at the length limit is an early sign of a prompt problem
A small daily sample of live outputs scored with the evaluation rubric
User signals such as thumbs-down rates and retries

Most LLM failures return a perfectly healthy HTTP 200, which is why the last two items matter. Rate limits matter too: our guide to the Claude rate exceeded error shows what throttling looks like from the client side.

Managing LLM cost and latency

LLM cost scales with tokens, the word fragments a model reads and writes. Every request pays for its input and its output, so the cheapest optimisation is usually sending less.

Long system prompts repeat on every call. Prompt caching targets exactly this: the provider stores the processed prefix of a prompt and reuses it on the next request. Anthropic's prompt caching documentation prices cache reads at 0.1 times the base input price on most models, with a default lifetime of five minutes that refreshes each time the cache is used. Writing to the cache costs 1.25 times the base price, so caching pays off when the same prefix comes back within that window.

Put stable content first and variable content last, because caching works on prefixes. A timestamp at the top of the system prompt changes the prefix on every call and quietly disables caching.

Model routing is the other lever. Send easy requests, such as classification or short extraction, to a small fast model and reserve the large model for the hard cases. Track cost per successful task as the headline number. A cheaper model that needs three retries costs more than an expensive one that answers once.

LLMOps tools: what to buy and what to build

The tooling market splits into a few categories, and most teams end up with one product from each:

Tracing and observability platforms that ingest OpenTelemetry-style traces and show cost per request
Evaluation frameworks that run test sets and judge models in continuous integration
Prompt management, which can be your Git repository
LLM gateways that sit between your app and providers to handle retries and fallbacks

Buy observability early, since building a trace viewer is a poor use of an engineering quarter. Build your evaluation set yourself. No vendor knows what a correct answer looks like for your users, and the dataset is the one asset in this stack that gets more valuable every month.

A minimum viable LLMOps setup for a small team fits on one page:

Prompts in Git, each with an ID that appears in traces
A pinned model version, upgraded only through an evaluation run
About 100 evaluation cases run on every change
Traces with token counts and finish reasons
A weekly review of the worst-scored live outputs

That catches most regressions before your users do. The rest can wait until the dashboards ask for it, and they will ask, usually during a demo.

For the engineering depth behind each layer, AI Engineering covers prompt workflows and agent architectures for production in one volume. LLM Systems Engineering goes below the API into fine-tuning and training, for teams that eventually outgrow a rented model.

LLMOps Explained: How to Keep LLM Apps Working After Launch