The AI systems engineering problem is that the model is the smallest part of what you ship. A convincing demo takes an afternoon. Turning that demo into something people can depend on takes months, and almost none of those months are spent on the model.
What eats them is everything wrapped around it:
- Getting the right information into the context window, and keeping the wrong information out.
- Evaluation that tells you whether a change made things better or worse.
- Failure handling for a component whose failure mode is confident, well-formatted text.
- Cost and latency budgets that survive traffic tripling.
- Access control, audit trails and whatever your regulator asks for next.
This is an old observation in new clothes. In 2015 a team at Google published Hidden Technical Debt in Machine Learning Systems and made the case with a diagram, noting that "only a small fraction of real-world ML systems is composed of the ML code." The box representing that code was small enough to be funny. Eleven years later it is smaller.
Where the AI systems engineering problem actually lives
Three findings from that paper survive the move to language models almost unchanged.
Glue code is the first. Connective tissue accumulates around the interesting component: adapters and retry wrappers among others. It gets written fast, tested rarely, and it outnumbers the code anyone set out to write.
Configuration debt is the second. Settings multiply and interact, and almost none of them are versioned. Anyone who has traced a production incident to a temperature value someone nudged for a demo in March knows this one personally.
The third has the best name: CACE, meaning changing anything changes everything. Adjust one input and behaviour shifts across the whole pipeline in ways no unit test predicted. With a language model this bites harder than it did with a gradient-boosted tree, because the component you swapped is a probability distribution over text and it has opinions about your prompt formatting.
Add one failure the 2015 authors did not have to worry about. Your most important dependency now ships on someone else's release schedule. A provider retires a model version, your tuned prompt starts producing subtly different output, and the weights you would need to inspect are not yours to inspect. We worked through that trade in when to build an LLM and when to skip it. The systems answer holds either way: pin what you can version, and evaluate what you cannot.
Building agentic AI applications with a problem-first approach
Start from the failure you want to remove. That sounds like advice from a management book, and it is the decision that most reliably separates a system that ships from one that demos.
Building agentic AI applications with a problem-first approach means writing down what currently goes wrong, how often it goes wrong, and what a fix is worth, before anyone picks an architecture. Teams that skip the step build an agent because agents are interesting, then discover the task wanted a database query.
Two questions do most of the work:
- What does the system have to be right about, and how would you know when it was wrong?
- What does a wrong answer cost when nobody catches it?
The second question sets your autonomy budget. A task that repeats often and fails cheaply can run unattended. A task where a mistake reaches a customer needs a person in the loop, and the engineering effort moves off the model and onto the review interface. Our survey of agentic AI use cases that are actually in production found that gap doing most of the sorting between pilots and deployments.
Once scope is settled, the binding constraint is context. Anthropic's engineering team describes context as a finite attention budget, and reports that a model's ability to recall information accurately declines as the token count in the window grows. They call the effect context rot. The target they set is the smallest set of high-signal tokens that produces the outcome you want.
That reframes a lot of design work as capacity planning. Retrieval quality and history compaction are both ways of spending a budget you cannot expand by paying more.
What an applied AI engineer builds around the model
An applied AI engineer builds product features on top of models somebody else trained. The title is new and the work is recognisable. The deliverable is rarely a prompt.
Here is the shape of an LLM feature that holds up in production:
| Layer | What it does | What happens without it |
|---|---|---|
| Input handling | Validates and normalises what reaches the model | Malformed input becomes malformed output |
| Retrieval | Selects the context the answer depends on | The model invents the missing part |
| Prompt and tools | Defines the task and the actions available | Behaviour drifts with every reword |
| Output contract | Enforces a schema on what comes back | A parser downstream fails at 3 a.m. |
| Evaluation | Scores changes against fixed examples | Every release is a guess |
| Observability | Records inputs, outputs and tool calls | Incidents cannot be reconstructed |
| Cost controls | Caps spend and latency per request | A retry loop bills like a crypto miner |
The evaluation row is where most teams underinvest. Without a scored set of examples, a prompt change is a vibe and a model upgrade is a coin flip you have no way to grade. Our piece on practices for training AI models with prompts covers building that scored library and keeping it useful across versions.
The unglamorous rows matter more than they look. An output contract catches the day the model returns a polite paragraph where your parser expected JSON, which it will do eventually, and always to the one downstream service with no error handling.
The AI security engineer inherits a new attack surface
An AI security engineer works on a surface that conventional application security does not cover. The root cause is structural: a language model has no reliable way to separate instructions from data. Both arrive as text in the same window.
Prompt injection follows from that directly. Text inside a retrieved document or a web page can carry instructions, and a model reading it may follow them. When the same model holds credentials and can call tools, an injected instruction becomes an executed action.
The controls that work are structural, and every one of them is dull:
- Grant the narrowest permissions that still let the task complete.
- Require human confirmation for any action with an irreversible effect.
- Treat every tool result as untrusted input, including results from your own services.
- Log each tool call with its arguments, so an incident can be reconstructed afterwards.
- Keep long-lived credentials out of the model's reach entirely.
None of those are model settings. They are system design, which is this article's argument arriving in the place where it costs the most to ignore.
AI transformation is a problem of governance
Past a certain scale the engineering problem turns into an organisational one. AI transformation is a problem of governance as much as a problem of infrastructure, and the AI governance problem usually surfaces as a question nobody in the room can answer: who approved this system, and against what criteria.
NIST's AI Risk Management Framework, released in January 2023, gives that question a usable spine. It splits the work into four functions:
- Govern: the policies and accountability structures the other three sit inside.
- Map: establishing context and identifying where harm could occur.
- Measure: assessing and tracking the risks you mapped.
- Manage: acting on them in priority order.
Govern is the function teams skip, because it produces documents instead of features. It is also the one that decides whether the other three happen once, or once a quarter for as long as the system runs. For the organisational view of the same question, Agentic AI in Business works through autonomy boundaries and accountability for the readers who sign off on these systems.
Where the autonomous AI software engineer fits
The autonomous AI software engineer is the version of all this that engineers meet first. An agent reads the repository, edits files and runs the test suite until something passes.
It works because software happens to be unusually well shaped for an agent. The environment is text the model can read directly, and the grader is a test runner that returns a verdict in minutes. Very few domains hand you a feedback loop that honest.
What it changes is where the bottleneck sits. Producing a change got cheap. Deciding whether the change is correct did not, so review becomes the scarce skill, an argument we made at length in whether software engineers will be replaced by AI.
Which brings the whole thing back around. A coding agent is a system with a feedback loop, a permission boundary and a review step. Remove those and what remains is a very expensive text generator with commit access.
The ratio from 2015 still holds. Budget the model at a tenth of the work and spend the rest on what surrounds it, because when an AI project runs late the model is almost never the reason.
Four things to settle before committing to an AI feature:
- Write down the failure you are removing and what it costs today.
- Build the evaluation set before you write the prompt.
- Decide the permission boundary before you grant the first tool.
- Name the person accountable for what the system outputs.
Do that and the model becomes what it should have been from the start: the easiest dependency in your stack, and the only one that gets better while you sleep.