The failure is rarely the model. It is asking one prompt to satisfy too many constraints at once — and having nothing in place to catch it when it quietly gets one wrong.
A demo runs once, on input someone chose, watched by a person who will forgive a retry. Production runs thousands of times on input nobody inspected, and the failures are seen by customers — or by no one at all, which is worse. Systems that look identical in a demo behave completely differently under those two conditions, and the difference is almost never the model.
The most common structural failure I find is a single large prompt carrying every requirement at once. It works during prototyping, when you are testing one requirement at a time, and degrades as soon as all of them apply simultaneously.
The tell is characteristic: output that is never wrong in the same way twice. Voice is right but format slips. Format is right but a rule is dropped. Each individual run looks like bad luck; in aggregate it is the architecture telling you the request is overloaded.
The fix is decomposition — re-architecting one large request into focused stages, each with a single job, each verifiable on its own. It costs more calls and gets you an output you can actually reason about. Where a stage matters enough, a second pass cross-checks the first.
A crash is cheap. Something raised, something logged, somebody knows. The failures that cost real money are the ones that produce confident, well-formatted, wrong output and return HTTP 200.
So the question worth asking about any AI step is not "what happens if it fails" but "would we find out". Concretely, for each stage:
Reliability and safety converge on the same question — what is this system permitted to do? An AI step that can take irreversible action is a different risk class from one that returns text, and the distinction is often never drawn explicitly.
Worth separating deliberately: destructive or irreversible operations, anything touching customer-visible state, anything that spends money, and anything that reads data the output should not be able to leak. Each of those wants either a hard block or a human checkpoint — and a human checkpoint is only real if the human has enough context to say no. An approval step nobody can meaningfully evaluate is theatre.
The same logic extends to data at rest. On one production system, sensitive fields — phone numbers, messages, passwords — are encrypted with salt and pepper and GPU-resistant hashing, with decryption keys held off-server. The AI layer being clever does not exempt the boring parts from being correct.
Most teams evaluate AI output by looking at it. That scales to roughly a dozen examples and then stops working entirely, which means quality regressions ship unnoticed until a customer reports one.
Evals fix this: a fixed set of representative inputs, an explicit definition of a correct response, and a score you can watch move. They do not need to be sophisticated to be useful. A few dozen real cases with clear pass criteria will catch more regressions than any amount of re-reading prompts, and they turn "it feels worse since we changed the model" into a number.
Once that exists, drift becomes visible instead of anecdotal, and you can change models on evidence rather than vibes.
If your AI is unpredictable in production, the highest-yield sequence is: find the overloaded prompt and split it; add verification to the stage whose failure costs most; write down what the system is never allowed to do; then build a small eval set so you can tell whether any of it helped.
That is the same order the free AI Systems Audit works through — mapping failure points, missing guardrails and output drift against your actual system, and ranking them by what they cost you.