- An eval suite that runs before every deploy is the single highest-leverage safety net.
- Every LLM call needs a fallback path for when the model is slow, wrong, or down.
- Cost caps prevent one bad prompt or retry loop from producing a five-figure bill.
- Ship behind a feature flag so a bad release can be killed in seconds, not hours.
Build the eval suite before the feature
The instinct is to build the feature first and test it manually. The better order is to write a small eval suite — real examples with expected outputs — before the integration code, so every future change gets checked automatically instead of by a person eyeballing outputs.
Every call needs a fallback path
LLM APIs are slower and less predictable than a typical database call, and they occasionally return something wrong or unusable. A production integration needs a defined fallback for each of those cases — a cached answer, a simpler rule-based response, or a graceful 'try again' — not a blank screen or an error page.
Cap cost before it caps you
A single retry loop or an unexpectedly long user input can turn a normal request into an expensive one. Setting hard limits on token usage, retries, and per-user request rate prevents an edge case from becoming a surprise five-figure bill at the end of the month.
Ship behind a flag, not a big-bang release
Rolling an LLM feature out to 5% of users behind a feature flag, then watching quality and cost metrics before expanding, catches problems while they're small. A full release with no kill switch means a bad prompt change is live for everyone until someone notices and reverts it manually.
Monitor quality, not just uptime
Standard uptime monitoring won't catch a model that's technically responding but giving worse answers than last week. Production LLM features need their own quality dashboard — sampled outputs reviewed regularly, plus automated checks against the eval suite — treated as a first-class metric, not an afterthought.