The config that was different in prod and broke the model

A team at a mid-market SaaS shipped an AI feature that behaved beautifully in staging and strangely in production. Same code, same model version, same prompts as far as anyone could tell. The outputs in prod were wilder and less consistent, and it took the team two days of staring at identical code to find the cause. Production ran the model at a higher temperature than staging did. One number, set differently in two config files, was enough to change the feature’s behavior from steady to erratic.

Once they saw it, it was obvious. Temperature controls how much the model varies its output. Staging had it low, so responses were tight. Prod had it high, left over from an experiment months earlier, so responses wandered. The code was blameless. The bug lived in a config value that no one had reviewed, that differed silently between environments, and that no test would ever catch because tests exercise code, not configuration drift.

AI config is behavior, and it deserves the same rigor as code

Teams treat model configuration as settings, a layer beneath the real work. But for an AI feature, config is behavior. Temperature, max tokens, the system prompt, the region the model runs in, the model version itself: change any of these and the feature acts differently, sometimes dramatically. These values have as much influence on what users see as the code does, and yet they often live outside the discipline that code gets.

Code is versioned, reviewed, and diffed. Config frequently is not. It gets edited directly, drifts between environments, and carries stale experimental values that no one remembers setting. So a feature can be identical in code across staging and prod and still behave like two different features, because the config underneath it silently disagrees.

What teams that keep config honest do

They keep AI config in version control, so every value has a history and a diff, the same as code.
They review config changes, because a temperature or a system-prompt edit can change behavior as much as a code change.
They check environment parity, so staging and prod differ only where they are meant to, and a drift like a stray temperature gets caught.
They treat the system prompt as reviewed, versioned config, not a string someone edits live in a dashboard.
They surface the effective config a feature is running under, so when behavior looks wrong, the actual values are one glance away instead of two days of hunting.

The principle is simple. If a value changes how the model behaves, it belongs under the same controls as the code that changes how the feature behaves. Anything less leaves a whole class of bugs that no test can see.

How we approach it at Density Labs

In an AI Opportunity Assessment, our fixed two-week engagement at $2,500, we pull the AI config for every environment and diff it. More than once we have found the environments quietly disagreeing on temperature, max tokens, or the system prompt, which means the feature that passed in staging is not the feature running in prod. We also check whether config is versioned and reviewed at all, or whether it is edited live where a stray keystroke becomes a production behavior change.

We scope the work to bring config under control: version it, review it, and add a parity check between environments so drift is caught before it ships. We do this before more features inherit the same loose config practice, because config drift is a bug that hides in plain sight and costs days to find each time. The Diagnose scopes that real production work.

Treat AI config as versioned, reviewed, parity-checked config. When it changes model behavior, it is not settings, it is code by another name.