Data contracts: the interface an AI feature breaks first

A data engineering lead at an insurance company walked me through a pipeline that had started dropping records without raising a single alarm. The model read a claim, produced a structured summary, and the next step loaded that summary into their system. It had run cleanly for weeks. Then, slowly, rows stopped landing. No errors, no failed job, just fewer records than there should have been. The model had started phrasing one field slightly differently, and the code reading it had quietly failed to parse it and moved on.

Free-form output meets a strict consumer

A model produces text. Even when you ask for JSON, you are asking a system whose whole nature is to generate plausible language, not to honor a schema the way a typed function does. Most of the time it gives you what you asked for. Sometimes it renames a field, wraps the object differently, adds a stray sentence before the JSON, or returns a value in a shape the consumer did not expect.

The system on the other side is the opposite. Your loader, your database, your downstream service expects an exact shape. A specific field, a specific type, a specific structure. When the model’s output drifts even slightly from that shape, the consumer has two ways to react, and both are bad if you did not plan for them. It can throw, which at least makes noise. Or it can silently skip what it cannot parse, which is worse, because now you are losing data and everything looks green.

This is the interface that breaks first, and it breaks quietly. An upstream prompt change, a model version update, or just the natural variation in generated output is enough. There was no schema sitting between the model and the system, so nothing was there to catch the drift. The pipeline did not fail loudly. It failed by degrees, and by the time someone noticed the missing rows, the cause was weeks back.

Put a validated contract in the middle

The lesson is to place an explicit, validated contract between the model and anything that consumes its output. The model is allowed to produce whatever it produces. Nothing downstream sees that raw output until it has passed a schema check that either confirms the shape or rejects it loudly.

What that contract does for you.

It defines the exact expected shape, so “correct output” is a specification and not a hope.
It validates every model response against that shape before any consumer touches it.
It rejects and routes non-conforming output somewhere visible, so drift raises an alarm instead of silently dropping data.
It gives you a place to add a repair or retry step, so a near-miss can be corrected rather than lost.
It turns a model version change into a caught test failure instead of a slow production leak.

A validated contract converts silent corruption into a loud, catchable error. That trade is almost always worth it, because a loud failure you can fix in an hour and a silent one you find weeks later after the data is already wrong.

How we approach it at Density Labs

Most AI pipelines that lose data do not lose it to a dramatic bug. They lose it at the seam between a model that generates language and a system that demands a schema, with nothing validating the handoff. This is a large part of why so many AI efforts stall on integration and operations rather than on model quality.

Our AI Opportunity Assessment is a fixed two-week engagement, priced at $2,500, that scopes the real production work before you build. Part of that is defining the data contract between the model and every system it feeds, and deciding what happens when output does not conform, so drift shows up as an alert instead of a mystery.

The model will surprise you. A contract decides whether that surprise is a caught error or a silent leak.