Ask for JSON and you get JSON with an apology on top

An integration engineer at a payments company sent me a stack trace at the end of a long day. His pipeline had fallen over on a JSON parse error. The strange part was that the JSON looked perfect. He had pasted it into a validator and it passed. So why was his code refusing it?

Because his code was not receiving the JSON he pasted. It was receiving the whole model response, and the model had been polite. The actual string started with “Sure, here is the JSON you asked for:” followed by a Markdown code fence, then the object, then a closing fence. His parser got a friendly sentence where it expected an opening brace, and it gave up on character one.

The model answers a person, not a parser

His prompt said, in plain words, “return the result as JSON.” The model did. It also did what it does with almost any request, which is to address the human it imagines is reading. It framed the answer. It wrapped the code in a fence, the way it would in a chat window, because that is how JSON is presented to people. Everything it added was helpful in a conversation and fatal in a pipeline.

This is the trap in asking for a format using prose. A sentence like “respond in JSON” is a request to a writer, and a writer packages things for a reader. The model has no way to know that a program, not a person, is on the other end and that the program wants the raw object and nothing else. So you get valid JSON with an apology on top, and your parser, which is stricter than any human, rejects the first friendly word.

Prose is not a format contract

The lesson is that describing a format in the prompt does not enforce that format. It is a suggestion the model usually honors in spirit and rarely honors to the byte. Sometimes the preamble is there. Sometimes the fence is there. Sometimes the model adds a trailing note explaining a field. The variation is the whole problem, because your parser needs the same exact shape every time and the prose request cannot promise that.

There are two real fixes, and they stack:

Use structured-output constraints. Most current models offer a JSON or schema mode that forces the raw object, with no preamble and no fence. Turn it on. This removes the wrapping at the source instead of cleaning it up later.
Strip and validate anyway. Even with structured mode, do not trust the raw string. Extract the object, parse it, and validate it against the shape you expect before anything downstream touches it.
Never assume the string is clean. Treat the model response as untrusted text until you have parsed it successfully. Feeding it straight into a strict parser is how you get a 2 a.m. stack trace.
Fail loudly on a parse error. If the object does not parse, catch it and handle it. Do not let a malformed response flow through as if it were data.

The order matters. Constrain first, so the model stops adding the wrapping. Validate second, so the one time it slips through, you catch it instead of shipping garbage into your system of record.

How we approach it at Density Labs

In the AI Opportunity Assessment, our fixed two-week, $2,500 engagement that scopes real production work before you build, we look for exactly this pattern: a prompt that politely asks for JSON and a parser that assumes it got clean JSON. It works in the demo and breaks in the week. We turn on structured output where the model supports it, and we add a strip-and-validate step regardless, so a stray preamble becomes a caught error rather than an outage. The cost of finding this in week one is a code review. The cost of finding it in production is a paged engineer.

The model is talking to a person it cannot see. Constrain the output so it stops being polite.