The diff that compiles, passes tests, and does the wrong thing
Every check was green. The change still solved a different problem than the one on the ticket, because the checks measured working, not correct.
The diff that compiles, passes tests, and does the wrong thing
A principal engineer at a retail company walked me through a change that still annoyed him. A ticket asked for the system to apply a discount to the highest-priced item in a cart. An agent wrote the change, it compiled, the existing tests passed, and a new test the agent added passed too. It shipped. A week later finance noticed the discount was landing on the highest-quantity item, not the highest-priced one. The agent had misread the intent, written code that did its own version consistently, and added a test that confirmed its own misreading. Everything was green. The behavior was wrong.
Passing checks confirm working, not correct
There is a gap that automated checks cannot close, and AI-generated code sits right in it. A compiler confirms the code is valid. A test suite confirms the code does what the tests say it should. Neither of them confirms that what the code does is what a human actually wanted. That last judgment lives outside the machine, in whether the behavior matches the intent, and no green check can supply it.
When a person writes code, they carry the intent in their head as they write, and the tests they add tend to encode that intent, however imperfectly. When an agent writes code, it works from its interpretation of the prompt, and that interpretation can be subtly off. The dangerous part is that the agent will then build a consistent world around its misreading. The code is internally coherent. The test it adds checks the behavior it built, which is the wrong behavior, so the test passes and vouches for the mistake. You end up with a change that is self-consistently wrong, wearing a full set of passing checks.
This is why “the tests pass” is a weaker statement than it used to be. A test written alongside AI-generated code can inherit the same misunderstanding as the code. Both are downstream of the same prompt. When they agree, they may just be agreeing on the wrong thing.
Verify intent separately from the code
The lesson is that someone has to check the change against what was actually wanted, as a separate act from checking that it runs. That check cannot be delegated to the same process, or the same prompt, that produced the code. It needs a human who holds the real intent and compares it to the observed behavior.
What teams do to catch self-consistently wrong changes.
- Confirm the behavior against the requirement by hand, running the real scenario and checking the outcome, not just that the suite is green.
- Read the tests as critically as the code, since an agent-written test can encode the same misreading as the agent-written code.
- Keep the person who owns the requirement in the loop on acceptance, so intent is checked by someone who holds it.
- Write at least one test from the requirement independently of the change, so it is not downstream of the same prompt.
- Treat a passing suite as necessary, not sufficient, and never as evidence that the change is correct.
The green checks are still worth having. They just answer a narrower question than people assume, and the gap between “it works” and “it does what we wanted” is exactly where AI-generated changes fail.
How we approach it at Density Labs
Most of the wrong-but-passing changes we see come from the same place. The intent lived in one person’s head, it got compressed into a prompt, the agent interpreted the prompt slightly off, and then it built code and tests that agreed with each other about the wrong thing. Every automated gate was satisfied. The only gate that would have caught it was a human comparing the behavior to what was actually asked.
Our AI Opportunity Assessment is a fixed two-week engagement, priced at $2,500, that scopes the real work of shipping AI features and AI-assisted code. Part of that is deciding where a human verifies intent, separate from the checks that only confirm the code runs, so a self-consistent misreading does not sail through on a wall of green.
The suite tells you the code works. It cannot tell you the code is right. That is still a person’s job.