Evals in a Rails codebase: three layers, no new platform
Shape tests, golden transcripts, and an LLM judge. RSpec can run all three.
How do you test a feature whose output is different every time? This question stalls more Rails AI work than any technical blocker, and the answer fits in the test suite you already have.
Layer 1: shape tests. Deterministic, fast, run on every commit. The model output parses, references real records, contains no forbidden claims, fits length limits. These are ordinary RSpec tests around your validator, and they catch the failure mode that matters most: the confident fabrication.
Layer 2: golden transcripts. Twenty to fifty real inputs from production with a known good output, checked into the repo like fixtures. A rake task runs the current prompt and model against all of them and reports pass, near miss, or fail. This is your regression suite for prompt changes. Edited a prompt? The golden run tells you whether category B tickets got worse, in numbers, before your users do.
Layer 3: the LLM judge. For qualities a regex cannot check (tone, helpfulness, whether the reply actually answers the question), a second model call grades the first against a rubric. Judges drift and have taste, so calibrate: have the judge grade ten outputs a human already graded, and check agreement before trusting it in CI.
The workflow that makes this compound: every production bug becomes a permanent eval. The weird ticket that produced the embarrassing reply goes into the golden set the same day. Six months in, your eval suite encodes more institutional knowledge about your domain than any prompt does.
None of this needs an eval platform. A fixtures directory, a rake task, a results table.
The pass rate this machinery produces is also the number that unlocks stakeholder trust, which is why the AI Readiness Assessment builds a golden set for your top workflow in week one. The number does the convincing.