Density Labs / Blog /Rails AI

Fail closed: circuit breakers for LLM features in Rails

The runaway invoice is the most preventable AI incident, and most Rails teams have no breaker for it.

Ask a team what could go wrong with their new AI feature and they list wrong answers. The incident that actually gets an engineering leader summoned to finance is different: the feature worked, in a loop, all weekend.

A retry storm against a paid API, an agent stuck re-asking the same question, a batch job that embedded the same corpus nightly. The unit economics of LLM calls turn ordinary bugs into invoices with an extra zero or two. This class of incident is near fully preventable with two boring mechanisms.

A budget breaker, enforced in code:

class LlmBudget
  def self.spend!(tokens:, cost:)
    today = Ledger.today
    raise BudgetExceeded if today.cost + cost > DAILY_CEILING
    today.increment!(tokens: tokens, cost: cost)
  end
end

Every model call goes through the ledger. When the ceiling trips, the feature degrades to its non AI fallback, and a human gets paged during business hours instead of a bill at month end. Set the ceiling at three times a normal day, not thirty.

Fail closed on anything that acts. When the model call errors, times out, or returns something your validator rejects, an AI feature that only drafts can fail open and show nothing. An AI feature that acts, sends, charges, or changes records must fail closed: no valid decision, no action, escalate to a human. The queue backing up is a Tuesday problem. The agent acting on garbage is a lawyer problem.

Neither mechanism needs new infrastructure. A table, a ceiling, a raise, and an escalation path you would build for any payment integration.

This is also the difference between a demo and a production system, which is the whole subject of measuring your AI with evals, tomorrow.