Fourteen months, one refund resolution agent, a payments ledger it can write to, and an on-call rotation that includes us. This is what the graphs actually did.
What happened to the escalation rate?
The escalation rate started at 46% and settled at 29% after month five, then stayed flat. It did not trend to zero and we never expected it to — a stable escalation rate is the sign of a correctly bounded agent, not a ceiling on its capability.
The drop came almost entirely from two changes: widening the value ceiling from $80 to $250 once the trace record justified it, and teaching the agent to recognise three entity types it had previously punted on.
Did costs drift?
Cost per resolved ticket drifted up 19% over the first four months before we caught it, driven almost entirely by retrieval context growing as the knowledge base grew. Nothing about the agent changed; the world around it did.
Now cost per resolved task is a gated metric in CI and a monthly review item. Drift is normal. Undetected drift is the problem.
What broke?
Three incidents, all worth writing down.
The first was a malformed carrier webhook that put the agent into a retry loop; it burned $340 in ninety minutes before the spend ceiling — which we had set per run, not per hour — stopped it. We now cap spend on both axes.
The second was a model upgrade that improved every eval score and quietly changed the tone of customer-facing drafts enough that support noticed before our dashboards did. We added a tone dimension to the golden set the same week.
The third was a permissions change during an unrelated migration that widened a tool grant nobody reviewed. Nothing bad happened; it was luck. Capability grants are now code-reviewed like any other change, and diffs to them page a human.
How did model swaps go?
We swapped the underlying model three times in fourteen months, and each swap took under a day of work because the harness answered the only question that mattered. Two swaps improved the score and shipped; one regressed on escalation correctness and did not.
This is the practical argument for the eval harness that nobody makes up front: it is what makes your model choice reversible.
What would we do differently?
Cap spend per hour as well as per run from day one, and treat capability grants as reviewable code from the first commit rather than the first incident. Both were cheap to add and expensive to add late.
Is a supervised agent worth running?
Over fourteen months the agent resolved 71% of tickets without a human, held a hard value ceiling with zero policy breaches, and cost less to operate each quarter than the one before it. Autonomy is a dial, not a switch — and the dial stayed where the evidence put it.