Ulyssify blocks distracting apps and websites. It has an assistant, Anka, that sets blocks up for you in plain language. Last week it failed in a very instructive way:

The site had landed in a different list with different behavior. The assistant confirmed a thing that never happened, then guessed when challenged.

The tempting diagnosis is "the model is not smart enough." We traced the transcript instead and found something better: no model at any size could have answered correctly. The facts were never in its context. They got lost in two places.

Leak one: the agent's view of your data goes stale

Our Settings screens never fall behind, because they render whatever the backend returns. The assistant's view was different: a summary function someone wrote by hand, listing a few fields, on the day the feature shipped. When the product later grew schedules with real hours, nobody updated that summary, and nothing forced anyone to.

So the assistant could not see that a midnight schedule already existed, and it proposed rebuilding things the user already had. Not stupidity. Blindness.

The fix: parity is a test, not a promise

We stopped trusting anyone to remember. A CI test now reads the real fields the UI renders and demands that every one of them is either shown to the assistant or excluded with a written reason. Add a field to the UI without deciding for the assistant, and the build goes red with a message telling you which move to make.

Leak two: the assistant was narrating its plan, not the result

The wrong-list bug was worse. The server had executed the action and stored exactly what happened, including which list the site really landed in. But the assistant's follow-up message was composed from its own earlier plan, plus a generic "executed successfully." The stored truth was thrown away one step before the model saw it.

Now the follow-up is built from the stored result: the real list, the real state, the real ids. If something lands in an unexpected place, the assistant says so, because that is what it was told. And nothing from the request can inject into that message; every fact is server-generated.

The part that was never broken: two doors, one gate

One property saved us from this being a dangerous story instead of an embarrassing one, and it is the piece of our design we would defend hardest. The assistant does not get its own API. Its actions enter the backend one layer below HTTP and run the exact same gated functions a human click runs.

This matters because of who the adversary is. In a commitment device, the person most motivated to defeat a block is the user themselves, at 1 a.m., negotiating with their own past decisions. If the assistant had its own write path, every prompt would be a potential loophole. Because it shares the human path, anything that loosens enforcement queues behind the user's own cooldown, no matter how the request was phrased. The assistant could be wrong. It could not be a bypass.

Note what this is not: it is not "ask a human before acting," which is well covered elsewhere. It is a stronger claim about plumbing. "The assistant can do everything a human can" should mean capability parity, not a parallel API surface, because a human cannot skip the cooldown either.

The approval half of this is well covered in the field: 12-Factor Agents makes the case for keeping a human in the loop, and Anthropic's guide to writing tools for agents covers returning meaningful, high-signal results to the model. The plumbing claims, that the agent's write path IS the human's gated path and that its confirmations are rebuilt from stored server truth, are the part we have not seen written up, so we are writing it up.

We closed the loop by turning the incident into a permanent test: the exact configuration from that conversation is now a fixture, and CI asserts the assistant's context contains every fact needed to get it right.