We build Ulyssify with a fleet of AI coding agents. On a busy day the fleet opens and lands dozens of pull requests across our backend, web, iOS, macOS, Windows, and Android surfaces. Humans direct and approve; the agents do the implementation. This post is about the part nobody warns you about: once your agents can do almost anything, the hard problem stops being capability and becomes judgment.
The failure mode of a capable agent
A junior engineer who can technically do anything, with no sense of which actions are reversible, is not an asset. Agents are the same. An agent that will happily refactor a file will, given the chance, also force-push a shared branch, run a destructive migration against real data, or deploy to production because the tests were green. Green tests mean nothing failed. They do not mean the change was safe to ship. So the first thing we built was not a better agent. It was a way to tell the agent which decisions are its own to make.
Three tiers: AUTO, NOTIFY, ASK
Every recurring decision our agents face is sorted into one of three tiers.
AUTO is for anything reversible with a small blast radius: branch off main, run the tests, open a pull request. The agent just does it.
NOTIFY is for actions that are recoverable but worth seeing: the agent does the thing, then lists it in its run summary so a human can veto a wrong call. The rule for NOTIFY is strict: it only qualifies if the undo is one cheap command. If the undo is fuzzy or expensive, it was never NOTIFY.
ASK is for the one-way doors: promoting to production, a destructive migration, anything that touches money or enforcement or a public commitment. The agent stops and waits for a person.
Two rules sit above all three tiers. One-way doors always ask. And when the agent is not highly confident, it asks. A clarifying question is cheaper than a confident wrong build, every time.
The finding that changed how we write rules
We keep our agents' operating rules as written instructions, and for a while we just kept adding more. Then we audited a month of the fleet's actual behavior, and the result was uncomfortable: rules backed by a real mechanism, a git hook that refuses the commit, a CI job that blocks the merge, a permission gate, had a violation rate of essentially zero. Rules that lived only as prose were followed about three quarters of the time, and adherence got worse as we added more prose, not better.
The lesson we took: instruction-following is a budget, not a guarantee. Every rule you write as prose competes with every other rule for the model's attention, and the pile always grows. So a rule you truly never want violated does not belong in the prose at all. It belongs in a mechanism that makes the wrong action impossible. We now treat promoting a never-violate rule from an instruction into a hook or a gate as the real fix, and adding another paragraph as the thing to avoid.
Verify the mechanism, not the pedigree
One more habit that saves us repeatedly. When an agent, or a human, recommends a safeguard, it is tempting to trust it because it is well-reasoned and has history behind it. But a recommendation is a claim, and the only question that matters is whether the mechanism actually holds. Our one-line test for any protection: is the value it depends on known to the party enforcing it, or is it volunteered by the party it is meant to constrain? A limit the constrained party can dodge by simply staying quiet is not a protection, however good its reasoning. Finding the line of code that enforces it settles the question faster than any amount of debate about whether the idea is sound.
None of this is finished. It is a living system, and we get it wrong in new ways every week. But the direction has been consistent: give the agents real judgment about which decisions are theirs, and put the decisions that matter most behind mechanisms rather than words. We will share more of how the fleet runs as we go.