AI release gates for LLM-built apps
When AI models write most of the code and several people and agents push to the same branch, “the tests passed” isn't enough. These are the four gates every Quantlys Conclave release has to clear, what they caught, and what still broke.
Fast code, shared branch
Quantlys Conclave is a notarized Mac app in which a council of AI models plans an app, then writes, verifies and deploys it to the user's own GitHub, Vercel and Supabase accounts. Its code is split across three repositories (api, app and cloud), and the api suite alone runs about 9,800 tests.
Not everyone writing that code is a person. AI coding lanes run on two Macs, and remote human teammates ship to the same main branch. That's fast, but it creates a specific risk: a change can be correct where it was written and still break the build that actually ships. And because Conclave deploys into users' own accounts, a bad release doesn't stay on my machine.
Four gates, in order
- Full tests, every time. The whole suite runs on every release candidate, not only the tests near the change. When models write code, side effects often land far from the edit.
- Rehearsal on the exact main. The release is rehearsed on the exact commit that will ship, not on a branch that looked the same yesterday. If main moves, the rehearsal runs again.
- Flags-off parity. New work ships behind feature flags, and with the flags off the release has to behave like the previous one. Unfinished features can't change anything for current users.
- Rollback before ship. The rollback is tested before the release goes out, not written after something breaks. If I can't go back in one step, it doesn't ship.
Around the gates sit a few habits that never get skipped: no secrets in code or git, test payments capped at $1, and production database changes only after a backup and a passing run on a copy.
Six releases, and the runs that failed
On November 3 and 4, 2026, six notarized releases went out, 0.6.104 through 0.6.109, each with a verified rollback. The next one, 0.6.110, was in QA when this was written.
The gates didn't make quality automatic. Five training runs, T1 to T5, scored 0 out of 5 twice before the fixes landed. The tests were green; the outcome wasn't. Quality caught up, but later than it should have.
Two other things broke the flow. Teammates sometimes shipped to main without posting first, which invalidated a rehearsal and forced a re-cut. And six releases in two days was too heavy a cadence, because every release pays for every gate. Before launch, I moved to bundling changes into fewer releases.
The same lesson showed up in another product. In Tablerene, an AI phone host for restaurants, code was merged but database migrations hadn't been applied to the live database, and the owner's admin page returned a 404. The code gate passed. The deploy gate didn't exist yet.
What I'd keep, and what I'd change
- Passing tests are a floor, not proof. Add an outcome check: a real run scored against what the user asked for.
- Rehearse what ships, not what you meant to ship. “The exact main” is the whole point.
- A rollback you haven't run is only a hope.
- Gates cost time, so bundle. Let risk set the release cadence, not enthusiasm.
- Code and data ship separately. Check the live database, not just the merge.
If AI writes your code
Put your rigor at the release boundary rather than in line-by-line review. Decide the gates once, write them down, and apply the same gates to human and AI changes. Ask everyone, people and agents alike, to post before they push to main. And keep one rule above the rest: no release without a rollback you've already run.
The gate that matters most is the one that lets you undo.