Case study 01 · Release engineering

AI release gates for LLM-built apps

~9,800 api tests0.6.104–0.6.109Rollback before ship

When AI models write most of the code and several people and agents push to the same branch, “the tests passed” isn't enough. These are the four gates every Quantlys Conclave release has to clear, what they caught, and what still broke.

01 · The problem

Fast code, shared branch

Quantlys Conclave is a notarized Mac app in which a council of AI models plans an app, then writes, verifies and deploys it to the user's own GitHub, Vercel and Supabase accounts. Its code is split across three repositories (api, app and cloud), and the api suite alone runs about 9,800 tests.

Not everyone writing that code is a person. AI coding lanes run on two Macs, and remote human teammates ship to the same main branch. That's fast, but it creates a specific risk: a change can be correct where it was written and still break the build that actually ships. And because Conclave deploys into users' own accounts, a bad release doesn't stay on my machine.

02 · The approach

Four gates, in order

  1. Full tests, every time. The whole suite runs on every release candidate, not only the tests near the change. When models write code, side effects often land far from the edit.
  2. Rehearsal on the exact main. The release is rehearsed on the exact commit that will ship, not on a branch that looked the same yesterday. If main moves, the rehearsal runs again.
  3. Flags-off parity. New work ships behind feature flags, and with the flags off the release has to behave like the previous one. Unfinished features can't change anything for current users.
  4. Rollback before ship. The rollback is tested before the release goes out, not written after something breaks. If I can't go back in one step, it doesn't ship.

Around the gates sit a few habits that never get skipped: no secrets in code or git, test payments capped at $1, and production database changes only after a backup and a passing run on a copy.

03 · What happened

Six releases, and the runs that failed

On November 3 and 4, 2026, six notarized releases went out, 0.6.104 through 0.6.109, each with a verified rollback. The next one, 0.6.110, was in QA when this was written.

The gates didn't make quality automatic. Five training runs, T1 to T5, scored 0 out of 5 twice before the fixes landed. The tests were green; the outcome wasn't. Quality caught up, but later than it should have.

Two other things broke the flow. Teammates sometimes shipped to main without posting first, which invalidated a rehearsal and forced a re-cut. And six releases in two days was too heavy a cadence, because every release pays for every gate. Before launch, I moved to bundling changes into fewer releases.

The same lesson showed up in another product. In Tablerene, an AI phone host for restaurants, code was merged but database migrations hadn't been applied to the live database, and the owner's admin page returned a 404. The code gate passed. The deploy gate didn't exist yet.

04 · Lessons

What I'd keep, and what I'd change

  • Passing tests are a floor, not proof. Add an outcome check: a real run scored against what the user asked for.
  • Rehearse what ships, not what you meant to ship. “The exact main” is the whole point.
  • A rollback you haven't run is only a hope.
  • Gates cost time, so bundle. Let risk set the release cadence, not enthusiasm.
  • Code and data ship separately. Check the live database, not just the merge.
05 · For another team

If AI writes your code

Put your rigor at the release boundary rather than in line-by-line review. Decide the gates once, write them down, and apply the same gates to human and AI changes. Ask everyone, people and agents alike, to post before they push to main. And keep one rule above the rest: no release without a rollback you've already run.

The gate that matters most is the one that lets you undo.

About the author

Santosh Adari

Santosh Adari is Associate Director, IT Data Engineering at Gilead Sciences and the founder of Quantlys, which he builds on nights and weekends from Apex, North Carolina. Quantlys is a native app for Apple Silicon Macs; inside it, Conclave plans an app with a council of AI models, then builds it and deploys it to your own GitHub, Vercel and Supabase accounts.

About Santosh Adari · All case studies · Built with Quantlys · LinkedIn