Multi-agent LLM orchestration, explained
Multi-agent LLM orchestration means coordinating several language models or AI agents, each with a defined job, so they draft, critique, combine and verify work together instead of trusting one model's first answer. This page explains the pattern Santosh Adari builds on at Quantlys: five frontier models draft in parallel, rank each other blind, an arbiter synthesises the winner-weighted answer, and a build-and-verify layer runs the result before it ships.
What is LLM orchestration?
LLM orchestration is everything that happens around a language model call: deciding which model gets which prompt, chaining calls together, calling tools, carrying context from one step to the next, and checking the output before anyone relies on it. Almost every serious AI product does some of it, even when only one model is involved.
Multi-agent orchestration is a specific shape of LLM orchestration. Instead of one model doing everything, several agents or models take separate roles, hand work to each other, and check each other. The value comes from disagreement: a model that came at a problem differently is the one most likely to catch the mistake a lone model would have shipped.
The hard part is not getting several models to answer. It is deciding, fairly and auditably, which answer wins, and then proving the result actually works.
Four steps from
many models to one answer
All three live Quantlys products came out of one patent-pending multi-agent LLM orchestration engine. Its design council is called The Conclave, and it runs in four steps:
- Draft in parallel. Five frontier models by default (expandable to any number) each write an independent draft, blind to each other, so you get genuinely different attempts rather than one answer echoed five times.
- Peer-rank, blind. Every model critiques and ranks the other drafts. Drafts are shown anonymously (A to E in the default five-model run), and self-votes are struck.
- Arbiter synthesis. An arbiter merges the highest-ranked ideas into one winner-weighted plan and writes down why, so the decision is auditable.
- Build & verify. A crew of agents builds the app and a verify layer runs it before it ships, with 117 permanent checks across web, iOS and Android.
Why the ranking
has to be blind
The obvious way to get multi-model LLM consensus is to show every model all the answers and ask which is best. The problem is that models are not neutral judges of their own work.
Across 675 ranking ballots, models put their own work first. Consensus without blinding is partly self-promotion.
That self-preference is why the Quantlys engine anonymises drafts before peer-ranking and strikes any vote a model casts for its own draft. What is left is a ranking by peers that came at the problem differently, which is the signal worth having.
The same lesson applies to the evaluation itself. Santosh's own evaluator once had a bug that inverted a whole conclusion, which is why his rule is to measure blind, hold out the tests, and grade the grader. If you are building your own multi-agent system, a short checklist:
- Hide authorship before any model judges another model's output.
- Discard self-votes rather than trying to correct for them afterwards.
- Keep held-out tests the models never see, and check that the grader can fail.
- Log every ballot, so a surprising result can be traced to a cause.
The arbiter,
or chairman, pattern
Ranking tells you which drafts are strongest. It does not give you a single plan. The last reasoning step, often called chairman synthesis or a judge model, is where one model reads the peer-ranked drafts and writes the answer the system will act on.
In Quantlys this role is the Arbiter. It does not just pick the top-ranked draft. It merges the highest-ranked ideas into one winner-weighted plan, so a strong idea from a lower-ranked draft can still survive, and it records why each choice was made. That record matters more than it sounds: when a shipped product behaves oddly, you can see which decision caused it.
Compared with simple majority voting, arbiter synthesis keeps the benefit of several perspectives without averaging them into something bland, and it leaves an audit trail instead of a tally.
Orchestration that
ends in a working app
A consensus answer is only useful if it survives contact with a compiler and a real device. In Quantlys, the Conclave designs and a crew of agents builds. The stock crew is a PM, an architect, a developer and QA, and teams can compose their own agents with their own roles, tools and standards.
- Typed handoffs. Each stage emits a structured artifact the next stage validates, so when something breaks you know which boundary broke.
- Deterministic QA. Test suites run with model calls stubbed, so a passing run means the product behaves, not that a model agreed with itself that day.
- Real native builds. Native Swift for iOS and Android builds signed with your own keystore, under your own Apple and Google accounts.
- Your code, your host. Quantlys can host the apps it builds or deploy them to your own host, and it can import an existing code repository.
- Enterprise options. On-prem and hybrid deployments are available for teams that need audit trails and policy controls inside their own environment.
Where this pattern
runs today
Every product below was built with the same Conclave that Quantlys users get:
- GreenBook: a bilingual (English and Spanish), local-first CRM for solo landscapers that works with no signal, free on Google Play.
- Quantlys Meeting: browser video meetings where the working session writes a PRD instead of a recap; MIT licensed on GitHub.
- Tablerene: voice agents that answer a restaurant's phone during service, take to-go orders and book tables.
- ClipNow: live walk-in haircut wait times from local salons, hosted on Quantlys.
More on the products and the company: Santosh Adari, founder of Quantlys.
When multi-agent orchestration
is worth it
Running five models costs more and takes longer than running one. It pays off when a wrong answer is expensive and the output can be checked: application design, code, specifications, research and compliance work. For a quick lookup or a casual summary, a single well-prompted model is usually the better product decision.
The rule of thumb Santosh uses as a product owner: add orchestration where it removes a failure you have actually seen, and define done as what the user sees, not what the pipeline reports.
Multi-agent orchestration:
common questions
What is multi-agent LLM orchestration?
Multi-agent LLM orchestration is the practice of coordinating several language models or AI agents, each with a defined role, so that they draft, critique, combine and verify work together instead of trusting a single model's answer. In Quantlys, five frontier models draft in parallel, rank each other blind, and an arbiter synthesises the winner-weighted answer before a build-and-verify layer runs the result.
How is LLM orchestration different from multi-agent orchestration?
LLM orchestration is the broader term: routing prompts, chaining calls, calling tools and managing context around one or more models. Multi-agent orchestration is a specific shape of it, where several agents or models with separate roles hand work to each other and check each other, with structured handoffs between stages.
Why should models rank each other blind?
Because models favour their own work. Across 675 ranking ballots in Santosh Adari's own evaluation, models put their own answer first. If drafts are labelled by author, consensus is partly self-promotion, so Quantlys anonymises the drafts and strikes self-votes before ranking is counted.
What does the arbiter or chairman model do?
The arbiter, sometimes called a chairman or judge model, reads the peer-ranked drafts and merges the highest-ranked ideas into one winner-weighted plan, and records why it chose what it chose, so the decision can be audited later.
Who is Santosh Adari?
Santosh Adari is an AI Product Owner at Gilead Sciences in Commercial Data & Analytics, and the founder of Quantlys (quantlys.ai), an AI app builder and LLM orchestration platform. He is based in Apex, North Carolina.
Is Quantlys the same as Quantly or Quantalys?
No. Quantlys, spelled with an s, is the AI app builder and LLM orchestration platform at quantlys.ai, founded by Santosh Adari and operated by Agility Business Services, Inc. It is not affiliated with similarly named companies such as Quantly or Quantalys.