Orchestrating specialised agents: choosing, synthesising, showing what is going on

The first version of an agent is one model with a handful of tools, and it works. Around the tenth tool it degrades: the model picks its tools less reliably, the context fills up with schemas that are useless for the question at hand, and when the answer is wrong there is no way to tell which part failed.

The usual next step is a set of narrow agents, each good in its own domain, behind an orchestrator. That is what I did with Chimera, the super-agent in my Quant project. The direction is right, and it brings four problems that are better known before you build.

The loop

1. Choose before calling

The naive orchestrator queries every agent and lets synthesis sort things out. It is simple, and it fails on three counts at once: latency is that of the slowest agent, cost is the sum of all agents, and answer quality drops as off-topic findings drown the good ones.

Selection can do without a model. In most systems the question carries enough signal (the entities it names, its vocabulary, the screen it comes from) for a light classifier or a few explicit rules. In Chimera, selection works on domain, intent and keywords, with no LLM call. When a model really helps at this step, I have it return a short ranked list, so the orchestrator applies its own budget.

Two properties to plan for from the start.

Selection you can read back. I log which agents were considered, which were picked, and why. "The orchestrator did not ask the agent that had the answer" is a common failure, and without that log it cannot be diagnosed.

Selection you can force. A caller who knows which agent it wants must be able to ask for it. Routing remains a heuristic, and a heuristic needs an escape hatch.

The maths, intuitively: the slowest one decides

When n agents are queried in parallel, the answer only arrives once the slowest has replied: orchestrator latency is a maximum. Cost, on the other hand, is a sum: every agent called gets paid for.

The maximum has an unpleasant property. Suppose each agent is slow one time in 20, meaning it exceeds its p95 in 5% of cases, independently of the others. For the answer to be fast, no agent may be slow, which happens with probability 0.95 × 0.95 × … , that is 0.95^n. Hence:

P(at least one slow agent) = 1 − 0.95^n

With 3 agents, one answer in 7 waits for a slow agent (14%). With 12 agents, almost one in two (46%). At 12 agents, each agent's p95 becomes something that happens all the time. Choosing before calling therefore cuts the bill, which is a sum, and the perceived latency, which is a maximum. Bounded parallelism and a per-agent timeout then trim whatever tail is left.

2. Structured findings out of each agent

The decision that holds the architecture together: an agent returns a structured finding. A free-text paragraph is unusable for synthesis.

from dataclasses import dataclass
from typing import Literal

@dataclass(frozen=True)
class Finding:
    agent: str                     # who produced it
    kind: Literal["fact", "anomaly", "absence"]
    statement: str                 # a single claim, in natural language
    confidence: float              # class="tok-num">0..class="tok-num">1, calibrated by the agent itself
    evidence: list[str]            # IDs that resolve to a source
    computed_at: str               # ISO-class="tok-num">8601 — synthesis needs the age
    inputs_digest: str             # what the agent looked at, for reproducibility

Each field has a reason to be there.

kind="absence" is a real result. "I looked and found nothing" is different information from "nobody asked me" and from "I crashed". Without that distinction, synthesis cannot tell silence from a negative answer, and it guesses.

evidence holds IDs. An ID can be resolved, checked and shown to the user. A quote copied into the finding is only a claim about the source, and it survives even after the source is corrected.

confidence comes from the agent. The confidence a model assigns to itself means almost nothing. Confidence computed by the agent's own logic (how many rows agree, whether a threshold was crossed, how much data was missing) is usable. When an agent has no way to calibrate itself, I would rather leave the field empty than make up a number.

3. Synthesis is a job of its own

Concatenating findings or asking the model to "summarise all this" is not enough. Synthesis has its own ways of failing.

Contradictory findings get shown. Two agents disagreeing is information, often the most useful of the whole run. An orchestrator that quietly keeps the most confident finding destroys it.

Age gets passed on. A finding computed six hours ago and one computed thirty seconds ago do not carry the same weight, and users should see how old what they are told is.

Abstaining stays possible. If the selected agents returned nothing usable, the right answer is "I don't know, here is what I looked at", with the list. It is the hardest behaviour to keep, because any fluent answer looks better in a demo.

The synthesis prompt must not invent evidence. I ground it strictly in the findings passed in, then check that every evidence ID it cites was actually in its input. The check fits in a few lines and catches the error that destroys trust fastest.

In Chimera I also handled the case where the LLM is offline: the orchestrator still returns the agents' real findings, with a flag saying synthesis could not run.

4. Freshness, and when to rerun an agent

Agents produce findings on a schedule, and a question arrives at any time. So the orchestrator has to decide, agent by agent, whether the existing finding is still good or the agent needs to run again.

The rule I use: a freshness budget per agent, set from how fast its source data really changes. Anything derived from a nightly job gets a one-day budget, anything derived from a live feed, a few minutes. A single global default suits nobody.

Rerunning agents is where an orchestrator becomes dangerous. So I frame it explicitly:

• On request. Answering with dated data, and saying so, is safer than starting work nobody asked for.
• Bounded. A hard cap on how many agents can be rerun per question. In Chimera it is 5. Without a cap, one request can turn into a load nobody sized.
• Asynchronous. The user gets the answer right away, marked as dated, and the refresh serves the next question.
• Authorised. Triggering compute is a privileged action. In my setup it is admin-only and off by default.

5. Make the state readable

A system that wakes several agents, waits for some, skips others and sometimes refuses to answer is opaque from the outside. Two seconds of silence and then a paragraph: users cannot tell whether it is thinking, stuck, or had nothing.

A live status display belongs here: a progress indicator, a status line, an animation tied to what the system is doing. In Chimera it is a 3D orb that changes with the state (working, thinking, anomaly, restitution). The form matters less than the engineering rule behind it.

I never display a state I do not measure. An indicator driven by a timer is a decoration that lies under load, at the very moment users most need to know what is happening. If a state cannot be observed, it does not appear.

Degraded and offline are states too. The display has to be able to show "an agent is unreachable" and "answer built from partial data". When Chimera runs on its local fallback, a DEMO banner says so clearly. A visualisation that can only look healthy turns an outage into confusion.

What it costs

This architecture has a price, and it is predictable:

• More moving parts. Each agent is a deployment, a schedule, a failure mode and a permission.
• Selection errors become the most common failure. In a system that has been running for a while, a bad answer comes more often from "we asked the wrong agents" than from "the agent was wrong". Hence the need to log selection.
• Latency is that of the slowest selected agent. Bounded parallelism and per-agent timeouts are required.
• Findings need versions. A schema change on one kind of finding propagates to synthesis and to everything that stored one.

It is worth it when the domains are really distinct, each needing its own tools, its own data access and its own permissions. With three tools and a single data source, a single agent remains the right answer.