LLM Agent Orchestration: Decisions That Actually Matter in Production

The demo works. Then you ship it.

When I started building the agent layer for CVChatly, I had a clear picture in my head: a set of specialised agents, each handling a slice of the job-search workflow — CV analysis, job matching, interview coaching, feedback generation — coordinated by an orchestrator that would route tasks, manage context, and keep everything coherent. Clean. Modular. Elegant.

Two months into production, I had a system that worked, but not for the reasons I expected. The elegant parts caused the most problems. The unglamorous decisions — how to pass context between agents, when to retry, how to handle a tool call that returns nothing useful — were what actually determined whether users got a good experience or a spinner that eventually timed out.

This article is about those decisions. Not the theory of agent orchestration, which is well covered elsewhere. The specific choices I had to make, why I made them, and what I would do differently.

The first real decision: orchestration pattern

Before you write a single line of agent code, you need to decide how your agents will relate to each other. There are two broad patterns in practice, and they have very different implications.

Sequential pipelines pass outputs from one agent to the next in a defined order. Agent A analyses the CV, Agent B compares it to a job description, Agent C generates a gap report. Predictable, debuggable, easy to test. The failure mode is brittleness: if Agent B gets a malformed output from Agent A, the whole pipeline stalls.

Dynamic orchestration uses a central orchestrator — itself an LLM call — that decides at runtime which agent to invoke next based on what has happened so far. More flexible. Much harder to reason about. The orchestrator can get into loops, make surprising routing decisions, and when something goes wrong, the trace is often a wall of nested tool calls that is genuinely difficult to read.

For CVChatly I started with dynamic orchestration because it felt more powerful. I moved most critical paths back to sequential pipelines after the first month. Dynamic orchestration is now reserved for genuinely open-ended tasks where the user's intent is ambiguous. For anything where I know the steps in advance, a pipeline is more reliable, cheaper to run, and far easier to debug at 11pm when something breaks.

The lesson: start with the simplest pattern that solves the problem. Add dynamism only where you can justify the complexity cost.

Context management is where most agent systems quietly break

Each LLM call is stateless. Your agents do not remember anything between invocations unless you explicitly pass that information to them. This sounds obvious, but the practical implications are easy to underestimate.

In a multi-agent system, context has to travel. Every time you hand off from one agent to another, you are making a decision about what to include in the next prompt. Include too little and the receiving agent lacks the information it needs. Include too much and you are paying for tokens that add noise and increase the chance of the model losing focus on what actually matters.

I went through three iterations of context management before settling on something that works reliably. The approach that failed first was passing the full conversation history at every step. It was expensive and the agents started producing outputs that were distracted by irrelevant earlier exchanges. The approach that failed second was summarising context with another LLM call before each handoff. Accurate summaries, but the latency was brutal and the cost added up fast.

What works for me now is a structured context object — a typed schema in TypeScript — that each agent reads from and writes to. It contains only the fields that are relevant to the current task: the user's CV in a normalised form, the job description being evaluated, any scores or flags set by previous agents, and a short reasoning trace. Each agent gets exactly what it needs. Nothing more.

If you are building an agent system and context management feels like an afterthought, treat it as a first-class architectural concern. Define your context schema before you write your first agent prompt. You will refactor it anyway, but starting with a schema forces you to think about what information actually needs to flow between steps.

Tool design: the part nobody talks about enough

Agents become useful when they can take actions beyond generating text: calling an API, querying a database, running a calculation, retrieving a document. These capabilities are exposed as tools — functions the model can choose to call — and the quality of your tool design has an outsized impact on how reliably your agents behave.

The most common mistake I see, and made myself, is writing tools that do too much. A tool that fetches a job posting, parses it, extracts skills, and scores them against a CV is not a tool. It is a pipeline disguised as a function call. When it fails — and it will fail, because it is doing four things — you will not know which step caused the problem.

Good tools are narrow. They do one thing, return a clear and predictable schema, and fail loudly with a useful error message when something goes wrong. The model calling the tool needs to be able to understand the output without additional interpretation. If you find yourself writing prompt instructions that explain how to parse a tool's return value, the tool's output schema is wrong.

A few specific things that improved tool reliability in my system:

• Explicit error fields in the return schema. Instead of throwing exceptions that the orchestrator has to catch, tools return a structured object that always includes a success flag and an error message when applicable. The model can read this and decide what to do next.
• Idempotent tool calls where possible. If a tool can be called twice with the same input and produce the same result without side effects, retrying on failure is safe. Design for this from the start.
• Logging every tool call with inputs and outputs. Not just errors — every call. This is the only way to debug agent behaviour that surprised you in production. Without full tool call logs, you are guessing.

Retry logic, fallbacks, and knowing when to stop

LLM outputs are non-deterministic. Tools fail. APIs return unexpected responses. In a single-agent system this is manageable. In a multi-agent system, failures compound. An agent that retries three times, each time calling two tools that also retry, can generate a lot of cost and latency before it gives up or produces a result.

You need explicit policies for three things: when to retry, when to fall back to a simpler path, and when to stop and surface an error to the user rather than continuing to burn tokens on a task that is not going to resolve cleanly.

My current approach: agents retry a failed tool call once, with a short wait. If the second call fails, the agent returns a structured failure result rather than retrying again. The orchestrator — or the pipeline step — decides whether to attempt a fallback path or terminate the task. Tasks that have been running for longer than a defined threshold are terminated and flagged for review rather than allowed to continue indefinitely.

This sounds conservative, and it is. A task that fails fast and clearly is better than one that runs for thirty seconds and then produces a confused or partially correct output. Users can handle a clear failure message. They cannot handle an answer that looks plausible but is wrong.

Setting sensible timeouts and retry budgets is also how you keep costs predictable. An agent system without explicit termination conditions is an agent system with unpredictable costs. I covered cost control in more detail in a previous article, but retry logic is one of the highest-leverage places to apply it.

What I would do differently from day one

Looking back at the first two months of building the CVChatly agent layer, here is what I would change if I were starting again:

1. Define the context schema first. Before any agent prompt, before any tool, write out what information needs to exist at each stage of the workflow and what format it takes. This single artefact forces every other decision to be more intentional.
2. Build the logging infrastructure before the first agent goes to production. I added proper tracing after the fact and it was painful. Full tool call logs and structured agent traces are not a nice-to-have. They are the only way to understand what your system is doing.
3. Start with sequential pipelines and add dynamism only when you can justify it. Dynamic orchestration is genuinely useful in the right places. It is not the default right answer.
4. Treat tool design as seriously as prompt design. A well-designed tool with a clear schema and explicit error handling will save you more debugging time than a perfectly crafted system prompt.

Agent orchestration is not magic. It is software architecture with a probabilistic component. The fundamentals — clear interfaces, explicit failure modes, observable behaviour — apply exactly as they do in any other distributed system. The difference is that when you ignore them in an LLM agent system, the failures are stranger and harder to trace than in a conventional API. That is not a reason to add more AI. It is a reason to be more disciplined about the engineering around it.