Prompt-optimization gains using a state-of-the-art optimizer MIPRO in single-agent and multi-agent settings.
While MIPRO consistently improves single-agent performance across all five diverse tasks, its natural
multi-agent extension yields highly variable effects across tasks and workflow topologies, ranging from
large gains to severe performance drops.
How much can prompt optimization help in MAS, and how does its effect vary across configurations?
Contributions
1
MAS-PromptBench: a benchmark for MAS prompt optimization
A comprehensive benchmark for evaluating system-prompt optimizers for MAS. It spans diverse MAS
configurations across three task domains (reasoning, coding, and tool calling), five workflow topologies
(both existing and newly constructed systems), communication protocols from free-form to highly structured
coordination, and varying team sizes. It also evaluates eight prompt optimizers — two that naturally
extend state-of-the-art single-agent methods and six designed specifically for MAS — providing a
foundation for proposing, analyzing, and comparing system-prompt optimization algorithms under controlled
MAS configurations.
2
Prompt optimization gains and failures for MAS
Systematically evaluating the gains over default system prompts achieved by four representative optimizers:
MAS-GEPA and MAS-MIPRO, built on GEPA and MIPRO, and the MAS-specific MAPRO and MASPO. The results
highlight the promise of prompt optimization for MAS: improvements reach up to +56.0
points. Yet they also reveal the need for principled algorithms tailored to multi-agent settings, as
performance can drop by as much as −12.0 points for certain configurations.
3
Insights into when prompt optimization works for MAS
Prompt optimization shows greater potential when tasks have explicit, controllable, and verifiable
agent-local behaviors, and when communication protocols adopt a free form that leaves agent messages open
to revision; it also needs to be workflow-topology-aware. Optimization becomes harder as team size grows
— confirming the challenges of scaling MAS prompt optimization and motivating principled, more
scalable, and robust algorithms.
Benchmark at a Glance
MAS-PromptBench
varies each configuration axis independently while holding the others fixed.
Overview of benchmark MAS-PromptBench. Given an input task, a multi-agent system produces a final solution
through interactions among LLM-based agents. Using state-of-the-art prompt optimizers, MAS-PromptBench
measures prompt-optimization gains across four axes: task distribution, workflow topology, communication
protocol, and team size.
We model a MAS as a tuple $\mathcal{M} = (\mathcal{A}, G, P)$: an ordered collection of $n$ agents
$\mathcal{A}$, an inter-agent coordination workflow $G$, and a communication protocol $P$. Each agent
$A_i = (\theta_i, \pi_i)$ pairs a frozen LLM with parameters $\theta_i$ and a learnable system prompt $\pi_i$.
System-prompt optimization maximizes the expected task score
$\mathbb{E}_{(x,e)\sim\mathcal{T}}[\mu(\mathcal{M}(x;\pi), e)]$ over the joint prompts $\pi$ under a budget of
$B$ MAS rollouts, where $e$ is the evaluation reference (e.g., a ground-truth answer or unit tests). Fixing an
optimizer and the base model, for a configuration $(\mathcal{T}, G, n, P)$ we measure the
prompt-optimization gain $\Delta$ as the difference between the optimized ($\pi^\star$) and initial
($\pi^0$) system scores:
$$\Delta(\mathcal{T}, G, n, P) := \mathbb{E}_{(x,e)\sim\mathcal{T}}\big[\,\mu(\mathcal{M}(x;\pi^\star), e) - \mu(\mathcal{M}(x;\pi^0), e)\,\big]$$
Workflow Topologies
Single
Sequential
Independent
Centralized
Decentralized
LLM AgentExpertiseTaskSolutionCommunication
The five workflow topologies in MAS-PromptBench. Arrows indicate message flow.
Single. A single LLM serves as the baseline.
Independent. $n$ agents solve the task in parallel without inter-agent communication, and their outputs are aggregated by majority vote.
Sequential. Agents form a directed chain $A_1 \to A_2 \to \cdots \to A_n$ with no backward edges; each agent receives the previous agent's output as input toward the final answer.
Centralized. A coordinator dispatches subtasks to sub-agents $A_1, \ldots, A_n$, collects their outputs, and aggregates them into the final answer; sub-agents do not communicate with one another.
Decentralized. All $n$ agents communicate over a fully connected graph and exchange messages for a fixed number of rounds; final-round outputs are aggregated by majority vote for question answering or best-of-$N$ test-pass for coding.
Findings
When does a better local prompt become a better system? We vary one MAS factor at a time and measure the prompt-optimization gains of MAS-GEPA, MAS-MIPRO, MAPRO, and MASPO.
Factor 1 · Task
Explicit, verifiable tasks gain more
System-prompt optimization is broadly promising: averaged over workflows, the best optimizer per workflow
improves all nine tasks, with the largest average gain of +10.4 points on SWE-Bench.
Individual configurations gain even more — with MASPO, Decentralized improves by
+22.3 on API-Bank and +20.5 on ToolHop, and Centralized by
+20.0 on SWE-Bench. Gains are larger for coding and tool-calling (average best-optimizer
gains of +5.9 and +5.5 points) than for reasoning
(+2.6), where the peak is only +11.3 on Independent HotpotQA.
Prompt-optimization gains for nine datasets on existing MAS frameworks; select an optimizer above. Each cell
reports baseline / optimized performance, followed by the signed change $\Delta$ in percentage points with its
standard deviation across three optimizer seeds. Blue indicates improvement,
orange indicates regression, and gray indicates no change.
Avg. Max: mean best-optimizer gain over topologies. MAPRO and MASPO are not run on Single.
We hypothesize that the difference comes from how explicitly a task decomposes into controllable local
behaviors. Coding tasks expose verifiable artifacts that compilation and tests can check, and tool-calling tasks
offer structured interfaces — explicit function names and output formats — that system prompts can
directly shape, so local prompt improvements survive downstream coordination. Reasoning tasks rely on
correlated logical steps with implicit intermediate feedback, so local improvements are often discarded,
overwritten, or amplified before reaching the final answer.
Takeaway. Prompt optimization shows greater potential on tasks with explicit,
controllable, and verifiable agent-local behaviors, such as coding and tool-calling, than on reasoning tasks.
Factor 2 · Workflow Topology
Multi-agent systems need topology-aware optimizers
Prompt-optimization gain Δ (pp) of each optimizer across workflow topologies, averaged over nine tasks.
MAPRO and MASPO are not run on Single.
No evaluated optimizer transfers uniformly across topologies. MAS-GEPA gains +6.0 on
Sequential but only +0.2 on Centralized, and MAS-MIPRO gains +7.5
on Sequential but only +1.2 on Independent and Decentralized. The best optimizer also
changes with the topology: MAS-GEPA leads on Independent, MAS-MIPRO on Sequential and Centralized, and MASPO on
Decentralized. Sequential yields the largest gains (+4.8 on average across the
four optimizers), suggesting that an improved prompt early in the chain propagates to every downstream agent;
Independent and Decentralized yield the smallest (+1.2 and
+1.0), as aggregating parallel agent outputs may absorb much of the effect of prompt revisions.
Full results by topology
Prompt-optimization gains of MAS-GEPA, MAS-MIPRO, MAPRO, and MASPO for five workflow topologies on nine tasks. Each cell reports baseline / optimized performance, followed by the signed change $\Delta$ in percentage points with its standard deviation across three optimizer seeds.
Takeaway. Multi-agent systems need topology-aware prompt optimizers.
Factor 3 · Communication Protocol
Freeform messages leave more room to optimize
We compare three protocols with increasing structure: Freeform natural-language messages with no required
fields; Semi-structured messages with prescribed slots (status, evidence, confidence, next step) filled in
natural language; and Structured JSON-style reports whose slot values follow a constrained, finite set of options.
Prompt-optimization gain Δ (pp) by communication protocol, averaged over HotpotQA, LiveCodeBench, and
BFCL and four MAS topologies. Freeform yields the largest gain for every optimizer.
Averaged over MAS-GEPA, MAS-MIPRO, and MASPO, the gain is +4.9 points under Freeform
messages, compared with +2.4 under Semi-structured and +3.0
under Structured messages. The advantage is largest for MAS-MIPRO, where Freeform (+5.3)
is about twice Semi-structured (+2.7) and Structured (+2.6).
Freeform messages let the optimizer revise both the content and the form of what each agent passes downstream,
whereas a fixed template already determines much of each message, leaving less for prompt optimization to change.
Full results by communication protocol
Prompt-optimization gains across communication protocols on HotpotQA, LiveCodeBench, and BFCL. Each cell reports baseline / optimized performance, followed by the signed change $\Delta$ in percentage points with its standard deviation across three optimizer seeds.
Takeaway. Freeform communication leaves agent messages open to revision, giving MAS
prompt optimization more room to improve than protocols that fix message structure in advance.
Factor 4 · Team Size
Larger teams make optimization harder
Prompt-optimization gain Δ (pp) across team sizes, averaged over HotpotQA, LiveCodeBench, and BFCL and
four MAS topologies. The dashed line is the mean across the three optimizers.
As teams grow ($n \in \{4, 8, 10\}$), gains generally shrink: averaged over MAS-GEPA, MAS-MIPRO, and
MASPO, they fall from +4.9 points at $n{=}4$ to +2.6 at
$n{=}8$ and remain at +2.6 at $n{=}10$. Larger teams enable finer-grained
specialization but also add handoffs and intermediate states, making local improvements harder to preserve. The
effect is clearest for MAS-GEPA (+5.1 → +2.4 →
+1.7), while MAS-MIPRO retains most of its gain (+5.3,
+4.3, +4.0), so the effect of team size also depends on the optimizer.
Full results by team size
Prompt-optimization gains across team sizes on HotpotQA, LiveCodeBench, and BFCL. Each cell reports baseline / optimized performance, followed by the signed change $\Delta$ in percentage points with its standard deviation across three optimizer seeds.
Takeaway. Larger team size increases the challenge of prompt optimization for MAS,
as local agent improvements may fail to produce system-level gains.
Ablation · Optimizers & Task Models
The topology effect holds across optimizers and task models
We repeat the workflow-topology study on HotpotQA, LiveCodeBench, and BFCL with four more MAS prompt optimizers:
HiveMind, MAMUT-GEPA, MASPOB, and TRUCE. The Sequential topology yields the largest gain for six of the eight
optimizers (+6.7 points on average), whereas Independent and Decentralized yield the
smallest (+1.4 and +1.2). The best optimizer again changes
with the topology: MAS-GEPA leads on Single and Independent, MAS-MIPRO on Sequential and Centralized, and TRUCE
on Decentralized.
Single
Independent
Sequential
Centralized
Decentralized
MAS-GEPA
+5.8
+3.8
+12.0
+4.6
+0.1
MAS-MIPRO
+0.4
0.0
+13.0
+6.3
+1.9
HiveMind
–
+0.2
+1.2
+1.0
−3.0
MAPRO
–
0.0
+0.6
+3.7
+0.4
MAMUT-GEPA
–
+1.9
+9.0
+3.8
+2.6
MASPO
–
+1.8
+8.8
+4.8
+1.9
MASPOB
–
+2.4
+6.5
+1.3
+1.0
TRUCE
–
+0.9
+2.3
−0.7
+4.3
Mean
+3.1
+1.4
+6.7
+3.1
+1.2
Prompt-optimization gain Δ (pp) of eight prompt optimizers across workflow topologies, averaged over
HotpotQA, LiveCodeBench, and BFCL. Darker cells mark larger gains; – marks optimizers not run on Single.
Replacing the task model Qwen3.5-9B with Llama-3.1-8B-Instruct, while keeping the optimizers, tasks, MAS
configurations, and reflection model unchanged, changes which topology benefits most: Sequential is the most
improved topology for Qwen3.5-9B (+8.6) but gains only +1.4
for Llama-3.1-8B-Instruct, which instead improves most on Independent (+13.6) and
Centralized (+12.8). Decentralized remains the least improved for both
(+1.1 and +0.5).
Qwen3.5-9B
Llama-3.1-8B-Instruct
Prompt-optimization gain Δ (pp) with Qwen3.5-9B and Llama-3.1-8B-Instruct as the task model, averaged
over HotpotQA, LiveCodeBench, and BFCL and over MAS-GEPA, MAS-MIPRO, MAPRO, and MASPO (Single: MAS-GEPA and
MAS-MIPRO only).
Full results for eight optimizers
Prompt-optimization gains of eight prompt optimizers for five workflow topologies on HotpotQA, LiveCodeBench, and BFCL. Each cell reports baseline / optimized performance, followed by the signed change $\Delta$ in percentage points with its standard deviation across three optimizer seeds. Single is run only with MAS-GEPA and MAS-MIPRO.
Full results by task model
Prompt-optimization gains with Qwen3.5-9B and Llama-3.1-8B-Instruct as the task model on HotpotQA, LiveCodeBench, and BFCL. Each cell reports baseline / optimized performance, followed by the signed change $\Delta$ in percentage points with its standard deviation across three optimizer seeds. MAPRO and MASPO are not run on Single.
Takeaway. The dependence of gains on topology persists across optimizers and task
models, but the topology that benefits most varies with the task model — topology-aware prompt optimizers
should also account for the task model.
BibTeX
@article{bai2026mas,
title={MAS-PromptBench: When Does Prompt Optimization Improve Multi-Agent LLM Systems?},
author={Bai, Juyang and Shi, Laixi},
journal={arXiv preprint arXiv:2606.23664},
year={2026}
}