Optimizers#

MAS-PromptBench optimizes the system prompts of a multi-agent system while the model weights stay frozen. Eight methods share one run protocol and one bridge to the real topology runners, so every method spends the same budget, is selected by the same rule and is scored on the same held-out rows.

What gets optimized#

A multi-agent system \( M \) is a topology plus a set of roles: a planner, a solver, a manager, a debater and so on. Each role has a system prompt, and the seed prompts live in configs/prompts/<topology>/<dataset>/<role>.txt. A method treats the joint set \( \pi = (\pi_1, \dots, \pi_K) \) of role prompts as the only variable; the model, the topology wiring, the tools and the scorer stay fixed. MIPRO's few-shot demos are rendered into the role prompts, so they are part of \( \pi \) too.

The quantity of interest is the prompt-optimization gain on the held-out test split:

Here \( \pi^0 \) is the seed prompt set, \( \pi^{\text{dep}} \) the deployed set, \( \mu \) the task metric and \( (x, y) \) a test instance with its reference answer. The deployed set is the method's incumbent only if it beats the seeds on validation; otherwise it is \( \pi^0 \) and the gain is zero. A job's result.json reports the estimate as delta_pp, in percentage points, and aggregation averages it over the three optimizer seeds.

No method writes to configs/. Optimized prompts stay in the job's --out folder.

The eight methods#

A grid cell is one (dataset, topology, framework, communication format, team size, task model) configuration of the experiment grid in optimizers/protocol/cells.py, run once per optimizer seed:

• GEPA and MIPRO cover every dataset on all five topologies, in each topology's native framework and in LangGraph, plus the communication formats and team sizes on HotpotQA, LiveCodeBench and BFCL, and those three datasets with meta-llama/Llama-3.1-8B-Instruct.
• MAPRO and MASPO cover the same cells except single.
• HiveMind, MAMUT-GEPA, MASPOB and TAVO cover HotpotQA, LiveCodeBench and BFCL on the four multi-agent LangGraph topologies.

The built-in identity method returns the seed prompts without a rollout; it runs on any grid configuration and gives the seed baseline of a cell. Any method runs on any registered pair with --allow-any-cell, as a non-conformant job.

One protocol for every method#

Every method runs as a job of the protocol mas-promptbench, python -m optimizers.protocol.run:

1. Optimize. The method gets the fixed, ordered train and validation rows, the frozen seed bundle and a ledger of 600 usable full-system rollouts, sampled at temperature 0.2, top-p 0.9 and up to 32,768 tokens. It returns its incumbent bundle.
2. Final validation. Uncharged and greedy: the seed and the incumbent run on the full validation split with paired per-item request seeds. The incumbent is deployed only if it is strictly better; a tie or a regression keeps the seeds. The decision is sealed in selection.json.
3. Test. After the lock, the seed and deployed bundles run on the held-out test split with the same paired seeds.

The methods differ only in step 1. Their reflection or proposal calls go to Qwen/Qwen3.5-122B-A10B-FP8 with thinking on and up to 48,000 output tokens, at temperature and top-p 1.0 unless the method sets them. Evaluation Protocol covers budgets, seeds and scorers in detail.

The real-runner bridge#

Every rollout runs through optimizers/bridge/, which wraps the topology runners instead of re-implementing them, so an optimized prompt runs unchanged in the benchmark.

1. Adapter (adapters/). One class per (dataset, registry key) pair exposes the per-role prompts (roles(), get_prompt(), set_prompt()) and run_example(), one execution of the real runner. It patches the runner module's prompt loader, model client, endpoint and team size in a private copy of the module.
2. Registry (registry.py). Maps each dataset and key, such as sequential_crewai or centralized_r8, to its adapter class.
3. Programs (programs.py, mipro_programs.py). Register one DSPy predictor per role, so GEPA and MIPRO edit the role instructions through named_predictors(). The other methods pass their candidates to the protocol's runner, directly or through its RunnerSession, which runs the same adapter.
4. Metric (datasets/<dataset>.py). load_all() and metric(), which returns a 0 or 1 score and feedback text for reflection.

The bridge also re-attaches what a method cannot edit: the protected final-output contract of the answering roles and a few format nudges. See Output contracts.

Method settings#

Each method's knobs are one frozen dataclass; the protocol fixes everything else. Only MAPRO and TAVO also read environment variables.

The run protocol README keeps the full table. MASPOB also needs torch, torch_geometric and sentence-transformers, which environment.yml installs; a CPU is enough.

Next steps#