Tutorial and reference
MAS-PromptBench Docs#
When does prompt optimization improve a multi-agent LLM system? These docs show you how to run the benchmark, optimize its system prompts with eight optimizers under one run protocol, and measure the gain they produce.
Start here#
- Overview The model of a multi-agent system, the factors it varies and the gain \( \Delta \).
- Installation Create the conda environment and the isolated Agents SDK install.
- Connect a Model Serve Qwen3.5-9B with vLLM and point the runners and optimizers at it.
- Quick Start Run a baseline, optimize its prompts and read the result.
The benchmark#
Each factor is varied on its own while the others stay at their defaults.
- Tasks GPQA-Diamond, HotpotQA, MATH, LiveCodeBench, APPS, SWE-bench Verified, BFCL, ToolHop and API-Bank.
- Workflow Topologies Single, Independent, Sequential, Centralized and Decentralized.
- Communication Protocols Freeform, Semi-structured and Structured messages.
- Team Sizes The same topology with 2, 4, 8 and 10 agents.
- Optimizers GEPA, MIPRO, MAPRO, MASPO, HiveMind, MAMUT-GEPA, MASPOB and TAVO, run over the real topology runners.
Extend and look up#
- Add a Dataset Task module, team spec, runners, prompts, splits and optimizer adapter.
- Add a Topology A new coordination structure or a framework variant of an existing one.
- Command-Line Flags Every runner and run-protocol flag, with defaults.
- Environment Variables Every setting the runners and optimizers read, with defaults.