API-Bank#

API-Bank shows the agents a dialogue between a user and an assistant that calls APIs, and asks for the next API call. The runner executes that call with API-Bank's own API code and checks the result against the recorded one.

The CLI and folder name for this task is apibank.

The task#

Each instance is a dialogue cut just before an API call. The prompt holds the dialogue so far, with the inputs and outputs of earlier API calls, plus API descriptions that depend on the level:

Agents do not execute APIs while solving. Each agent turn is one chat completion, and the answer must be exactly one call in brackets with keyword arguments, in the form [ApiName(arg='value')]. That is the API-Bank output contract. The runner takes the last bracketed call that parses. The Agents SDK runner also gives its peers the APIs as function tools.

How it is scored#

1. Parse the call. A missing call, positional arguments or bad syntax scores 0.
2. Match the name. The API name must equal the gold API name.
3. Execute and compare. The runner loads the API's class from the vendored API-Bank source (Level 3 uses its lv3_apis set), replays the earlier API calls in the dialogue, runs the predicted call, and compares its result with the gold result using that API's own check_api_call_correctness.

ToolSearcher calls use a separate scorer, set by --toolsearcher-scorer:

The official scorer uses sentence-transformers, which is in the environment; it loads sentence-transformers/paraphrase-MiniLM-L3-v2 on CPU by default.

Each line of results.jsonl has correct, the failing stage and error, and the predicted call. Accuracy is the fraction of correct lines. Independent and Decentralized teams submit their agents' most common call, ties going to the earliest agent.

Data#

The API-Bank source from AlibabaResearch/DAMO-ConvAI ships in the repository at benchmarks/apibank/apibank_upstream/, under its own license, so nothing needs downloading. Set APIBANK_ROOT to use another checkout.

With the default level, all, the runner reads the eval manifest benchmarks/apibank/apibank_eval_ids.json directly: 100 IDs, 33 from Level 1, 33 from Level 2 and 34 from Level 3. So --limit 100 scores exactly the eval set. The optimizers' train and validation splits come from the 445-task curated pool in benchmarks/apibank/apibank_pool_ids.json.

For a single level, the runner builds a task list on the fly: the first tasks whose gold call replays correctly, skipping those that need SearchEngine or Translate, up to 100 for Levels 1 and 2 or 245 for Level 3.

Run it#

Run every command from the repository root. To check the data and the gold-call replay without calling the model, use --summary. It covers the first --limit instances:

python -m topologies.single.apibank.langgraph_apibank --summary --limit 100

There is no smoke demo; with no arguments a runner solves the first 2 instances, and --batch changes nothing.

python -m topologies.single.apibank.langgraph_apibank --limit 100 \
  --out-dir results/topologies_baseline/single_apibank

python -m topologies.centralized.langgraph.apibank.langgraph_apibank --limit 100 \
  --out-dir results/topologies_baseline/centralized_langgraph_apibank

The runner writes predictions.jsonl and results.jsonl to --out-dir, emptying both first, and one JSON trace per instance under traces/. Without --out-dir, output goes to results/apibank/<style>/. API-Bank also has communication-protocol and team-size runners.

Optimize it#

Every optimizer runs through the same protocol command; change --method to switch. For example, MASPO on the Decentralized debate:

python -m optimizers.protocol.run --method maspo --dataset apibank --topology decentralized \
  --model qwen --seed 0 --out runs/maspo/apibank/decentralized/qwen/0

Combinations outside the experiment grid need --allow-any-cell; see Optimize a task.

Flags#

Beyond the common flags (--batch, --limit, --offset, --only, --out-dir, --out):

--limit defaults to 2.

Related#

• BFCL and ToolHop, the other tool-calling tasks.
• Workflow Topologies for what each runner variant does.
• Evaluation Protocol for how the eval IDs and splits are used.