APPS#

APPS is a set of Python programming problems, from introductory exercises to competition tasks. Agents write a solution, and the runner counts it as solved only if it passes every test it runs.

The task#

The runner frames each problem the way the APPS paper and common harnesses do: QUESTION:, the problem text, a format directive, then ANSWER:. The directive depends on the problem:

• Standard input: Use Standard Input format. The program reads stdin and prints the answer.
• Call-based: the starter code is shown, followed by Use Call-Based format. The solution defines the named function.

Agents can call python_exec(code, stdin), which runs a snippet in a fresh Python subprocess with a 10-second timeout and returns stdout, stderr and the exit code.

The answering agent must end with one fenced python code block holding the submitted solution. The runner takes the last fenced block that parses as Python, preferring blocks labelled python or py over bare fences.

How it is scored#

The runner runs the extracted code against the problem's tests, with the APPS reference timeout of 4 seconds per test. Each test passes if the output matches through a cascade of checks:

• Standard input: stripped output equal; else equal line by line; else all numeric tokens close (numpy.allclose, relative 1e-5, absolute 1e-6).
• Call-based: return value equal; else numerically close; else equal as sets; else equal as sets of floats rounded to 3 places. These tests run in a subprocess with a reliability guard and a memory cap (APPS_CALL_BASED_MEMORY_BYTES, 4 GB by default).

An instance scores 1 only if every test passes (APPS strict accuracy, reported as pass@1); otherwise 0. The batch prints the score over all instances, over instances with extracted code, and per difficulty tier. Independent and Decentralized teams submit their agents' most common program, and only that program is tested.

Data#

The runner loads the Hugging Face dataset codeparrot/apps, split test. Each row's ID is its problem_id as a string, and its tests come from the input_output field. Rows without usable tests are skipped.

APPS problems can have hundreds of tests, so the runner keeps the first 20 per problem by default. Change this with --max-tests-per-row; a value of 0 or less keeps every test. The cap changes the score, so keep it fixed when you compare runs.

The 50 eval IDs are problem IDs 0 to 49, all interview-level, listed in benchmarks/apps/apps_eval_ids.json. They are the first 50 rows, so --limit 50 scores exactly that set.

Run it#

Run every command from the repository root. With no arguments, the single-agent runner runs two built-in problems, one standard-input and one call-based, and prints the extracted code and test results:

python -m topologies.single.apps.langgraph_apps

A batch needs --batch:

python -m topologies.single.apps.langgraph_apps --batch --limit 50 \
  --out-dir results/topologies_baseline/single_apps

python -m topologies.centralized.autogen.apps.autogen_apps --batch --limit 50 \
  --out-dir results/topologies_baseline/centralized_autogen_apps

Optimize it#

Every optimizer runs through the same protocol command; change --method to switch. For example, MASPO on the Sequential pipeline:

python -m optimizers.protocol.run --method maspo --dataset apps --topology sequential \
  --model qwen --seed 0 --out runs/maspo/apps/sequential/qwen/0

Combinations outside the experiment grid need --allow-any-cell; see Optimize a task.

Flags#

Beyond the common flags (--batch, --limit, --offset, --only, --out-dir, --out), all eight APPS runners accept:

Related#

• LiveCodeBench and SWE-bench Verified, the other coding tasks.
• Workflow Topologies for what each runner variant does.
• Evaluation Protocol for how the eval IDs and splits are used.