Guide 06 · 1,000 concurrent, stress-tested

Run evals and RL rollouts at scale

Benchmark 500 model outputs, roll out RL episodes, grade a class of solutions — each in its own clean environment. The multi-node scheduler bin-packs sandboxes across the fleet; the history API gives you the exact cost of every single run.

import concurrent.futures as cf
from pilldrops import Sandbox

def grade(solution):
    with Sandbox.create(template="python-3.11", timeout=300) as sb:
        sb.files.write("/work/sol.py", solution)
        sb.upload("./tests", "/work/tests")
        r = sb.run("cd /work && python -m pytest tests -q", timeout=120)
        return r.exit_code == 0

with cf.ThreadPoolExecutor(max_workers=100) as ex:
    results = list(ex.map(grade, solutions))

$ pilldrops history --limit 500 --json | jq '[.[].cost_usd] | add'
# total dollars for the whole eval sweep — per-run detail included