RL & EVAL ENVIRONMENTS

Disposable RL environments, a thousand at a time.

Fan out one sandbox per rollout, run the episode, collect the reward, tear it all down. Each environment is an isolated microVM — real filesystem, real processes, zero cross-talk — and PillDrops is stress-tested to 1,000 concurrent.

Start free — $5 credits Read the evals guide

Stress-tested to 1,000 concurrent sandboxes · per-second billing · E2B_API_KEY works too.

fan out N rollouts, collect reward

# pip install pilldrops
from pilldrops import Sandbox
from concurrent.futures import ThreadPoolExecutor

def rollout(policy):
    with Sandbox.create(template="code-interpreter") as sb:
        sb.upload("./env", "/env")
        r = sb.run(f"python /env/episode.py --policy {policy}")
        return float(r.stdout)      # reward for this episode

with ThreadPoolExecutor(max_workers=64) as pool:
    rewards = list(pool.map(rollout, policies))
print(sum(rewards) / len(rewards))  # 0.8123

// npm i @pilldrops/sdk
import { Sandbox } from '@pilldrops/sdk'

async function rollout(policy: string) {
  const sb = await Sandbox.create({ template: 'code-interpreter' })
  await sb.upload('./env', '/env')
  const r = await sb.run(`python /env/episode.py --policy ${policy}`)
  await sb.kill()
  return parseFloat(r.stdout)     // reward
}

const rewards = await Promise.all(policies.map(rollout))

1,000

concurrent sandboxes (tested)

per-episode

cost attribution

pause/resume

waiting env = storage-only

warm-pool

fast boots for fan-out

$5 free

no card required

[01] — Fan out to 1,000 / one sandbox per rollout

A fresh environment for every episode, torn down after.

Each rollout gets its own microVM, so environments never leak state into one another and a crashed episode can't poison the batch. Spin up the whole harness at once — the scheduler places sandboxes across the fleet and reclaims them when the run ends.

⬢

1,000 concurrent

Stress-tested to 1,000 live sandboxes at once — enough to run a full eval suite or an RL batch as a single fan-out.

◇

Best-fit scheduler

A best-fit, multi-node scheduler packs environments onto the fleet by available vCPU and memory — no hot nodes, no wasted headroom.

↻

Crash-safe adoption

If a host restarts mid-run, live sandboxes are re-adopted rather than orphaned — your reward collection keeps going.

◷

Warm pools

Pre-warmed pools cut cold-start latency, so a thousand-way fan-out reaches the first step fast instead of waiting on boots.

Set vcpu, memory_mb and timeout per environment so heavy episodes get room and cheap ones stay cheap.
Sessions run up to 24h — long enough for multi-turn agent evals and slow rollouts.
Self-hostable when the environment or the model can't leave your network.

[02] — Cost per episode / know what each rollout costs

Attribute cost to every rollout, not just the month.

Each sandbox is billed per-second on vCPU and RAM, and every run lands in your usage history keyed by sandbox id. Pull the history back to attribute cost per episode — and pause any environment that's waiting on a model turn so it bills storage-only until you resume.

per-episode cost + park an idle env

# each sandbox id is one episode — pull its usage back
$ pilldrops history --json
[{"id":"sb_7a1","vcpu_s":42.1,"gb_s":88.5,"cost_usd":0.00099}, ...]

# ...or over REST
$ curl -H "Authorization: Bearer $PILLDROPS_API_KEY" \
    https://pilldrops.com/v1/history

# env is waiting on a model turn? park it — storage-only billing
$ pilldrops sandbox pause sb_7a1
$ pilldrops sandbox resume sb_7a1   # pick up exactly where it left off

◷

Per-second, per-sandbox

Billed at $0.0000140/vCPU-s and $0.0000045/GB-s. Because every episode is its own sandbox, its cost is its own line in GET /v1/history.

⏸

Pause the waiters

An environment blocked on a slow policy or a human step bills storage-only at $0.15/GB-month while paused — then resume restores its full state.

[03] — Wire it into your harness / create, run, reward, kill

An eval harness sandbox in four calls.

The whole environment lifecycle is four SDK calls — create, upload, run, kill — which drops cleanly into any RL loop or eval runner. The full evals guide walks through grading, scoring and collecting rewards at scale.

STEP 01

Create the environment

Sandbox.create() boots a fresh microVM per rollout — or use it as a context manager so teardown is automatic.

isolatedmicroVM

STEP 02

Load env & policy

sb.upload() pushes your environment code and the policy under test; setup steps install anything the image lacks.

uploadreproducible

STEP 03

Run the episode

sb.run() executes the rollout and returns stdout, stderr and exit_code — parse the reward straight out of stdout.

rewardstdout

STEP 04

Tear it down

sb.kill() (or the with block exit) reclaims the sandbox the instant the episode ends — you only paid for the seconds it ran.

disposableper-second

Evals & scoring guide →API docs →

RUN THE BATCH

Fan out your first thousand rollouts.

Spin up disposable RL and eval environments in parallel, collect the reward, tear them down — with per-episode cost in your history. Starts on $5 of free credits, no card.

Start free Read the evals guide