[02] — Drive it from your agent loop / screenshot → decide → act
Close the perceive-act loop.
The pattern is simple: capture the screen, let your model decide, then send a click or keystroke back into the desktop. Below is the shape — the full loop, including how to wire screenshots into a vision model, lives in the guide.
agent_loop.py
from pilldrops import Sandbox
with Sandbox.create(template="desktop", ports=[5900]) as sb:
print(sb.url) # watch the desktop live over VNC
# 1. perceive — grab the current screen
sb.run("scrot /tmp/screen.png")
img = sb.read("/tmp/screen.png") # feed to your vision model
# 2. decide — your model returns an action (click/type)
# action = model.decide(img) -> {"x": 420, "y": 300}
# 3. act — send the input back into the desktop
sb.run("xdotool mousemove 420 300 click 1")
sb.run("xdotool type 'hello from the agent'")
Wiring screenshots into a model, handling coordinates, and running the loop to completion is covered end to end in the computer-use guide.
desktop templateVNC previewxdotoolscrotvision agents