Computer Use Agents
When an app has no API, a computer use agent uses the screen instead. It looks, clicks the right pixel, types if needed, and checks what changed.
Why Does This Exist?
You want an agent to file an invoice in a desktop app from 1998. No API. No neat JSON endpoint. The vendor stopped updating it before you were hired, and the only door in is the same buttons a person presses.
That is the gap tool use and function calling can't cross. Tool calling works when a model can name a function and its arguments, and your code runs it. No function, no call. This page covers the other path: reading screenshots, naming coordinates, and clicking, one verified step at a time.
Two neighbours keep this page honest. Agent harness design decides when the loop stops and what a failed call means. Agent sandboxing decides what the agent may touch. Here you learn the pixel loop itself.
Think of It Like This
A guest cook on an unfamiliar stove
You've stepped into a friend's kitchen to boil pasta, and you've never used this stove. You look at the knobs, find the one marked front left, turn it, then check the flame. Blue and steady, you drop the pasta. Pale or silent, you picked the wrong knob and you look again.
Point for point: your eyes are the screenshot, the knob position is the coordinate, your hand is the mouse, and the glance at the flame is the verification screenshot. Don't trust the turn alone. Trust the turn plus what you see next.
Where it stops: knobs stay put and flames look like flames. Software moves. Popups cover buttons, two buttons share one label, and a small window shift moves every pixel. A cook smells gas. An agent does not, which is why it must recheck the screen instead of trusting memory.
How It Actually Works
A computer use agent repeats four moves: see, ground, act, check. The model never touches the OS. It writes down what to do, and plain code outside the model does it.
The see, ground, act, check loop
Call the goal text ("pay the invoice marked 1042"). Call the screenshot at step by , plain pixels wide and high. Call the past steps , the short list of what you tried and saw. The model is a policy , a rule that picks the next action from those three inputs: . In plain words: given the goal, the current screen, and the short history, the policy proposes one action.
An action is small and fixed: , , , , or . In plain words: click at a point, type a string, scroll by an amount, pause, or declare done. The ReAct pattern runs underneath: think, act through the OS, observe the new screen, repeat. Vision comes from vision-language models: the same weights that read a photo read your screenshot.
Coordinates are relative, not pixels
Pixels shift when the window resizes, so agents speak in relative units from to . Let be the horizontal pixel and the screen width in pixels. Let be the vertical pixel and the screen height. Then and . In plain words: divide by the size, stretch to a thousand, round. Back to pixels for the driver: . Same for . You lose at most half a thousandth to rounding, which is under a pixel on most screens, and you gain an address that survives resizing.
Worked example: Submit at 750, 750
Take a by screen. The Submit button sits centered at pixel . That is the fixture the diagram draws.
Across: . You get , times is .
Down: . You get , times is .
So the model emits . Plain code converts back: and , and the driver clicks there. The next screenshot must show the invoice marked paid. If it does not, the agent does not push ahead. It grounds again.
That is the trade, stated plain. Each step costs a full vision call, so ten steps cost ten reads plus ten writes to the log. You pay latency and tokens to reach software no API can touch. When a real API exists, skip this whole page and use tool use and function calling. Direct calls run faster, cost less, and break less often.
The assumption underneath: what you see is what you can click, and the screen holds still between see and click. Popups, lazy loading, or a resolution change break it. The click lands on the wrong pixel, still clicks something, and the log says success.
Code
The point of this snippet is the round trip the diagram draws: pixels to relative and back. It uses the fixture numbers, so you can check the arithmetic.
def to_normalized(px: float, size: float) -> int: """Map a pixel position to 0-1000 space.""" return int(round(1000 * px / size))
def from_normalized(n: int, size: float) -> float: """Map back to pixels for the click driver.""" return round(n * size / 1000, 1)
W: float = 1280.0H: float = 800.0x: int = to_normalized(960.0, W)y: int = to_normalized(600.0, H)print((x, y)) # -> (750, 750)print(from_normalized(750, W)) # -> 960.0print(from_normalized(750, H)) # -> 600.0Both directions land on the fixture: (750, 750) out, 960.0 and 600.0 back. If your driver disagrees by more than a pixel, your or is stale.
Watch Out For
Clicking the look-alike after the screen shifts
Symptom: the agent reports Submit clicked, but the log shows the same invoice screen again, or a different dialog open. A popup covered the button, the list scrolled, or two buttons share one label, so 750, 750 now points at empty space or the wrong control.
Fix: screenshot after every action and demand proof of change before the next step. Keep and fresh from the latest screenshot, ground again instead of replaying old coordinates, and stop after two misses with no change. Route destructive steps through human-in-the-loop approval, so a stray coordinate cannot pay twice.
Running computer use on the machine you care about
Symptom: a wrong coordinate opens your mail, selects a thread, and archives it, or types a password into a search box. Nothing crashes. The damage looks like you did it.
Fix: Never run this loop on your own laptop. Run only inside a disposable virtual machine with no secrets, no personal files, and closed network, exactly as agent sandboxing describes. Mount one work folder, destroy the machine after the run, and require approval before delete, send, or pay.
The Quick Version
- Computer use agents drive software through pixels when no API exists: screenshot in, coordinates and keystrokes out.
- Every step is see, ground, act, check: read screen , map goal to in 0 to 1000 space, run it outside the model, then re-screenshot.
- On a 1280 by 800 screen, pixel (960, 600) is (750, 750), because 1000 times 960 over 1280 is 750 and 1000 times 600 over 800 is 750.
- When a real API exists, use direct tool calls instead: they run faster, cost less, and break less often than vision plus clicks.
- Run only in a disposable sandbox and pause for approval before delete, send, or pay, because a wrong coordinate still clicks something.