Skip to content
AI360XpertBeta

Computer Use Agents

When an app has no API, a computer use agent uses the screen instead. It looks, clicks the right pixel, types if needed, and checks what changed.

A computer use agent turns a screenshot into button coordinates and a click, then checks the new screen
A computer use agent turns a screenshot into button coordinates and a click, then checks the new screen

Why Does This Exist?

You want an agent to file an invoice in a desktop app from 1998. No API. No neat JSON endpoint. The vendor stopped updating it before you were hired, and the only door in is the same buttons a person presses.

That is the gap tool use and function calling can't cross. Tool calling works when a model can name a function and its arguments, and your code runs it. No function, no call. This page covers the other path: reading screenshots, naming coordinates, and clicking, one verified step at a time.

Two neighbours keep this page honest. Agent harness design decides when the loop stops and what a failed call means. Agent sandboxing decides what the agent may touch. Here you learn the pixel loop itself.

Think of It Like This

A guest cook on an unfamiliar stove

You've stepped into a friend's kitchen to boil pasta, and you've never used this stove. You look at the knobs, find the one marked front left, turn it, then check the flame. Blue and steady, you drop the pasta. Pale or silent, you picked the wrong knob and you look again.

Point for point: your eyes are the screenshot, the knob position is the coordinate, your hand is the mouse, and the glance at the flame is the verification screenshot. Don't trust the turn alone. Trust the turn plus what you see next.

Where it stops: knobs stay put and flames look like flames. Software moves. Popups cover buttons, two buttons share one label, and a small window shift moves every pixel. A cook smells gas. An agent does not, which is why it must recheck the screen instead of trusting memory.

How It Actually Works

A computer use agent repeats four moves: see, ground, act, check. The model never touches the OS. It writes down what to do, and plain code outside the model does it.

The see, ground, act, check loop

Call the goal text gg ("pay the invoice marked 1042"). Call the screenshot at step tt by oto_t, plain pixels WW wide and HH high. Call the past steps hth_t, the short list of what you tried and saw. The model is a policy π\pi, a rule that picks the next action ata_t from those three inputs: at∼π(at∣g,ot,ht)a_t \sim \pi(a_t \mid g, o_t, h_t). In plain words: given the goal, the current screen, and the short history, the policy proposes one action.

An action is small and fixed: click(x,y)\text{click}(x, y), type(text)\text{type}(text), scroll(dx,dy)\text{scroll}(dx, dy), wait()\text{wait}(), or stop()\text{stop}(). In plain words: click at a point, type a string, scroll by an amount, pause, or declare done. The ReAct pattern runs underneath: think, act through the OS, observe the new screen, repeat. Vision comes from vision-language models: the same weights that read a photo read your screenshot.

Coordinates are relative, not pixels

Pixels shift when the window resizes, so agents speak in relative units from 00 to 10001000. Let xpxx_{px} be the horizontal pixel and WW the screen width in pixels. Let ypxy_{px} be the vertical pixel and HH the screen height. Then x=round(1000⋅xpx/W)x = \text{round}(1000 \cdot x_{px} / W) and y=round(1000⋅ypx/H)y = \text{round}(1000 \cdot y_{px} / H). In plain words: divide by the size, stretch to a thousand, round. Back to pixels for the driver: xpx=round(x⋅W/1000)x_{px} = \text{round}(x \cdot W / 1000). Same for yy. You lose at most half a thousandth to rounding, which is under a pixel on most screens, and you gain an address that survives resizing.

Worked example: Submit at 750, 750

Take a 12801280 by 800800 screen. The Submit button sits centered at pixel (960,600)(960, 600). That is the fixture the diagram draws.

Across: x=round(1000⋅960/1280)=round(750)=750x = \text{round}(1000 \cdot 960 / 1280) = \text{round}(750) = 750. You get 960/1280=0.75960 / 1280 = 0.75, times 10001000 is 750750.

Down: y=round(1000⋅600/800)=round(750)=750y = \text{round}(1000 \cdot 600 / 800) = \text{round}(750) = 750. You get 600/800=0.75600 / 800 = 0.75, times 10001000 is 750750.

So the model emits click(750,750)\text{click}(750, 750). Plain code converts back: 750⋅1280/1000=960.0750 \cdot 1280 / 1000 = 960.0 and 750⋅800/1000=600.0750 \cdot 800 / 1000 = 600.0, and the driver clicks there. The next screenshot must show the invoice marked paid. If it does not, the agent does not push ahead. It grounds again.

That is the trade, stated plain. Each step costs a full vision call, so ten steps cost ten reads plus ten writes to the log. You pay latency and tokens to reach software no API can touch. When a real API exists, skip this whole page and use tool use and function calling. Direct calls run faster, cost less, and break less often.

The assumption underneath: what you see is what you can click, and the screen holds still between see and click. Popups, lazy loading, or a resolution change break it. The click lands on the wrong pixel, still clicks something, and the log says success.

Code

The point of this snippet is the round trip the diagram draws: pixels to relative and back. It uses the fixture numbers, so you can check the arithmetic.

def to_normalized(px: float, size: float) -> int:    """Map a pixel position to 0-1000 space."""    return int(round(1000 * px / size))

def from_normalized(n: int, size: float) -> float:    """Map back to pixels for the click driver."""    return round(n * size / 1000, 1)

W: float = 1280.0H: float = 800.0x: int = to_normalized(960.0, W)y: int = to_normalized(600.0, H)print((x, y))  # -> (750, 750)print(from_normalized(750, W))  # -> 960.0print(from_normalized(750, H))  # -> 600.0

Both directions land on the fixture: (750, 750) out, 960.0 and 600.0 back. If your driver disagrees by more than a pixel, your WW or HH is stale.

Watch Out For

Clicking the look-alike after the screen shifts

Symptom: the agent reports Submit clicked, but the log shows the same invoice screen again, or a different dialog open. A popup covered the button, the list scrolled, or two buttons share one label, so 750, 750 now points at empty space or the wrong control.

Fix: screenshot after every action and demand proof of change before the next step. Keep WW and HH fresh from the latest screenshot, ground again instead of replaying old coordinates, and stop after two misses with no change. Route destructive steps through human-in-the-loop approval, so a stray coordinate cannot pay twice.

Running computer use on the machine you care about

Symptom: a wrong coordinate opens your mail, selects a thread, and archives it, or types a password into a search box. Nothing crashes. The damage looks like you did it.

Fix: Never run this loop on your own laptop. Run only inside a disposable virtual machine with no secrets, no personal files, and closed network, exactly as agent sandboxing describes. Mount one work folder, destroy the machine after the run, and require approval before delete, send, or pay.

The Quick Version

  • Computer use agents drive software through pixels when no API exists: screenshot in, coordinates and keystrokes out.
  • Every step is see, ground, act, check: read screen oto_t, map goal gg to click(x,y)\text{click}(x, y) in 0 to 1000 space, run it outside the model, then re-screenshot.
  • On a 1280 by 800 screen, pixel (960, 600) is (750, 750), because 1000 times 960 over 1280 is 750 and 1000 times 600 over 800 is 750.
  • When a real API exists, use direct tool calls instead: they run faster, cost less, and break less often than vision plus clicks.
  • Run only in a disposable sandbox and pause for approval before delete, send, or pay, because a wrong coordinate still clicks something.