Computer Use
An agent that looks at the screen and works the mouse and keyboard
An agent loop in which a model looks at a screenshot and proposes mouse and keyboard actions such as clicking, typing and scrolling, while your application carries them out on a desktop or in a browser and returns the next screenshot.
Classic automation talks to an API: you say "create this record" and the service returns JSON. But many applications have no API, or none that is open to you. Computer use means the model works the graphical interface the way a person does: it looks at the screen, decides where to click, types and scrolls.
The model never touches the screen itself. The loop goes like this:
- Your application sends the model the task and a screenshot.
- The model returns the next action:
left_click [x, y],type "...",key "ctrl+s",scrolland so on. - Your application performs that action in a virtual machine, container or browser it controls and sends back a fresh screenshot.
- The loop continues until the task is done, an error occurs or a confirmation is needed.
As of November 2026, all three major providers ship an API tool for
this: Anthropic's computer_toolset_20260801 toolset (17 member tools
for screenshots, mouse and keyboard), the computer tool in OpenAI's
Responses API, and the computer_use tool in the Gemini API (browser,
mobile and desktop environments). In every case your application
runs the environment that executes the actions.
Screenshots are not the only way. For tasks that stay inside a
browser, Anthropic's browser use tool and OpenAI's BrowserToolset
can also read the page through the accessibility tree and element
references, which is usually sturdier and cheaper than guessing
coordinates from pixels.
Think of a remote support call. The other person shares their screen, you look and say "press the blue Save button at the top right", they press it, you see the new screen and say the next step.
In computer use the model gives the instructions and the code you wrote presses the button. The difference is that the model reads every piece of text on the screen, including malicious text on a web page. So the side that presses the button has to be able to say "this is a payment, let me ask the user first."
An accounting team downloads invoices every month from an old supplier portal that has no API, and files them in a folder. They hand this to a computer use agent.
The agent runs in a Linux VM whose network access is limited to the portal's domain. The model sees the login screen; it is not given the username and password, so the team logs in themselves and leaves the session open. The model clicks the "Invoices" tab, sets the date filter and downloads the PDFs one by one. When a page shows a banner saying "Set up automatic payment", the application ignores it: payment and form-submission actions require human approval in code. A screenshot of every step goes to the log.
curl https://api.anthropic.com/v1/messages \
-H "content-type: application/json" \
-H "x-api-key: $ANTHROPIC_API_KEY" \
-H "anthropic-version: 2026-10-21" \
-d '{
"model": "claude-opus-5-5",
"max_tokens": 1024,
"tools": [{"type": "computer_toolset_20260801"}],
"messages": [{"role": "user", "content": "Save a picture of a cat to my desktop."}]
}'
# Run the tool_use blocks in the response (screenshot, left_click,
# type...) in your own VM and send them back as tool_result blocks.from google import genai
client = genai.Client()
interaction = client.interactions.create(
model="gemini-3.8-flash",
input="Search for 'Gemini API' on Google.",
tools=[{"type": "computer_use", "environment": "browser"}],
)
# Coordinates come back normalized to 0-1000;
# scale them to your screen size and run the action yourself.from openai import OpenAI
client = OpenAI()
response = client.responses.create(
model="gpt-6.1-sol",
tools=[{"type": "computer"}],
input="If the Filters panel is closed, click Show filters, then type penguin in the search box.",
)
print(response.output)- Legacy web portals and desktop applications that have no API, or whose API you can't access
- End-to-end testing of user flows in web apps (form filling, sign-up, the steps before checkout)
- Frequent but low-risk workflows that require copy-and-paste between several applications
- Research and data-gathering tasks where a person watches alongside and approves every critical step
- When the job has an official API, CLI or MCP server; that path is faster, cheaper and more reliable
- On your own computer, with your real accounts and no isolation; use a separate VM, container or clean browser profile
- Unattended, unconfirmed use for irreversible work (money transfers, bulk deletes, accepting contracts)
- Jobs that must repeat thousands of times within seconds; every step is a model call plus a screenshot
Treating page text as instructions
The model reads every piece of text on the screen. A hidden instruction in a web page, email or image may be followed even when it conflicts with yours (prompt injection). Anthropic and Google offer classifiers that scan screenshots, but the providers themselves say this is not complete protection. Treat screen content as untrusted data.
Running the agent in your own session
If the agent can reach your open email, bank or admin sessions, an injection can turn into real damage. Use a separate VM or a clean browser profile, restrict network access to allowed domains and don't hand passwords to the model.
Leaving critical actions unconfirmed
Tie hard-to-reverse steps such as purchases, sending messages, submitting forms, deleting data and accepting terms to an approval gate in code. Relying only on the model's 'ask for confirmation' signal (Gemini's safety_decision, OpenAI's pending safety checks) is not enough on its own.
Getting the coordinate scale wrong
The coordinates the model returns are in the pixel space of the screenshot you sent (normalized to 0-1000 in Gemini). If you downscaled the image or are on a Retina display and don't scale the coordinates back, clicks drift consistently in one direction.
Trusting the agent's 'done'
The model can claim the task is finished and be wrong. Verify the outcome from the final state of the environment (the downloaded file, the saved form, the database record), not from the agent's answer.