Computer Use and Browser Agents: How Screen-Reading AI Works and When It's Safe to Use
How computer use agents work: the screenshot → model → mouse/keyboard loop, pixels vs the accessibility tree; the API tools and in-browser agents from Anthropic, OpenAI and Google as of November 2026; choosing between API and GUI automation; what OSWorld and WebArena measure; prompt injection, VM isolation and confirmation gates; where it works well and badly; and a minimal safe setup.
Updated:
Summary
An AI agent usually works through tools, meaning APIs. But much of the work people do has no API: an old supplier portal, a desktop accounting program, an admin panel you can only reach in a browser. Computer use means the model works these interfaces the way a person does: it looks at the screen, decides where to click and types.
This guide covers how these agents work technically, what each provider offers as of November 2026, when to choose GUI automation over an API, what benchmarks such as OSWorld and WebArena measure and, most importantly, how to run these agents safely in a world where a web page can give your agent instructions.
Note: Everything here was checked on November 8, 2026 against the providers' official documentation (Anthropic, OpenAI, Google), the OSWorld and WebArena papers and OSWorld's official results table. This field moves very fast: model names, tool versions and products can change within months. Check the page in the sources section once more before relying on a detail.
How does a screen-reading agent work?
The model never touches the screen. Computer use is an agent loop with two sides: the model decides, your application (the executor) carries out the action.
- Your application sends the model the task, the tool definition and a screenshot of the environment.
- The model reads the screen and returns an action:
left_clickwith coordinates,typewith text,keywith a key combination,scroll,waitand so on. - Your application performs that action in a virtual machine, container or browser that it controls.
- It sends the new screenshot back to the model as the result.
- The loop continues until the task is done, an error occurs, the step limit is reached or human approval is needed.
Technically this is a special case of function calling: the tool is "the screen" and the actions are mouse and keyboard. What makes it possible on the model side is that multimodal models can read buttons, text and positions in a screenshot.
The providers' docs differ on a few practical details:
- Coordinate space: With Anthropic, the model returns coordinates in the pixel space of the screenshot you sent; if you downscaled it, you need to scale the coordinates back. Gemini returns coordinates normalized to 0-1000, and you convert them to your screen size.
- Batched actions: In Anthropic's
computer_toolset_20260801toolset, the model can request several actions in one turn; your application runs them in order and stops at the first failure. - Zoom: The same toolset's
zoommember lets you send a region of the screen at full resolution, for reading small text.
Pixels or the accessibility tree?
An agent can "see" the screen in two ways:
| Pixels (screenshots) | Accessibility tree / DOM | |
|---|---|---|
| What it sees | An image of the screen | The page's structure: buttons, fields, labels, element references |
| Where it works | Any application: desktop, browser, mobile | Mainly the browser (and apps that expose accessibility data) |
| How it clicks | By guessing coordinates | By element reference (such as e12) |
| Cost | Image tokens on every step | Usually less; page text |
| Weak spot | Small elements, scaling errors, scrolling | Poorly marked-up pages, canvas-based interfaces |
In November 2026 the two approaches are used together. According to its docs, Anthropic's browser use tool (browser_toolset_20260801) works with the page both through its structure (accessibility tree, elements, forms, tabs) and through screenshots. OpenAI's docs likewise recommend a BrowserToolset that reads page text and controls for tasks that stay in the browser, and a separate ComputerToolset for pixels and mouse/keyboard. A practical rule: if you stay in the browser, read the structure; if you need to leave it for the desktop, switch to pixels.
Who offers what as of November 2026?
What follows is based on the providers' own documentation. What a provider says about its product's performance is not an independent measurement; we flag that wherever it comes up.
API tools for developers
Anthropic: the computer use toolset. Added to the Messages API as {"type": "computer_toolset_20260801"}, with no beta header. According to the docs, Claude 5.5 and later models support computer use through this toolset, which provides 17 member tools such as screenshot, zoom, left_click, type, key and scroll. The older computer_20251124 and computer_20250124 versions remain for earlier models behind beta headers. Your environment executes the actions; Anthropic publishes a Docker-based reference implementation. The toolset definition adds about 4,500 input tokens to a request, and screenshots are billed separately as image input. There is a separate browser use tool for tasks that stay in the browser.
OpenAI: the Responses API computer tool. Used in the Responses API with tools=[{"type": "computer"}]. OpenAI's docs describe three paths: the SDK's ComputerToolset (screenshots and mouse/keyboard), BrowserToolset (page text, controls and tabs) and a separate Agents API guide for a browser hosted by OpenAI. They also describe a "code execution" path in which the model writes code that drives the interface with Playwright or PyAutoGUI. Integrations with hosted environments such as Browserbase, Browser Use, Daytona and E2B are listed. In some cases the API can return safety checks that must be acknowledged before continuing.
Google: the Gemini API computer_use tool. Google announced the Gemini 2.5 Computer Use model in October 2025. According to the November 2026 docs, the tool is added as {"type": "computer_use", "environment": "browser"} and supports browser, mobile and desktop environments. Supported models include gemini-4-argon and gemini-3.8-flash, which the docs recommend for computer use. Along with an action, the model can return a safety_decision of require_confirmation, in which case your application has to ask the user. There is also opt-in prompt injection scanning that looks for hidden instructions in screenshots. Google provides a Docker-based reference implementation.
End-user products: agents inside the browser
- Claude in Chrome (Anthropic): According to Anthropic's help center, it's available on paid plans (Pro, Max, Ultra, Team, Enterprise) in the Chrome side panel, in Claude Cowork and in Claude Code. It can't access high-risk site categories such as adult or pirated content, and asks for permission before entering financial sites. On the same page Anthropic says that, despite its safety classifiers, use is "still risky."
- Gemini in Chrome, auto browse (Google): Announced by Google in January 2026 and opened as a preview to AI Pro and Ultra subscribers in the US. According to the Chrome help page, it aims to ask for confirmation before starting a task, sending communications, submitting forms, modifying data and accessing sites with highly sensitive financial or health data, and it hands the task back to the user for steps such as finalizing payments, accepting terms and creating accounts. The same page states plainly that Gemini can make mistakes, including completing a purchase without permission.
- OpenAI: ChatGPT agent, introduced in 2025, is no longer available according to OpenAI's help center, which recommends ChatGPT Work for longer, multi-step tasks. The ChatGPT Atlas browser was retired on August 9, 2026; OpenAI says it is moving browser-based agentic features into ChatGPT and Codex. I verified these two points from search-index summaries of OpenAI's help pages; I could not access the full text of the pages.
API or GUI automation?
Computer use is automation's last resort, not its first choice.
| API / CLI / MCP | Computer use (GUI) | |
|---|---|---|
| Speed | Milliseconds | Seconds per step (model call + screenshot) |
| Cost | Low | Image tokens and the tool definition on every step |
| Reliability | Deterministic, versioned contract | Breaks when the UI changes; clicks can miss |
| Coverage | Only what the API exposes | Anything you can do on screen |
| Security surface | Narrow: defined endpoints | Wide: every piece of text on screen is model input |
| Auditing | Request/response logs | Screenshot and action logs |
The decision order is simple:
- Is there an official API, CLI or MCP server? Use it.
- Does the task stay in the browser, with well-structured pages? Use a browser tool based on the accessibility tree / DOM.
- If neither, or it's a desktop application, use pixel-based computer use.
In practice a mix works best: the providers' docs also declare computer use alongside bash and text-editing tools, so the agent does file work in the terminal and only the necessary part in the interface.
Benchmarks: what do OSWorld and WebArena measure?
OSWorld (Xie et al., April 2024) is one of the first scalable agent test environments running inside a real operating system. It supports Ubuntu, Windows and macOS and contains 369 tasks (or 361, excluding 8 that need Google Drive). The tasks cover real web and desktop applications, file operations and workflows spanning several applications. Each task has an initial state and an evaluation script that checks the result by execution. When the paper was published, humans solved 72.36% of the tasks while the best model solved only 12.24%; according to the paper, the main problems were locating interface elements correctly (GUI grounding) and operational knowledge.
On July 28, 2025 the project was upgraded to OSWorld-Verified: community-reported faulty tasks were fixed and results were collected again. The project site asks that comparisons be made against the new results, so old and new numbers aren't directly comparable. The official table says it verifies results by running agent code on the project's own side.
The top results I saw on the OSWorld-Verified table on November 8, 2026 (with a 100-step limit):
| System | Organization | Type | Date | Success |
|---|---|---|---|---|
| Intelligence-Indeed Agent | Intelligence Indeed | Agent framework | July 25, 2026 | 90.19% |
| claude-fable-5 | Anthropic | General model | August 1, 2026 | 85.96% |
| claude-opus-5 | Anthropic | General model | August 1, 2026 | 83.39% |
Keep three things in mind when reading this table. First, the "agent framework" rows add planning and tool layers around a model, so they don't measure the same thing as the "general model" rows. Second, the paper's 72.36% human result belongs to the original task set; I haven't seen a human result re-measured for the Verified version, so "agents have beaten humans" doesn't follow from this data. Third, OSWorld numbers in providers' announcements can come with different step limits and setups; only results from the same table with the same settings are comparable.
WebArena (Zhou et al., July 2023) is a realistic, reproducible environment for web agents. It hosts fully functional sites in four domains: e-commerce, a social forum, collaborative software development and content management, plus tools such as a map and knowledge resources such as user manuals. Tasks are long-horizon and scored on the functional correctness of the result. In the paper, the best GPT-4-based agent reached 14.41% success, against human performance of 78.24%. I couldn't verify a current official results table for WebArena, so I'm not giving newer numbers here.
For the general limits of benchmarks, see Benchmark and Evals. In short: don't trust a number in a table until you have tested your own workflow, in your own environment, with your own success criteria.
Security: prompt injection, sandboxing and confirmation
The security problem with computer use is that everything the agent reads is a potential instruction. Anthropic's docs say so plainly: in some circumstances Claude follows commands found in content even when they conflict with your instructions, and instructions on web pages or in images can override yours. This is a prompt injection attack, and for screen-based agents the surface is huge: every web page, every email, every ad.
The providers' recommendations are almost identical:
- Isolated environment: Anthropic recommends a dedicated virtual machine or container with minimal privileges. OpenAI asks for a fresh browser profile or an isolated desktop, Google for a sandboxed VM or container. See Agent Sandbox for details.
- Keep sensitive data away: Don't give the agent account credentials or sensitive data. OpenAI's docs say to keep credentials out of page text and screenshots.
- Narrow the network: Restrict internet access to an allowlist of domains.
- Screen content is untrusted: In OpenAI's words, text in a page, document or tool result cannot grant permission or override the user's instructions.
- Human approval for critical actions: Purchases, sending data, hard-to-reverse changes, accepting terms, cookie consent. OpenAI counts typing sensitive information into a form as transmission too. This is the most concrete application of the human-in-the-loop principle. If the agent needs to pay for something, see Agent Payments.
- Bound and verify: Set step, time or cost limits, support cancellation, and check the result from the real state of the environment rather than the model's final answer.
Extra layers on the provider side: Anthropic uses classifiers that automatically scan computer use tool results (screenshots included) and, on a possible injection, steer the model to check whether the instruction really came from you. Google offers an out-of-model, per-step safety service that assesses each action before it runs, plus optional screenshot scanning. OpenAI's API can return safety checks that require acknowledgement before continuing. None of these is complete protection, as the providers' own docs say. The real defence is your environment and your approval gates.
Where does it work well, and where badly?
What follows is based on the limitations and general guidance in the providers' docs, not on my own measurements.
Where it works well
- Old web portals and internal tools with no API and an interface that rarely changes.
- Short, well-defined tasks whose outcome can be checked: "fill this form with this data", "download the reports for this date range".
- User-flow testing in web applications (both the OpenAI and Google docs give this as an example).
- Research and data-gathering tasks that a person watches alongside.
Where it works badly
- Latency and cost: Every step is a model call and a screenshot; not suited to high-volume work.
- Small or dense interfaces: Anthropic's docs list the common reasons clicks miss: unscaled coordinates, detail lost when downscaling from 4K screens, similar-looking elements. If accuracy is consistently poor, they suggest 1280x720 as a baseline resolution.
- Scrolling, spreadsheets, niche applications: Anthropic notes that scrolling may not take effect in some applications and that reliability drops in niche applications or across several applications at once.
- Creating accounts and posting on social media: Anthropic says Claude's abilities here are limited.
- CAPTCHAs: Google lists bypassing CAPTCHAs among the high-risk actions developers can block.
- Work that must be perfect or involves sensitive data: Anthropic advises against using Claude without human oversight for tasks requiring perfect precision or sensitive user information.
A minimal safe setup
If you are going to run your own computer use agent, here is the minimum checklist to start with:
- Environment: Start with the provider's Docker reference implementation or a separate VM. Don't use your everyday computer or your own browser profile.
- Accounts: The environment should only hold least-privilege accounts opened for this task. Do the login step yourself if you can; don't send passwords in the prompt.
- Network: Restrict outbound traffic to the domains the task needs.
- Approval gate: Catch purchase, send, delete, accept-terms and form-submit actions in your executor code and ask a person. The model's "ask for confirmation" signal is an extra layer, not a replacement.
- Limits: Set maximum steps, time and cost; have a "stop" button.
- Logs: Record every step's screenshot and action, so you can later see what was done and why.
- Verification: Add a step that checks whether the task is done from the final state of the environment (file, record, page).
Roughly, the idea of an approval gate on the executor side looks like this (pseudocode, using the Anthropic toolset's member names):
RISKY_WORDS = ("buy", "pay", "send", "delete", "accept", "submit")
def run_action(name, args, last_screenshot_text):
# Text on screen is not an instruction; only the user's task is.
if name in ("left_click", "key") and any(w in last_screenshot_text.lower() for w in RISKY_WORDS):
if not ask_human(f"The agent wants to do: {name} {args}. Approve?"):
return "Not executed: the user declined this action."
return execute_in_vm(name, args)This is a crude heuristic; the real work is done by a policy that recognizes the action at the application level (which domain, which button, which form). But even as a heuristic, the approval gate has to live outside the model, in your code.
Further reading
- Computer Use and Vibe Coding: short definitions.
- AI Agent, Agent Loop and Function Calling: the foundations computer use builds on.
- Agent Sandbox, Prompt Injection and Human-in-the-loop: the three legs of a safe setup.
- Coding Agents Compared: the code-side version of the same permission and sandbox logic.
- Agent Protocols and LLM Evals and Observability: connecting agents to each other and measuring them.
Sources
- Anthropic — Computer use tool: https://platform.claude.com/docs/en/agents-and-tools/tool-use/computer-use-tool
- Anthropic — Browser use tool: https://platform.claude.com/docs/en/agents-and-tools/tool-use/browser-use-tool
- Anthropic — Computer use reference implementation: https://github.com/anthropics/claude-quickstarts/tree/main/computer-use-demo
- Anthropic Help Center — Get started with Claude in Chrome: https://support.claude.com/en/articles/12012173-getting-started-with-claude-in-chrome
- Anthropic Help Center — Use Claude in Chrome safely: https://support.claude.com/en/articles/12902428-using-claude-in-chrome-safely
- OpenAI — Computer use (API guide): https://developers.openai.com/api/docs/guides/tools-computer-use
- OpenAI Help Center — ChatGPT agent: https://help.openai.com/en/articles/11752874-chatgpt-agent
- OpenAI Help Center — Evolving Atlas into ChatGPT for browser-based agentic work: https://help.openai.com/en/articles/20001371-evolving-atlas-into-chatgpt-for-browser-based-agentic-work
- Google — Gemini API Computer use: https://ai.google.dev/gemini-api/docs/computer-use
- Google — Introducing the Gemini 2.5 Computer Use model (October 7, 2025): https://blog.google/technology/google-deepmind/gemini-computer-use-model/
- Google — Computer use reference implementation: https://github.com/google/computer-use-preview/
- Google — Chrome gets new Gemini 3 features, including auto browse (January 2026): https://blog.google/products-and-platforms/products/chrome/gemini-3-auto-browse/
- Google Chrome Help — Ask Gemini in Chrome to complete tasks for you with auto browse: https://support.google.com/chrome/answer/16821166
- Xie, T. et al. — "OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments", 2024. https://arxiv.org/abs/2404.07972
- OSWorld project site and OSWorld-Verified results table: https://os-world.github.io/
- Zhou, S. et al. — "WebArena: A Realistic Web Environment for Building Autonomous Agents", 2023. https://arxiv.org/abs/2307.13854
Frequently asked questions
What is computer use?
Computer use is an agent loop in which a model looks at a screenshot and proposes mouse and keyboard actions such as clicking, typing and scrolling, while your application performs those actions in a virtual machine or browser and returns a fresh screenshot. The model never touches the screen; your code executes the action.
How is computer use different from ordinary API automation?
API automation talks to a service's programmatic interface; it is fast, cheap and deterministic. Computer use works the graphical interface built for people; it helps with applications that have no API, but because every step is a model call plus a screenshot it is slower, more expensive and more brittle. If there is an official API, CLI or MCP server, use that first.
Which providers offer a computer use tool?
As of November 2026, Anthropic offers the computer_toolset_20260801 toolset in the Messages API, OpenAI offers the computer tool in the Responses API and Google offers the computer_use tool in the Gemini API. In all three, an environment you control executes the actions. For tasks that stay in the browser, Anthropic's browser use tool and OpenAI's BrowserToolset can also read the page through the accessibility tree.
Do ChatGPT agent and Atlas still exist?
According to OpenAI's help center, ChatGPT agent is no longer available and ChatGPT Work is recommended for longer, multi-step tasks. The Atlas browser was retired on August 9, 2026, and its browser-based agentic features are moving into ChatGPT and Codex.
How good are computer use agents?
When the OSWorld paper came out (April 2024), the best model solved 12.24% of tasks and humans 72.36%. On the official OSWorld-Verified leaderboard on November 8, 2026, the top entry is an agent framework dated July 25, 2026 reporting 90.19% success. These numbers come from a controlled test environment; they don't guarantee the same success on your own work.
Is computer use safe?
Not by itself. The biggest risk is prompt injection: the model reads every piece of text on the screen and may follow a hidden instruction on a web page. All the providers recommend running the agent in an isolated VM or container, limiting the network to allowed domains, keeping sensitive data away from the agent and getting human approval before actions such as purchases, sending and deleting.
Should I give the agent my password?
Preferably not. Do the login step yourself or use the browser's take-over feature, and make sure the agent's environment only holds a least-privilege account opened for that task. Anthropic's documentation also notes that using computer use in applications that require login increases the risk from prompt injection.