Computer Use Agents - Virtual Desktops for GUI Agents
Running agents that drive a real browser or desktop session, the infrastructure they require, and how this differs from headless code execution
What a Computer Use Agent Actually Is
Same models, a completely different observation channel
A headless coding agent works in text: command in, stdout and an exit code out. A computer use agent works in pixels: screenshot in, coordinates and keystrokes out, no exit code. Success is inferred from the next frame, and task state lives in whatever is rendered.
That produces a failure mode with no analogue in shell work. Anthropic's computer use documentation warns that "Claude might make mistakes or hallucinate when outputting specific coordinates." A shell agent that invents a flag gets a clean error back. A GUI agent that invents a coordinate clicks something real.
Headless shell agent
Text in, text out. Exit codes give the loop a hard success signal. State lives in the filesystem and in version control, so it is inspectable and diffable. Reset is a fresh container.
GUI computer use agent
Screenshots in, coordinates out. No exit codes. State lives in pixels and application memory, so it is neither diffable nor easily logged. Reset means tearing down a desktop session.
The Infrastructure a Desktop Session Requires
A display stack, not just a container image
Anthropic's documentation states that computer use "requires a sandboxed computing environment," then enumerates it: a "virtual X11 display server (using Xvfb)"; a "lightweight UI with window manager (Mutter) and panel (Tint2) running on Linux"; and "pre-installed Linux applications such as Firefox, LibreOffice, text editors, and file managers," all inside a Docker container.
That is a bill of materials: display server, window manager, desktop shell, a real application, a screenshot pipeline, an input injection path, and a viewer. None of it exists in a build container.
What a GUI agent container carries
# Illustrative. Package list taken from Anthropic's computer-use-demo
# reference Dockerfile in the anthropic-quickstarts repository.
# Virtual display, input injection, screenshot capture, remote view
apt-get install -y xvfb xterm xdotool scrot imagemagick sudo mutter x11vnc
# Desktop shell plus the applications the agent is expected to drive
apt-get install -y firefox-esr x11-apps xpdf gedit xpaint tint2 \
galculator pcmanfm unzip
# Browser-facing view, cloned from source in the reference image
# noVNC v1.5.0 + websockify v0.12.0
# Ports exposed by the reference image
# 5900 VNC
# 6080 noVNC desktop
# 8501 Streamlit agent UI
# 8080 combinedThe reference implementation is a demo, not a platform
The computer-use-demo README says so directly: "The components are weakly separated: the agent loop runs in the container being controlled by Claude, can only be used by one session at a time, and must be restarted or reset between sessions if necessary." Concurrency and a separate control plane are yours to build. Anthropic opens its limitations section with "Computer use is in beta."
A desktop session is a long-lived stateful workload with a viewer attached, so it fits request-scoped scheduling badly. Treat these as pods with real lifetimes, under the posture in Kubernetes agent sandboxes; they need no GPU capacity unless the application demands it.
Headed browsers are not headless browsers
You do not need an agent to hit this. Playwright's CI docs state that "on Linux agents, headed execution requires Xvfb to be installed" and that you should "add xvfb-run before the actual command." Its browser documentation confirms these are different binaries: a "regular Chromium build for headed operations and a separate chromium headless shell for headless mode."
The display stack is the price of a rendered browser, not an agent-specific tax. But a headed session carries a display server, window manager, desktop shell, VNC server, web client and full browser instead of one process, so per-session cost is materially higher - and no first-party source publishes a figure for that gap.
Why This Separates the Sandbox Vendors
Most agent sandboxes hand you a container and a shell
A shell sandbox is a commodity: image, filesystem, process, text. A desktop session needs a display stack maintained per operating system, a streaming path to a human observer, session persistence across a multi-step task, and an API that expresses clicks rather than commands. Every claim below comes from Daytona's Computer Use documentation and is not independently verified.
Vendor claim: the feature
Computer Use "enables programmatic control of desktop environments within sandboxes," providing "mouse, keyboard, screenshot, screen recording, and display operations for automating GUI interactions."
Vendor claim: the stack
The docs describe starting "all computer use processes (Xvfb, xfce4, x11vnc, novnc)." The VNC access page labels those an X virtual framebuffer, a desktop environment, a VNC server, and a web client.
Vendor claim: coverage
"Computer Use is available for Linux and Windows. macOS support is currently in private alpha." The API covers click, drag, scroll, type, hotkey and screenshots, plus an accessibility tree - a non-pixel way to locate a control.
The same vendor marks how this market moved. In April 2025 Daytona announced Daytona Cloud and repositioned as agent-native infrastructure, moving away from its origins as a development-environment manager. Its public repository is not archived, but the README states "This repository is no longer maintained" and that core development moved to a private codebase in June 2026.
Operating System Support and Licensing
Linux is the easy case and everything ships there first
| OS | Stack | Licensing consideration | Ephemerality |
|---|---|---|---|
| Linux | Xvfb, window manager, desktop shell, VNC server and web client in one image | None beyond the applications you install | Per-task: start, drive, destroy |
| Windows | A real VM or session host with its own remoting path, not a container | Per-user entitlements. Microsoft Learn: Windows 365 users "must have licenses for Windows E3, Intune, Microsoft Entra ID P1, and Windows 365"; Azure Virtual Desktop needs "an eligible license for each user that accesses" it; a Windows Server session host needs an RDS CAL per user or device | Possible, but entitlements count per identity |
| macOS | Apple hardware. AWS: "Mac instances are available only as bare metal instances on Dedicated Hosts," one per host | The macOS license is "For use on Apple-branded Systems," permits two virtualized instances on Apple hardware you own or control, and bars their use "in connection with service bureau, time-sharing, terminal sharing, relay service or other similar types of services" | Not ephemeral. Apple requires "a minimum period of twenty-four (24) consecutive hours"; AWS applies a matching 24 hour minimum |
The macOS constraint is structural, not a pricing quirk. The host is held and billed for a full day, and Apple's lease clause requires "sole and exclusive use and control" throughout. If your design assumes disposable desktops, macOS breaks it.
The second point is one for counsel. Microsoft states that "anyone accessing Azure Virtual Desktop or any other Online Services must be licensed," explicitly including contractors. An autonomous agent holding its own desktop session is neither obviously a human user nor a device, so whether it counts as a licensed user is unresolved - and it bears on how you model agent identity. Microsoft's Listed Providers guidance (October 2019) separately bars on-premises licenses bought without Software Assurance from dedicated hosted cloud services offered by Listed Providers. Get an answer before you scale a fleet.
Security Posture - A Much Larger Blast Radius
A logged-in browser is a bigger grant than any API token you would issue
An agent with a shell holds the credentials you injected, which you can scope and revoke. An agent driving a real browser holds whatever that session holds.
Session cookies beat scoped tokens
A profile carries live sessions for every service it has signed into, at whatever privilege the human holds. No read-only variant, no per-scope grant, rarely a short expiry.
No allowlist granularity at the click
You can allow or deny a hostname. You cannot allow "read the invoice list" and deny "press Delete" in a rendered page - both are the same event on the same origin.
Every rendered page is untrusted input
OpenAI's guide is blunt: "Run Computer use in an isolated browser or VM, keep a human in the loop for high-impact actions, and treat page content as untrusted input." Anthropic warns of risks "heightened when interacting with the internet."
VNC is a remote control surface
The viewer that lets you watch the agent lets anyone reaching that port take the keyboard. The reference image exposes VNC on 5900 and noVNC on 6080. Treat both as privileged.
Never attach an agent to a human's browser profile
It is the fastest way to stand one up and the worst decision available: it hands the agent that person's entire authenticated surface. Anthropic's guidance is to "always carefully review and verify Claude's computer use actions and logs."
The workable posture is unglamorous. Give the session its own empty profile, signed in only to the origins the task needs. Put the desktop behind the same network policy as every other agent workload - see agent egress control. Keep it disposable, per sandbox environments and microVM isolation. Give it its own identity, pair that with prompt injection defense, and write the incident path into your agent runbooks.
Reliability - Early, Slow, and Improving Fast
The published numbers do not support unattended operation
"OSWorld 2.0: Benchmarking Computer Use Agents on Long-Horizon Real-World Tasks," an arXiv preprint from XLANG Lab (HKU) and collaborators, June 2026, reports that under its "primary binary-completion metric at 500 steps," Claude Opus 4.8 "scores best but still completes only 20.6% of tasks at a 54.8% partial score," while "GPT-5.5 is far more token-efficient yet plateaus near 14%." Its tasks "require substantially longer human operation time, with a median of about 1.6 hours." No human success-rate baseline is published for it.
The 2024 rows below show the trajectory, not the state of the art. They are the figures published alongside the original benchmark, and nothing on this page is a current leaderboard standing. Check the OSWorld-Verified leaderboard itself before citing any of them. Quote dates whenever you quote numbers.
| Benchmark | Agent result | Human baseline | Source and date |
|---|---|---|---|
| OSWorld 2.0 | 20.6% binary completion at 500 steps, 54.8% partial | Not published | arXiv preprint, XLANG Lab (HKU), June 2026 |
| OSWorld (original) | 12.24% best model | over 72.36% | arXiv, 2024 - stale, trajectory only |
| Windows Agent Arena | 19.5% in the Windows domain | 74.5% unassisted human | Microsoft, arXiv, 2024 |
| WebArena | 14.41% end-to-end, best GPT-4-based agent | 78.24% | NeurIPS 2024 |
Speed is the other constraint. Anthropic warns that "the current computer use latency for human-AI interactions might be too slow compared to regular human-directed computer actions," and advises focusing "on use cases where speed isn't critical." OpenAI's January 2025 research post reported "a 38.1% success rate on OSWorld" against human performance of 72.4%, and said "we don't expect CUA to perform reliably in all scenarios just yet."
The model surface churns. OpenAI's deprecations page gives computer-use-preview a shutdown date of July 23, 2026; that preview line was retired and the capability folded into newer general models. Anthropic's runs behind a beta header, computer-use-2025-11-24. Pin versions and expect to re-test. Full citations sit on the sources page.
When You Need a Desktop, and When You Do Not
Most coding work does not need one
Genuinely needs a GUI
Testing a desktop application. Driving a vendor web console with no API. Visual regression work. Reproducing a bug that only appears in a rendered browser. Workflows behind an SSO portal with no machine path.
Does not need a GUI
Editing code. Refactoring. Running tests. Dependency and build work. Reading logs. Code review. Migrations. Essentially everything a coding agent does all day, and all of it faster and more verifiable through a shell.
A desktop session costs more per hour, fails more often, produces no exit codes to gate on, and hands the agent wider credentials than a shell would. Choosing it when a shell would do multiplies cost and blast radius for nothing. The right default is a headless sandbox, with a GUI session reserved for the narrow steps that cannot be done another way - kept short, watched, and destroyed afterwards. Verify those steps with agent evals.
Related Topics
The infrastructure a GUI agent session depends on
Sandbox Environments
The disposable execution layer a desktop session sits on
MicroVM Isolation
Hardware-backed boundaries where a container namespace is not enough
Agent Egress Control
Network policy for the desktop, including the origins a driven browser may reach
Agent Evals
How to measure a non-deterministic workflow before you trust it
Prompt Injection Defense
Mitigations when the observation channel is attacker-controlled
Autonomous Development
Where unattended agent work is heading, and what must be true first