How an AI Agent Controls Mouse and Keyboard Input on Screen

An AI agent controls mouse and keyboard input by looking at screenshots of a computer and replying with actions, such as a click at one spot followed by a line of typing. A small program on your side carries out each action on a real or virtual desktop, takes a fresh screenshot, and sends it back. The model sees pixels and issues instructions, and your code does the clicking.

The industry calls this computer use. It lets an agent work in software that has no API, which describes a surprising share of the software businesses run on. This guide covers how the loop works and where it earns its keep, along with the safeguards to set up before you hand an agent your cursor.

How an AI agent controls mouse and keyboard input

The plain version: the agent works in a loop of looking and acting. It looks at a screenshot, decides the next step, and asks for one or more actions. Then it looks again to see what happened.

OpenAI described this loop when it launched its Computer-Using Agent in January 2025. The model adds screenshots to its context, reasons about the next step, then clicks or types "until it decides that the task is completed or user input is needed." Anthropic's version works the same way. In its computer use tool documentation, a single toolset entry gives Claude 17 actions, among them screenshot and left_click, and your application "runs every call in an environment you control."

A typical turn looks like this:

1. Your code sends the task and the latest screenshot to the model.
2. The model replies with tool calls, for example a click at pixel coordinates [512, 742] followed by a screenshot.
3. Your code performs each action in order on the desktop and returns the results.
4. The model checks the new screenshot and either continues or reports back.

Coordinates are in screenshot pixels, counted from the top left. If you shrink screenshots before sending them, which you will to save tokens, you have to scale the model's coordinates back up before clicking. Forget that step and the agent clicks with total confidence on a spot slightly up and to the left of every button, like a tourist pointing at the wrong landmark.

Screen control or an API: picking the right interface

Screen control is the slowest and most fragile way for an agent to do anything. Use it when nothing better exists.

Every action costs a round trip and usually a screenshot, which is an image the model has to read. A task that an API call finishes in 200 milliseconds can take dozens of steps through a GUI, meaning the graphical user interface of buttons and menus people see. Layouts also change without warning, and a redesign that moves one button can break an agent that worked yesterday. Front-end teams tend to ship those redesigns on a Friday, mostly out of tradition.

The order of preference runs like this:

• An API or a structured tool, when the system offers one. Fast and easy to audit.
• A browser tool that reads the page structure, when the task stays inside websites. Anthropic's docs point web-only tasks to its browser use tool for this reason.
• Full computer use, for desktop software and legacy systems with no other way in.

Our guide to text editor AI agents shows the structured end of this spectrum, where the agent edits files through a tool and never moves a cursor around an editor. Pick that route whenever the job allows it.

AI agent examples where screen control earns its keep

The best computer use jobs are repetitive and stuck behind a screen, with little at stake if one run fails. Good candidates include:

• Copying records from a legacy desktop app with no export into a spreadsheet.
• Filling the same internal web form for each row of a list.
• Running regression checks on your own app's interface, clicking through a flow the way a user would.
• Gathering figures from several dashboards that share no API into one summary.

OpenAI's launch page includes a useful reliability table. Creating a project in Todoist and adding a seven-item shopping list succeeded 10 times out of 10. A Redfin search for a townhouse with a budget and an energy-efficiency requirement among other filters succeeded 3 times out of 10. Repetitive clicks come easily to these agents. Open-ended judgement across unfamiliar interfaces is still hard.

The same table shows how much the prompt matters. A venue search succeeded 3 of 10 times with a loose prompt and 8 of 10 times when the prompt named the exact hours and told the agent to check the filters section. For more ideas that hold up in practice, see our roundup of agentic AI use cases in production.

Benchmarks tell the same story. The OSWorld benchmark of 369 real tasks across desktop and web apps found in April 2024 that the best model completed 12.24% of tasks, while humans managed over 72%. The authors traced most failures to GUI grounding, meaning the step of matching an instruction to the right element on screen. OpenAI's agent reached 38.1% on the same benchmark nine months later. Models have kept improving since, and the human score is still the bar, so plan for failed runs.

The security problem with an agent on your screen

An agent that reads your screen will also read whatever is on it, including text written by someone else. That is the central risk of computer use.

Anthropic's documentation is blunt about it: "In some circumstances, Claude follows commands found in content even when they conflict with your instructions." A web page or even text inside an image can carry instructions, a technique called prompt injection. An agent with your mouse and your logged-in sessions can act on them. Anthropic runs classifiers over screenshots to flag likely injections, and its docs say the other precautions still matter with those classifiers in place.

The safeguards both vendors describe come down to a short checklist:

• Run the agent in a dedicated virtual machine or container with minimal privileges.
• Keep sensitive data out of reach, including saved passwords and accounts the task does not need.
• Limit internet access to an allowlist of domains.
• Require a human to confirm anything with real consequences, such as payments or accepting terms.
• Cap the number of loop iterations so a confused agent cannot run up costs.

OpenAI's agent asks for confirmation before sensitive steps such as entering login details. If you build your own loop, put that check before each action runs. A single model response can contain a batch of several actions, so a check after the turn comes too late.

Building the loop with AI agent frameworks

A small script is enough to start. The loop itself fits in about 20 lines of code, and the real work sits in the environment and the guardrails around it.

Anthropic publishes a reference implementation that runs everything inside a Docker container, with a virtual display and a lightweight Linux desktop for the agent to work in. Most AI agent frameworks wrap the same pieces. Whatever you pick, you own these parts:

• The environment: a container or VM with a virtual display the agent can see.
• The action executor: code that turns "left_click at [x, y]" into a real click.
• The loop: send the screenshot, run the returned actions in order, stop at the first failure, repeat.
• The limits: a step cap and a confirmation gate.

Two prompting tips from Anthropic's docs make a visible difference. Ask the agent to take a screenshot after each step and say whether it worked, because models sometimes assume an action succeeded without checking. And when dropdowns or scrollbars give it trouble, tell it to use keyboard shortcuts. Tiny scrollbars defeat these models the way they defeat anyone using a trackpad on a moving train.

For a deeper treatment of tool design and agent loops in Python, Hyun Erwin's Agentic AI Engineering covers the components you will be wiring together.

Where to start with a computer use agent

Start small and boring. Pick one repetitive task you already do by hand, with nothing irreversible in it, and run it in a sandbox.

1. Write the task as numbered steps, the way you would brief a new temp.
2. Run it ten times and count the successes. One clean demo proves very little.
3. Read the failed runs screenshot by screenshot and add hints to the prompt where the agent got lost.
4. Connect it to anything real only after that, with a human approving the final step.

If the task turns out to have an API after all, switch to it and enjoy the speed. For a lighter assistant in day-to-day work, our AI personal assistant guide explains what those tools can see and do. Computer use is a powerful fallback, and the agent will happily click the same wrong button forty times unless your loop tells it to stop.