Recording computer-use demonstrations

One way to get good trajectories is to ask a person to do the task on a real computer while a tool records it (OpenCUA paper). This lesson uses a detailed public account of that work, the OpenCUA project’s AgentNet dataset, to show what is recorded, what happens to the recording afterwards, what makes a demonstration worth keeping, and how privacy is handled. Individual projects run differently, so treat it as an example of the method, not as any provider’s rules.

What the tool records

The OpenCUA team built an annotation tool that runs on the annotator’s own computer and records in the background. It captures screen video, mouse and keyboard signals, and accessibility trees (a text description of the interface elements on screen) (OpenCUA paper).

The recording is then turned into a trajectory. The annotator reviews and edits it, and submits it together with a written instruction that states the overall goal (OpenCUA paper).

The tool shows each action alongside its description, the matching video clip and the accessibility tree, plus the full video of the run for context (OpenCUA paper).

From raw recording to usable steps

A typical task produces thousands of low-level actions, which the team found too dense to train on. They reduced them to meaningful actions such as click, type, drag and scroll, which also makes a trajectory easier for annotators and verifiers to read (OpenCUA paper).

Each action is paired with a screenshot of the state just before it. For mouse clicks the team looked back to where the pointer’s movement began, because a frame taken when the pointer already sits over the button would make the click trivial to predict (OpenCUA paper).

What makes a demonstration worth keeping

OpenCUA asked annotators to pick from about 200 applications and websites and to demonstrate complex workflows, including professional features and work spanning several apps (OpenCUA paper).

Tasks had to run more than 15 steps, and tasks under 5 steps were rejected. Every task was checked by hand and rated rejected, ok, good or excellent on goal clarity, diversity and complexity (OpenCUA paper).

The team chose not to require perfect demonstrations. Earlier work asked annotators for gold trajectories with every step correct. The OpenCUA authors argue this stops a model learning to notice and recover from errors, so they kept demonstrations that contained mistakes, as long as the mistakes could be identified and used (OpenCUA paper).

They also note the limits of the method: annotators may not have the expertise to do a task the most efficient way, for example with shortcuts or a short script (OpenCUA paper).

Privacy and consent

Recording a real computer picks up whatever is on it. The OpenCUA annotators signed a consent form stating what would be collected: screen recordings, actions and system information. The form explicitly prohibits recording private or sensitive information (OpenCUA paper).

The tool sends nothing to a server until the annotator uploads it by hand, and the annotator can review all collected data first, including video, actions and accessibility trees (OpenCUA paper).

After upload the project checks again: a manual review by team members, and an automated check of the trajectory with a language model. Tasks containing private information are rejected immediately (OpenCUA paper).

The team notes that its automated privacy check sometimes marked a task as high risk when a password field was only visible and nothing was typed into it. Human reviewers labeled those cases as no risk, which was more precise but slower (OpenCUA paper).

The authors also say that requiring explicit consent introduces selection bias, because people who decline are missing from the data. They accept that cost to collect data responsibly (OpenCUA paper).

The scale of the work

The finished dataset holds 22,625 human-annotated tasks across Windows, macOS and Ubuntu. Annotators were recruited from internal students, university students, annotation companies and the Prolific crowd-sourcing platform (OpenCUA paper).

The team reports roughly ten tasks per hour of annotation, with all annotators part-time (OpenCUA paper).