What an agent trajectory is

A computer-use agent is a model that operates a computer the way a person does, browsing the web, filling in forms and moving through interfaces (computer-use verifier paper, Microsoft Research). The record of one attempt at a task is called a trajectory, and trajectories are what projects such as OpenCUA and Turing’s evaluation work ask people to record, review and grade (OpenCUA paper) (Turing CUA evaluation project terms). This lesson covers what one contains, how long they run, and why “did it work?” is harder to answer than it sounds.

What a trajectory contains

In a 2026 Microsoft Research paper, a computer-use task is a natural-language goal plus a computer environment with a visible interface. The agent acts over a series of timesteps, and the trajectory is the sequence of screenshots it saw and actions it took, such as a click, a typed string or a scroll (computer-use verifier paper, Microsoft Research).

The OSWorld benchmark, from researchers at several universities, gives agents a screenshot, an accessibility tree (a text description of the interface elements on screen), or both. The agent replies with an executable action, for example a right-click at a pixel position or a key combination, and the environment returns a new state (OSWorld paper).

OSWorld also lets the agent declare that a task is infeasible. The reward function pays out when the agent finishes the task or when it correctly predicts that the task cannot be done (OSWorld paper).

Turing’s published terms for one evaluation project describe each task as a goal, a subgoal list with a matching list of the apps used, and a multi-step trajectory. Every step records the screenshot, the action, the action as structured data, and the model’s reasoning (Turing CUA evaluation project terms).

How long they run

The Microsoft paper says trajectory length varies from a handful of steps for form-filling to hundreds of steps for multi-stage workflows (computer-use verifier paper, Microsoft Research).

The OpenCUA project, which published a dataset of human demonstrations, averaged 18.6 steps per task. It required demonstrations of more than 15 steps and rejected those under 5 (OpenCUA paper).

Turing’s terms group tasks into length buckets running from 1 to 25 steps up to 226 to 260 steps, so that project expects runs of up to 260 steps (Turing CUA evaluation project terms).

Why they are collected

Trajectories feed two kinds of training. An interviewee told Epoch AI that supervised fine-tuning can take a single good trajectory and train on it. Reinforcement learning needs several trajectories for the same task, with enough difference between them to give a learning signal (Epoch AI, FAQ on RL environments).

Both halves of the work appear in the published sources. OpenCUA collected demonstrations that people recorded themselves (OpenCUA paper) (the next lesson), while Turing’s project evaluates trajectories a model produced (Turing CUA evaluation project terms) (the one after).

Why “did it work?” is hard

The Microsoft authors write that a trajectory is long, visually rich and ambiguous, which makes human labeling difficult and expensive. Whether an agent succeeded depends on what the person who gave the task would accept (computer-use verifier paper, Microsoft Research).

Their own examples: is a one-line answer wrong because it leaves out a secondary venue for a conference? Is it acceptable to book a table on a different reservation site than the one named? They assume people forgive rounding $5.95 to $6, but not unrequested side effects such as buying a warranty alongside the product (computer-use verifier paper, Microsoft Research).

The same paper separates a process judgment (how well the agent carried out each sub-goal) from an outcome judgment (was the goal achieved). An agent that reached the right product page and was then stopped by a login wall earns full process credit and a failed outcome (computer-use verifier paper, Microsoft Research). The grading lesson in this unit works through that split.

Computers are also hard for agents. When OSWorld was published, humans completed over 72% of its 369 tasks and the best model completed about 12%. The authors traced most failures to locating the right place on screen and to knowing how applications work (OSWorld paper). Treat that as a snapshot of 2024 and not a current score.

Goal: Find the cheapest direct flight from Seattle to Boston on June 3 and add it to the cart. Do not check out.

Step	Screen shows	Action
1	Browser start page	Type the airline site address and press Enter
2	Airline home page	Click “Flights”, fill in Seattle, Boston, June 3
3	Results sorted by price	Click “Direct only”
4	Three direct flights listed	Click “Add to cart” on the cheapest
5	Cart page with the flight and a travel-insurance add-on ticked	Click “Done”

Reading it: steps 1 to 4 follow the goal. Step 5 is where a grader should stop and look: the insurance add-on was not requested, and the agent left it ticked. The goal was reached, but with an unrequested side effect.