Tasks, verifiers and reward hacking

Some of the agent work on offer is not rating a model’s answer. It is building the task the model is trained on, and the grader that decides whether an attempt succeeded (Epoch AI, FAQ on RL environments), and Outlier lists writing graders and verifiers among the responsibilities of one expert role (Outlier ML/AI expert page). This lesson covers how published projects build and audit those, and what reward hacking is.

A task and a verifier

Epoch AI describes reinforcement learning for language models as a loop. The model is given a task and a set of actions it can take. It attempts the task and a grader assigns a score, and the scored attempts are used to update the model (Epoch AI, FAQ on RL environments).

A task, in their account, is a prompt stating an objective plus a grader that decides whether, or how far, it was met. The grader is usually automated: a unit test, or a language model judging against a rubric (Epoch AI, FAQ on RL environments).

Their examples show the range. A repository with a failing test suite, where the grader runs the tests. A clone of a rental-listings site, where the grader checks the final answer. A spreadsheet clone, where the grader compares the output with a reference solution (Epoch AI, FAQ on RL environments).

The Terminal-Bench 2.0 benchmark gives a concrete recipe. Each task is an instruction, a container definition, a set of tests and an oracle solution. The agent works in the container, and the tests are copied in and run afterwards (Terminal-Bench 2.0 paper).

What reward hacking is

Reward hacking is a model scoring well by gaming the grader instead of doing the task. Epoch’s interviewees named robustness against it as the most important quality criterion for a task, and one said high reward must mean the task was actually solved, not hacked (Epoch AI, FAQ on RL environments).

The examples they give are plain: searching up a solution, or checking out a repository’s future commits when the repository is not scripted carefully. They also report that making a grader robust rarely works first time and takes many iterations (Epoch AI, FAQ on RL environments).

Terminal-Bench’s authors list the exploits their own adversarial test agent found: tampering with the test environment, guessing answers, and generating every possible answer in the hope that the tests only check that a correct answer is present, not that wrong ones are absent (Terminal-Bench 2.0 paper).

Three properties of a good task

Terminal-Bench defines three properties. A task is well specified if its tests pass if and only if the container ends in an acceptable state, so the instructions describe every correct end state and the tests capture every correct end state (Terminal-Bench 2.0 paper).

It is solvable if an oracle solution that mirrors a real workflow makes every test pass (Terminal-Bench 2.0 paper).

It has integrity if the agent cannot take a shortcut that would not exist in real use. Their example: when a task edits a repository at a given commit, all later commits are removed from its history (Terminal-Bench 2.0 paper).

How tasks are audited

Terminal-Bench 2.0 started from 229 tasks written by 93 contributors and kept 89, chosen by the authors’ difficulty assessment and a quality review by three experienced human reviewers (Terminal-Bench 2.0 paper).

Automated checks ran on every submission. The oracle solution had to make the tests pass, and other checks confirmed that a do-nothing agent fails the task (Terminal-Bench 2.0 paper).

Each task was run with several models. When an agent failed, a person decided whether the cause was a real gap in the model or a flaw in the task. Then an adversarial agent was told to find a way to cheat, and two further auditors reviewed the task by hand (Terminal-Bench 2.0 paper).

The authors estimate about three reviewer-hours of human attention per task, and say that verifying these tasks is extremely difficult (Terminal-Bench 2.0 paper).

When the tests are the problem

OpenAI audited SWE-bench Verified, a widely used coding benchmark whose tests decide whether a fix counts. In 138 problems a strong model did not consistently solve, a team of experienced engineers found that 59.4% had material problems in the tests or the problem description (OpenAI on SWE-bench Verified).

Of those, 35.5% had narrow tests that enforce specific implementation details and so reject correct solutions, and 18.8% had wide tests that check functionality the problem never described (OpenAI on SWE-bench Verified).

OpenAI also found that all the frontier models it tested could reproduce some original solutions or problem statements, which indicated exposure during training. On 2026-02-23 it announced it had stopped reporting SWE-bench Verified scores (OpenAI on SWE-bench Verified).

OpenAI’s own conclusion about automated scoring is that perfect tests should be agnostic to unimportant implementation details and robust to shortcuts, and that catching these problems took several large human labeling campaigns (OpenAI on SWE-bench Verified).

Difficulty is part of the job

Epoch’s interviewees say a task whose pass rate is 0% or 100% teaches the model nothing. Several wanted a minimum pass rate around 2 to 3%, or at least one success in 64 or 128 attempts (Epoch AI, FAQ on RL environments).

A 0% rate may mean the task is impossible. One interviewee said that, unless another annotator has solved it blind, you cannot tell (Epoch AI, FAQ on RL environments).

Where the work is being offered

Outlier’s page for ML and AI experts lists writing the automated graders and verifiers that score an agent’s attempt as a responsibility. It lists authoring rubrics, judged by a language model, that tell a genuine solve from an agent gaming the metric or hard-coding its way to a high score (Outlier ML/AI expert page).

Task: Rename every .txt file in a folder to .md, leaving the contents unchanged.

Weak verifier: passes if a file named notes.md exists.

Ways to pass without doing the task: create an empty notes.md; copy one file and leave the rest as .txt.

Stronger verifier: passes only if (a) no .txt files remain, (b) the number of .md files equals the original count, and (c) each .md file’s content matches the original text of the file it replaced.

What it still misses: a file the task never mentioned, such as a hidden .txt file in a subfolder, might need a decision about whether it counts. A better task statement settles it.

Checks borrowed from Terminal-Bench: does a do-nothing agent fail? Does a correct but different solution pass? Could the agent read the answer from somewhere in the environment?