How human data trains AI models

This lesson covers the three kinds of human data behind today’s AI models: examples of the right answer, judgments about which of two answers is better (InstructGPT paper, OpenAI), and tasks paired with a way to grade attempts at them (Epoch AI, FAQ on RL environments). For each, it explains where it came from and what it means for how you do the work.

The original recipe: demonstrations, then comparisons

OpenAI’s 2022 InstructGPT paper describes a three-step method for training a model to follow instructions. It used people in two ways: writing the response a model should give, and comparing the model’s own responses to say which was better (InstructGPT paper, OpenAI).

In the first step, labelers wrote demonstrations of the desired behavior for prompts, and the model was fine-tuned on those demonstrations with supervised learning. This is supervised fine-tuning, usually shortened to SFT (InstructGPT paper, OpenAI).

In the second step, labelers were shown between four and nine model outputs for the same prompt and ranked them. Those rankings trained a separate reward model, whose job was to predict which output a person would prefer (InstructGPT paper, OpenAI).

In the third step, the reward model’s score was used as the reward for reinforcement learning, and the second and third steps could be repeated with fresh comparisons collected on the improved model (InstructGPT paper, OpenAI).

Demonstrations: writing the answer

The same idea now covers agent work. Epoch AI’s interviews with people building reinforcement-learning environments describe gold trajectories, human-verified examples of completing a task successfully, which can be used as training data for supervised fine-tuning (Epoch AI, FAQ on RL environments).

Epoch also reports that supervised fine-tuning is the more practical choice when good examples are easy to produce but a reliable grader is hard to build, because it can learn from a single good attempt (Epoch AI, FAQ on RL environments).

Comparisons: saying which is better

Anthropic’s 2022 paper on training a helpful and harmless assistant collected comparisons one conversation turn at a time. Crowdworkers chatting with a model were shown two responses and asked to pick the more helpful and honest one, or, when deliberately trying to provoke harmful output, the more harmful one (helpful and harmless assistant paper, Anthropic).

In the InstructGPT setup, comparisons do not train the assistant directly. They train the reward model, and the reward model’s score is what the assistant is then optimized against (InstructGPT paper, OpenAI).

That is how a habit in human ratings can be amplified. One study traced a drift toward longer responses to reward models, which it found were easily influenced by length biases in the preference data they learned from. In two of its three test settings, a reward based on nothing but length came close to the full method’s results (length correlations in RLHF paper).

Tasks and graders: reinforcement learning

In modern reinforcement learning for language models, the model attempts a task, a grader scores the attempt, and the scored attempts are used to update the model. The grader is typically automated, such as a unit test or a model judging the attempt against a rubric (Epoch AI, FAQ on RL environments).

Each task is a prompt stating an objective plus a grader that decides whether, or how far, the objective was met. Epoch’s examples include fixing a bug so that a repository’s tests pass, and finding the cheapest listing that meets given conditions on a clone of a home-rental website (Epoch AI, FAQ on RL environments).

Epoch reports that math and coding tasks came first, that enterprise workflows such as filing reports or working in spreadsheets are a major growth area, and that interest is growing in longer tasks with many steps (Epoch AI, FAQ on RL environments).

For answers a test cannot check, researchers have turned to rubrics: a list of criteria written for one prompt, which a judge model scores the response against. Rubrics sit between a single right-or-wrong check and a coarse preference ranking (Rubrics as Rewards paper, Scale AI).

Writing those criteria is expert work. In one rubric dataset released in 2026, domain experts wrote between 10 and 40 criteria for each prompt (ComplexConstraints rubrics paper, Surge AI).

Reward hacking, and why graders need people

Interviewees in Epoch’s survey named robustness against reward hacking, a model scoring well without actually doing the task, as the most important quality criterion. One gave the example of a model searching for the solution instead of working it out (Epoch AI, FAQ on RL environments).

Platforms now recruit people for exactly this. Outlier advertises expert work, at up to $150 an hour, that includes writing the automated graders that score an agent’s attempt and rubrics that tell a genuine solution from an agent gaming the score (Outlier ML/AI expert page).

One founder told Epoch that domain knowledge and expert-level prompting matter more than machine-learning skills for creating tasks (Epoch AI, FAQ on RL environments).

Epoch also notes that data vendors which used to sell mostly human-labeled data, naming Mercor, Surge, Handshake and Turing, now sell these environments as well (Epoch AI, FAQ on RL environments).