How your work is checked: gold tasks, calibration and agreement

Submitting a task is rarely the end of it. Published projects describe screening before the first task (InstructGPT paper, OpenAI), calibration against known answers (computer-use verifier paper, Microsoft Research), blind second reviews (Turing CUA evaluation project terms) and automated checks on how you work (Prolific, authenticity checks). This lesson walks through each, and every figure in it comes from a named project, not an industry average.

Screening before the first task

The team behind OpenAI’s InstructGPT hired about 40 contractors after a screening test that measured how well each candidate’s rankings agreed with the researchers’ own, how well they flagged sensitive content, and how well they wrote responses to delicate prompts (InstructGPT paper, OpenAI).

The selection used soft cutoffs: 75% agreement with the researchers on flagging sensitive content and on ranking outputs, and a demonstration score of 6 out of 7 (InstructGPT paper, OpenAI).

NVIDIA’s HelpSteer3 paper describes similar gates. Its earlier HelpSteer dataset gave annotators a training course and preliminary assignments, and only those who succeeded joined the project. For HelpSteer3, specialist pools required a degree in a relevant field, work experience or proficiency in a specific language (HelpSteer3-Preference paper, NVIDIA).

Checking does not stop once you are in. Handshake says some projects require an assessment to confirm readiness or maintain quality, that new ones may be introduced as projects change, and that continued participation may depend on meeting them (Handshake AI Fellowship expectations).

Calibration against known answers

A gold item is one whose correct answer is already settled. In a 2026 Microsoft Research study of computer-use agents, annotators were first calibrated on 10 practice trajectories that already carried gold annotations, before judging any real ones (computer-use verifier paper, Microsoft Research).

Overlap: more than one person on each item

Projects often give the same item to more than one person and compare. In one of the computer-use study’s datasets, every trajectory had two annotators, the final success label was a majority vote, and a third annotator broke ties (computer-use verifier paper, Microsoft Research).

HelpSteer3 used three to five independent annotators per item, filtered out the annotations that disagreed most with the rest, and dropped items where even the three closest annotations were far apart (HelpSteer3-Preference paper, NVIDIA).

Agreement: what the numbers look like

Even trained raters disagree often. On InstructGPT’s comparisons, the training labelers agreed with each other about 72.6% of the time (InstructGPT paper, OpenAI).

Anthropic’s 2022 study measured about 63% agreement between its crowdworkers and its own researchers, which it called poor next to similar published work (helpful and harmless assistant paper, Anthropic).

On OpenAI’s GDPval tasks, expert graders comparing work deliverables agreed with each other 71% of the time (GDPval paper, OpenAI).

After its filtering, HelpSteer3 reported a weighted Cohen’s kappa above 0.8 for each subset, a measure of agreement that counts how far apart two ratings are, not just whether they match. The paper credits strict recruitment, quality control that removed underperforming annotators, and the filtering itself (HelpSteer3-Preference paper, NVIDIA).

Blind review and rework

Turing’s published terms for a computer-use evaluation project describe three roles. A base annotator evaluates each task first. A reviewer then re-evaluates 100% of tasks blind, without seeing the base annotator’s work, before comparing the two (Turing CUA evaluation project terms).

The reviewer sends a task back if step-level disagreement exceeds 5%, or if the two disagree on any task-level judgment. An auditor runs a further blind pass on about 20% of completed tasks to measure reviewer agreement (Turing CUA evaluation project terms).

The same terms say only approved tasks are paid, and that time spent on rework is treated as part of the task and not paid separately (Turing CUA evaluation project terms).

Their quality standard is stated plainly: two trained contributors applying the same rubric to the same task should reach materially the same conclusion, and scores must follow the rubric’s definitions rather than personal preference (Turing CUA evaluation project terms).

Review can also catch what you missed. When annotators in the computer-use study were shown an automated verifier’s reasoning, 16.6% of their outcome judgments changed, nearly all from success to failure, because the verifier pointed to a failure they had not seen (computer-use verifier paper, Microsoft Research).

Automated checks on how you work

Prolific runs optional authenticity checks that look at how a participant completes a study, not at what they write, to flag submissions not completed by the participant or completed by automated tools (Prolific, authenticity checks).

Prolific says the checks wrongly flag 0.6% of genuine submissions, that a flag is a signal for the researcher to review rather than a decision, and that repeated flags can bring a temporary restriction on taking studies (Prolific, authenticity checks).

Mercor monitors desktop activity while a contributor’s timer is running, using a tool called Insightful that tracks time, captures periodic screenshots and processes related activity signals (Mercor, Account Security and Conduct).