Skip to content
HumanSourcer

Unit 2: Evaluation · Lesson 2 of 4 · Free

Writing justifications that hold up in review

What justifications are used for, how long projects ask them to be, why platforms ban AI-written ones, and how to write one a reviewer can check, with a weak and a checkable example.

Published · Sources checked

Every factual claim below links to its source, beside the sentence, and we read each source at that link on the date above. Passages marked Our advice are our own judgment, and illustrations are written for the lesson. HumanSourcer is not affiliated with any company named here, and platforms change their rules: the linked page is the one that binds you.

Most comparison and rating tasks ask you to say why, and the people who use the data read those words. GDPval’s authors grouped expert justifications to work out why each model lost (GDPval paper, OpenAI), and HelpSteer3’s authors counted what annotators’ justifications mentioned (HelpSteer3-Preference paper, NVIDIA). This lesson covers what a justification is used for, how long it should be, why it has to be yours, and how to make one a reviewer can check.

What justifications are used for

In OpenAI’s GDPval, expert graders wrote detailed justifications for their rankings. The authors grouped those justifications to find why models lost: mostly instruction-following failures for some models, mostly formatting errors for another (GDPval paper, OpenAI).

HelpSteer3’s authors searched annotators’ justifications for keywords linked to each quality attribute. Close to half mentioned correctness and close to half coherence, and about one in three mentioned verbosity (HelpSteer3-Preference paper, NVIDIA).

Coding justifications mentioned verbosity more often than general ones, which the authors put down to the coding guidelines specifically asking whether code carried enough comments (HelpSteer3-Preference paper, NVIDIA).

In Turing’s computer-use evaluation project, every score of 3 or below needs a written callout and every score of 4 a justification. All comments must be specific, based on evidence, tied to the rubric and written in the project’s set structure (Turing CUA evaluation project terms).

The same terms ask evaluators to ground every judgment in what the run shows, such as the screenshot and the action taken, and ask reviewers to give evidence-based feedback on every disagreement (Turing CUA evaluation project terms).

How long

HelpSteer3 limited justifications to one or two sentences, 10 to 50 English words. Its predecessor set no limit, and its justifications ran more than twice as long on average, 483 characters against 215 (HelpSteer3-Preference paper, NVIDIA).

Written by you

Mercor prohibits using AI tools to write justifications or explanations, because the point is to assess your own reasoning rather than a model’s account of it, and says submissions evidently produced by AI will be flagged (Mercor, LLM Usage Policy).

Handshake AI goes further and also prohibits AI grammar and writing tools, unless it has authorized one in writing (Handshake AI, LLM prohibition policy).

The buyers of this data watch for it. A researcher interviewed by Epoch AI named making sure contractors are not just using LLMs as part of the grunt work of quality control (Epoch AI, FAQ on RL environments).

Illustration: The same verdict, justified two ways Written for this lesson, not taken from any platform or dataset.

The comparison is the library notice from the previous lesson: explain it to a ten-year-old in one sentence.

Weak: Response 1 is better. It is clearer, more helpful and easier to understand.

Checkable: Response 1 uses words a ten-year-old knows (“there aren’t enough workers”), where Response 2 uses “curtailed” and “staffing constraints”, so Response 2 misses the reader the prompt names. Response 1 runs to two sentences and drops the unchanged weekend hours, so it is only slightly better.

The weak version could be pasted under almost any comparison. The checkable one quotes the responses, names the instruction each breaks, and explains why the gap is small, so a reviewer can agree or disagree with each part.

Sources

In the order this lesson first cites them. Each was read at the link on the date shown.

  1. Patwardhan et al., OpenAI (arXiv). GDPval: Evaluating AI Model Performance on Real-World Economically Valuable Tasks (October 5, 2025)Checked
  2. Wang et al., NVIDIA (arXiv). HelpSteer3-Preference: Open Human-Annotated Preference Data across Diverse Tasks and Languages (May 16, 2025)Checked
  3. Turing. CUA Evaluation Contributor: Job Description and Engagement TermsChecked
  4. Mercor Talent Docs. LLM Usage PolicyChecked
  5. Handshake Help Center. Original Work and Prohibition on Usage of LLMs PolicyChecked
  6. Denain and Barber, Epoch AI. An FAQ on reinforcement learning environments (January 12, 2026)Checked

Keep going

Previous lessonJudging two responses side by sideNext lessonRubrics: writing criteria and grading against them

All lessons and the program · Graded practice for this unit, Side-by-side evaluation and Rubric grading and writing, are in an invite-only beta: join the waitlist to be invited, in signup order.

Part of the routes: First-time generalist, Domain expert, Coding and agentic, Multilingual.