Graded practice · Unit: Evaluation
Side-by-side evaluation practice
Compare two model responses to the same prompt, rate your preference on a 7-point scale, mark the reasons and write a justification.
Pairwise and response-evaluation work appears in 467 distinct ads from 20 platforms we track, at a median stated rate of $65/hr (as of October 17, 2026).
Try five free samples
Real items from a public preference dataset, scored the way the graded sets are. Click into the tool to use the keyboard: 1 to 7 to score, E to mark selected text as evidence, J to write, Ctrl or Cmd with Enter to submit, ? for the rest. Your answers stay in this browser.
Items from HelpSteer3 (NVIDIA), CC BY 4.0; prompts from WildChat-1M (ODC-BY). These are English items from the General and STEM domains of its preference validation set. Changed: the two responses may be shown in either order. The two model responses are the material being judged.
How your answers are scored
- The reference for each item is the original annotators' average score, rounded to the nearest integer; exact halves round to the even integer. You agree with it when your score is within 1 point of it on the 7-point scale.
- When the original annotators preferred opposite responses, the item is shown as "experts split", with their reasons, and is excluded from agreement and completion counts.
- The report also compares your agreement with the individual annotators to their agreement with each other, using the same within-1-point rule on the same non-split items, and shows what guessing would score. The annotators' agreement is an upper bound, because the dataset kept its most-agreeing annotators.
- Your justification gets a checklist, kept separate from agreement: 10 to 50 words, says which response, says what differs and points at evidence (a highlighted span, a quote of 4 or more words, or a matching number or code-like token). The typed-versus-pasted check is advisory; it cannot detect AI authorship. Neither the checklist nor your selected reason dimensions change your agreement score.
- A set has 20 items and a completion record needs 40 non-split scored answers, with no minimum agreement rate. Position bias is checked at the end of each set of 20, never item by item. Reading faster than 1,000 words a minute is flagged.
- After you answer, you also see what two AI judges said about the item, labelled as judges under evaluation. Their verdicts do not change your score or count toward completion.
Scorer sbs-1.0.0
Graded sessions
Sessions of 20 items, with your progress and a completion record, open to the waitlist first, in signup order, and are free during the beta. Join the waitlist. Invited already? Sign in.
Running your own annotation project? The same annotator works on your own files, privately in your browser.
What this is not
- No platform requires this practice, and none has endorsed it. HumanSourcer is not affiliated with any platform or dataset named here.
- It is a completion record, not a credential, and it promises no work or pay.
- Your answers are used to score and show your progress. They are never sold or used as training data. Optional contributions, if ever offered, would need your separate permission; see the privacy policy.
- Be wary of anyone who charges you for the promise of AI-training work. The FTC's advice: Don't pay for the promise of a job.