HumanSourcer Gym
Models train in gyms. So can you.
AI labs train models in RL gyms: a task, a grader, a score, repeat. This is the same loop for the people doing the grading.
- Each drill has a human reference answer.
- See where you and the reviewers differed.
- Practise the skills here, then nail the screening for your next role.
The drills
- Side-by-side evaluationPublicPick the better of two AI answers, then say why.
- Image annotationBetaDraw boxes on real photos, in five-photo skill sets.
- Rubric grading and writingBetaGrade answers against a rubric, or write the checklist yourself.
- Agent trajectory reviewAlphaRead an AI agent's steps and grade the run.
Beta drills are open to members who turn on beta programs, at signup or in their account. Alpha drills are still being built.
Free lessons, no account needed
See all 19 lessons- Foundations3 lessonsHow human work becomes model training data, how that work is checked, and the rules that get contributors removed.
- Evaluation4 lessonsJudging model responses: side-by-side preference, written justifications, rubrics and factual checks.
- Agentic work4 lessonsAgents that act on computers: what a trajectory is, how demonstrations are recorded, how runs are graded, and how tasks are made hard to game.
- Multimodal and physical3 lessonsAnnotating images, capturing video and first-person footage for robotics, and the consent questions recording raises.
- Expert work2 lessonsFor people with a degree or a profession: writing hard tasks in your field and grading expert work without knowing who produced it.
- Getting in and getting paid3 lessonsWhich network fits your background, what each screening evaluates according to public sources, and how pay and tiers work once you are in.
What the work looks like now
Every open listing we track, classified by the kind of work it asks for. Each distinct ad counts once, so a provider posting the same ad in fifty cities does not skew it. An ad can ask for several kinds of work, so the shares add up to more than 100%.
| Work | Share of ads | New ads / older | Median stated rate | In the program |
|---|---|---|---|---|
| Multilingual / translation | 39.6% | 23% / 47.5% | $30/hr | Multilingual track |
| Expert domain (STEM, medical, legal, finance) | 20.4% | 27% / 17.6% | $90/hr | Expert task and verifier |
| Evaluating AI responses | 13.7% | 11.1% / 13.9% | $70/hr | Side-by-side evaluation |
| Audio / speech | 13.4% | 7.8% / 12% | $33/hr | Not yet |
| Coding / software engineering | 8.2% | 8.2% / 6.5% | $75/hr | Coding and agentic track |
| Rubrics / grading | 7.6% | 8.3% / 7.7% | $75/hr | Rubric grading and writing |
| Side-by-side / preference ranking | 5.2% | 1.7% / 7.4% | $33/hr | Side-by-side evaluation |
| Prompt writing / red-teaming / safety | 4.5% | 4.2% / 3.9% | $60/hr | Not yet |
| Agents / computer use | 3.5% | 10.5% / 1.2% | $78/hr | Agent trajectory review, Trajectory capture |
| Search / ads / maps rating | 2.8% | 2.5% / 3.3% | $40/hr | Not yet |
| Video / egocentric / robotics | 1.7% | 2.1% / 1.4% | $32/hr | Multimodal and field capture track |
| Image annotation | 1.6% | 1.4% / 1.8% | $70/hr | Image annotation |
As of October 31, 2026: 3,976 distinct ads from 8,078 open listings. "New ads / older" compares the share of ads first seen in the last 30 days with the share among older ads. The rate is the median of distinct stated hourly rates in US dollars; listings that state no rate are left out, and a rate is shown only when at least 5 distinct rates support it.
How the program is organized
Six units. Every lesson is free and every claim in it links to its source. Practice sits beside the unit it belongs to. 19 of 19 lessons are available now. Each practice module below shows its current access: public samples, invited practice, a restricted pilot or planned work.
- Unit 1
Foundations
How human work becomes model training data, how that work is checked, and the rules that get contributors removed.
- How human data trains AI modelsFree lesson, live
- How your work is checked: gold tasks, calibration and agreementFree lesson, live
- What gets contributors removedFree lesson, live
- Unit 2
Evaluation
Judging model responses: side-by-side preference, written justifications, rubrics and factual checks.
- Judging two responses side by sideFree lesson, live
- Writing justifications that hold up in reviewFree lesson, live
- Rubrics: writing criteria and grading against themFree lesson, live
- Checking a response for factual errorsFree lesson, live
- Side-by-side evaluationFree samples; free practice with an account
- Rubric grading and writingFree grading samples; writing practice in beta
- Unit 3
Agentic work
Agents that act on computers: what a trajectory is, how demonstrations are recorded, how runs are graded, and how tasks are made hard to game.
- What an agent trajectory isFree lesson, live
- Recording computer-use demonstrationsFree lesson, live
- Grading an agent run step by stepFree lesson, live
- Tasks, verifiers and reward hackingFree lesson, live
- Agent trajectory reviewAlpha: being built
- Trajectory captureAlpha: being built
- Unit 4
Multimodal and physical
Annotating images, capturing video and first-person footage for robotics, and the consent questions recording raises.
- Image annotation conventionsFree lesson, live
- Video and egocentric captureFree lesson, live
- Consent and privacy when you recordFree lesson, live
- Image annotationBeta: open to members with beta programs on
- Unit 5
Expert work
For people with a degree or a profession: writing hard tasks in your field and grading expert work without knowing who produced it.
- Writing hard prompts an expert can verifyFree lesson, live
- Grading expert work blindFree lesson, live
- Expert task and verifierPlanned
- Unit 6
Getting in and getting paid
Which network fits your background, what each screening evaluates according to public sources, and how pay and tiers work once you are in.
- Which network fits your backgroundFree lesson, live
- What each screening evaluates, from public sourcesFree lesson, live
- How pay and tiers workFree lesson, live
Pick a route
Routes through the units for different starting points. The figures under each are the job ads in our data that ask for that kind of work.
First-time generalist
You have no AI-training experience yet and want to know what the work is, how it is checked and which network to apply to first.
Route: Foundations, Getting in and getting paid, Evaluation; practice: Side-by-side evaluation, Rubric grading and writing
623 ads from 25 networks, median stated rate $70/hr.
Generalist response rating is a shrinking share of new listings in our data, so this track ends with the rubric work that carries higher stated pay.
Domain expert
You have a degree or professional experience in a field such as medicine, law, finance, science or engineering.
Route: Foundations, Evaluation, Expert work, Getting in and getting paid; practice: Side-by-side evaluation, Rubric grading and writing, Expert task and verifier
813 ads from 22 networks, median stated rate $90/hr.
Expert listings carry the highest stated pay in our data and are a growing share of new listings.
Coding and agentic
You write software, or you are comfortable checking what an AI agent did on a computer, step by step.
Route: Foundations, Evaluation, Agentic work, Getting in and getting paid; practice: Rubric grading and writing, Agent trajectory review, Trajectory capture
417 ads from 24 networks, median stated rate $73/hr.
Agent and computer-use work is the fastest-rising work type in our data, from a small base.
Multilingual
You are fluent in a language other than English and want evaluation work in it.
Route: Foundations, Evaluation, Getting in and getting paid; practice: Side-by-side evaluation, Rubric grading and writing
1,573 ads from 24 networks, median stated rate $30/hr.
Language work is the largest category by number of listings and the lowest paid by stated hourly rate in our data.
Multimodal and field capture
You want image, video or recording work, including filming everyday tasks for robotics data.
Route: Foundations, Multimodal and physical, Getting in and getting paid; practice: Image annotation
123 ads from 23 networks, median stated rate $40/hr.
Image annotation is a small and flat share of listings; first-person video capture is small but rising.
How practice and feedback work
Current evaluation and rubric-grading samples use published human labels. Image tasks identify their photo references or synthetic geometry. Rubric writing uses self-review without a quality score. Planned modules below describe the feedback we intend to build; they are not available yet.
| Module | What you do | Feedback and scoring | Ads asking for it |
|---|---|---|---|
| Side-by-side evaluationFree samples; free practice with an account | Compare two model responses to the same prompt, rate your preference on a 7-point scale, mark the reasons and write a justification. | Agreement within one point of HelpSteer3's human reference label, the original annotators' average rounded with ties to even. The report compares your agreement with individual annotators to their agreement with each other and shows guessing baselines. Items where annotators preferred opposite responses are marked as split and excluded from agreement and completion counts. | 623 (15.7%) |
| Rubric grading and writingFree grading samples; writing practice in beta | Grade responses against published, expert-written rubric criteria, then write a rubric of your own for a new prompt. | Introductory grading compares your decision with the original physician labels. Items physicians split on are unscored. The restricted writing pilot uses self-review against published human rubrics, with no quality score; finishing every step of all three exercises earns a completion record. Longer graded sessions are not open. | 302 (7.6%) |
| Agent trajectory reviewAlpha: being built | Read a recorded agent run step by step, find the first step that went wrong, classify the error and judge whether the task succeeded. | Scored against the verdict of the public benchmark the run comes from, which checks success automatically. | 140 (3.5%) |
| Trajectory captureAlpha: being built | Complete a task in a mock app inside the page while every action is logged with a one-line reason, the way computer-use demonstrations are recorded. | A deterministic checker scores the end state and the trajectory. Nothing you do leaves your browser except the score. | 140 (3.5%) |
| Image annotationBeta: open to members with beta programs on | Draw and label bounding boxes on images from a public dataset, following written labeling rules. | The restricted pilot compares your boxes with the reference using geometric overlap. Photo tasks name their source and license; constructed-rectangle tasks are labeled as synthetic. Feedback is not a skill assessment and does not award completion credit. | 62 (1.6%) |
| Expert task and verifierPlanned | Write a hard task in your own field together with a check that can verify the answer, then test whether a model could game it. | Scoring design will be developed once the earlier practice modules have real learner data to calibrate against. | 813 (20.4%) |
What to expect
- Not a requirement
- No AI-training network currently requires this program. The free lessons help you understand the work and prepare to apply; each provider sets its own requirements. The FTC's advice on job scams is plain: "Don't pay for the promise of a job." Paying for practice here promises you no job at any network.
- Recognition is a goal
- Where a module offers a completion record, it records the practice you completed. No network recognizes our completion records yet. We want them to become useful evidence of your work for networks, but future recognition is not promised.
- Not a provider's screening
- We never reproduce a network's screening questions or the project guidelines its contributors work under. Lessons that describe a screening say only what the network itself has published, with a link.
- Not scored by a model
- Evaluation and rubric-grading feedback uses published human labels and shows disagreements. Image feedback measures overlap with its stated reference. Rubric writing is unscored self-review. Any AI commentary is labeled separately and does not determine your score or completion.
- Your answers and your choices
- Ordinary practice answers are stored to score them and show your progress. They are not sold or used to train a model. If optional contributions are offered later, training, commercial licensing and publication will need separate permissions. Declining will not affect your access or scores. Earlier work stays protected unless you explicitly authorize its reuse. Read the privacy policy and terms of use.
- A growing practice suite
- Our aim is to support evaluation, rubric writing, annotation and agent workflows with useful practice and clear feedback. Public samples and restricted pilots are steps toward that goal. Each module above states what you can access now. Planned work has no promised release date.
Start practising
Meanwhile, read the free lessons, no signup needed: start with the first one. The free tools are live now: browse the full network directory, check what these roles actually pay, or see open roles today.
Frequently asked questions
Is HumanSourcer Gym available yet?
19 of 19 free lessons are available without an account. Side-by-side evaluation: Free samples; free practice with an account. Rubric grading and writing: Free grading samples; writing practice in beta. Agent trajectory review: Alpha: being built. Trajectory capture: Alpha: being built. Image annotation: Beta: open to members with beta programs on. Expert task and verifier: Planned. Anyone aged 18 or over can create a free account; beta modules open to accounts with beta programs on.
What will it cost?
The lessons are free and will stay free. Graded practice will be paid, and its price is not set yet. Waitlist members get founding-member pricing locked in before the public price applies.
Will finishing it get me hired?
No, and nothing here should be read as saying so. No AI-training network currently requires this program. No network recognizes our completion records yet. It explains how the work is done and checked, and lets you practice it against data labeled by people, so you can apply knowing where your judgment stands. Hiring decisions are the network's alone.
Do I need experience with AI training or data annotation work?
No. There are routes for people applying to their first network and for people with a profession or a degree who want the better-paid expert work. Pick the route that matches your background.
How do you decide what to teach first?
From the job ads themselves. Every night we classify each open listing we track by the kind of work it asks for, counting each distinct ad once so a provider posting the same ad in fifty cities does not skew it. As of October 31, 2026 that is 3,976 distinct ads from 8,078 listings. The modules ship in order of demand and pay, weighed against how reliably we can score them.
How is this different from the free provider directory?
The free directory (76 networks) tells you what exists and what each network is known to pay. HumanSourcer Gym covers the work itself: how evaluation, rubric, agent and annotation tasks are done, how they are checked, and how your answers compare with trained human annotators.
When does it launch?
The free lessons and public samples are available now. We are expanding the practice suite in stages, with restricted pilots before wider access. There is no fixed date for the whole suite. New modules need suitable content, clear usage rights and reviewed feedback before wider release. Waitlist members hear first, in signup order.