Grading expert work blind

When the output being judged is a professional’s work product, not a short answer, the person grading it has to hold professional standards and still avoid knowing who made it. Two published evaluations describe how that grading was set up: OpenAI’s GDPval, which has occupational experts compare finished deliverables (GDPval paper, OpenAI), and HealthBench, where physicians wrote the criteria a response is graded against (HealthBench paper, OpenAI). This lesson covers what the graders were given, what they were asked to produce, and how often they disagreed.

What an expert deliverable is

GDPval’s tasks each have two parts: a request, often with reference files, and a deliverable that is the work product (GDPval paper, OpenAI). The tasks come from the representative work of industry professionals who averaged 14 years of experience, across 44 occupations (GDPval paper, OpenAI).

The deliverables span formats such as CAD design files, photos, video, audio, social media posts, diagrams, slide decks, spreadsheets and customer support conversations. A task can require reading up to 38 reference files (GDPval paper, OpenAI).

Tasks take an average of 7 hours for an expert to complete, and up to several weeks at the high end (GDPval paper, OpenAI). Grading is not a skim.

How the blind comparison worked

For its open-sourced gold subset of 220 tasks, GDPval used blinded pairwise comparisons. Experts in the relevant occupation were shown the request and the reference files and asked to rank two or more unlabeled deliverables (GDPval paper, OpenAI).

On average, grading one comparison for the gold subset took over an hour, and the experts gave detailed justifications for their rankings (GDPval paper, OpenAI).

The authors say they tried to keep comparisons as blind as possible, but that model-made samples may still have been identifiable by style: one model’s outputs often used em dashes, another’s often used first-person phrasing, and one occasionally named itself. File names were scrubbed of model identifiers, but the authors chose not to alter style or content (GDPval paper, OpenAI).

What graders weigh besides correctness

The GDPval authors note that, in addition to correctness, expert graders often consider subjective factors such as structure, style, format, aesthetics and relevance (GDPval paper, OpenAI).

The authors found that many of one model’s failures were obvious formatting errors. A prompt that told the model to check its layouts by rendering files as images and to avoid nonstandard characters eliminated black-square artifacts, which had affected over half of its generated PDFs, and cut egregious formatting errors in PowerPoint files from 86% to 64% (GDPval paper, OpenAI).

For model failures, a second group of expert graders rated how bad each one was on four levels: catastrophic, bad, acceptable but subpar, and a fourth option for graders who disagreed with the original grader and thought the model’s work was better. Catastrophic meant harmful or dangerously wrong if used in real life, such as insulting a customer or giving a wrong diagnosis (GDPval paper, OpenAI).

The most common rating was “acceptable but subpar”. About 29% of ratings were bad or catastrophic, and about 3% of failures were rated catastrophic (GDPval paper, OpenAI).

How much expert graders disagree

The GDPval authors report human inter-rater agreement of 71% on the gold subset. Their experimental automated grader reached 66% agreement with the human experts, which they describe as 5% below the humans’ agreement with each other (GDPval paper, OpenAI).

When the second graders rated the failures, 23% of ratings said the model’s work was in fact better than the human’s. The authors say that figure roughly corresponds to the level of inter-rater agreement they observed (GDPval paper, OpenAI).

Experts who build the tasks are reviewed too. Each of the 1,320 GDPval tasks received an average of five human reviews, with a minimum of three, and tasks were revised between rounds (GDPval paper, OpenAI).

Grading against criteria someone else wrote

HealthBench is the other approach. Instead of ranking two deliverables, each conversation has its own rubric of criteria written by physicians: 48,562 unique criteria from 262 physicians who have practiced in 60 countries (HealthBench paper, OpenAI).

Each criterion has a point value from -10 to 10, negative for behavior that is undesirable. A grader checks each criterion independently and awards the full points or none (HealthBench paper, OpenAI).

To test whether the grading can be trusted, the authors measured agreement between the model grader and physicians on a validated subset of criteria and found it was similar to the agreement between physicians (HealthBench paper, OpenAI).

Request: “Summarize the attached lease for a new tenant in one page.”

Memo A: one page, headings for rent, term, renewal and fees, every figure matching the lease, with one clause misread as a 30-day notice when the lease says 60.

Memo B: a page and a half, a polished layout, no errors in the figures, but it leaves out the early-termination fee.

Neither is the clear winner. A grader working blind has to decide which gap matters more to a tenant: a wrong notice period or a missing fee. The justification should name the clause in each case, and say whether the layout decided anything.

Before ranking, a careful grader checks whether the memos show signs of who wrote them, and sets that guess aside.