Writing hard prompts an expert can verify

Some expert work is not rating a model’s answer. It is writing the question the model is tested on, in your own field, so that a person with the right training can say whether an answer is right. Two published projects document how they asked experts to do that: the GPQA science questions (GPQA paper) and Humanity’s Last Exam (Humanity's Last Exam paper). This lesson covers what they required of a question, how they checked it, and how often their own checks disagreed.

What a good question had to be

GPQA is 448 multiple-choice questions written by people who had or were pursuing a PhD in biology, physics or chemistry (GPQA paper). Its writers were asked for questions that were both very difficult and had a clear, objective answer that experts in the field would agree on (GPQA paper).

The writer instructions list nine requirements, among them clarity, objectivity, difficulty, search resistance, and being stand-alone, meaning other experts could answer correctly even if they were not shown the answer choices (GPQA paper).

Humanity’s Last Exam, which holds 2,500 questions across dozens of subjects, asks for questions that are precise, unambiguous, solvable and non-searchable. It also requires original work or a non-trivial synthesis of published information, with a short answer that is easy to verify (Humanity's Last Exam paper).

Every HLE submission includes the question, the answer, a detailed rationale explaining the solution, the academic subject, and the contributor’s name and institution, to keep contributors accountable (Humanity's Last Exam paper).

Why “searchable” matters

GPQA’s writers were told to pretend to be a non-expert, think about what they would search the web for, run those searches, and see whether the question turned out easier than they had expected (GPQA paper).

The project then measured it. Skilled non-experts with PhDs in other fields, free to use any internet resource except a language model assistant, spent 37 minutes on average per question and reached 34% accuracy, where 25% is chance (GPQA paper).

HLE adds a second filter. Before submission, each question is tested against frontier language models, and a question is rejected if the models can answer it correctly (Humanity's Last Exam paper).

When a model gives the correct answer for faulty reasons, HLE’s authors encourage changing the question’s parameters, such as the number of answer choices, so that a lucky guess is less likely to count (Humanity's Last Exam paper).

How the questions were checked

GPQA ran four stages. An expert wrote the question. A first expert validator answered it and gave feedback. The writer revised it, and a second expert validated the revised version. Finally three non-experts tried to answer it (GPQA paper).

Experts in the matching field reached 65% accuracy on the final questions, and 74% once the mistakes they themselves identified in retrospect were set aside. The authors say many of those errors came from the sheer difficulty of the question and not from disagreement about the answer (GPQA paper).

HLE uses two review stages after submission. An initial round of feedback from several graduate-level reviewers comes first. Then the organizers and an expert reviewer approve the question, or do not (Humanity's Last Exam paper).

The error rate is not zero

Before release, HLE’s authors audited two samples of 200 questions each, with expert reviewers and a rebuttal phase for the original authors. After removing or revising questions of the kinds the first audit found, they estimated an expert disagreement rate of 15.4% for the public set (Humanity's Last Exam paper).

On a targeted review of a biology, chemistry and health subset, the rate was about 18% (Humanity's Last Exam paper). The authors say disagreement tends to be higher in health and medicine, and that this is in line with what is seen in other challenging expert-grade benchmarks (Humanity's Last Exam paper).

The first audit round looked for common kinds of imprecise question: open-ended formats, reliance on rounded numerical values, and submissions from authors with low acceptance rates (Humanity's Last Exam paper).

How the work was set up and paid

GPQA hired 61 contractors through Upwork, required each to have completed or be in a PhD program in the field, and paid mostly through bonuses that depended on quality. A writer earned a base payment per question, a bonus for each expert validator who answered correctly (to reward objective questions) and a bonus for each non-expert who answered incorrectly (to reward hard ones) (GPQA paper).

The authors estimated an average of about $95 an hour for that project, with a maximum possible of $150 (GPQA paper). That is one project’s estimate from 2023, not a rate any current provider offers.

The writer instructions for that project prohibit AI assistants at any stage, and say that anyone believed to be using them, or relying on other people for help, would not continue (GPQA paper).

First draft: “Which treatment is best for a patient with stage 2 hypertension?”

Why it fails the checks above: “best” depends on the patient, the guideline and the year, so experts would split. A search would also return a guideline summary.

Revised: a specific patient vignette with age, a named comorbidity, two prior medications that failed and a lab value, asking for the single next step that a named, current guideline gives for that exact case, with the answer stated in one short phrase.

What the revision does: it fixes the context so there is one defensible answer, names the standard a reviewer can check it against, and keeps the answer short enough to verify at a glance. It still needs a written rationale, and a search test: if the answer appears in the first guideline summary, it is too easy.