Judging two responses side by side

A side-by-side task shows you one prompt and two responses and asks which is better, and usually by how much (helpful and harmless assistant paper, Anthropic) (HelpSteer3-Preference paper, NVIDIA). This lesson covers the scales published projects use, the order of priorities their guidelines set, the biases a careful comparison has to correct for, and how often trained raters agree.

What you are asked for

In Anthropic’s 2022 study, crowdworkers were shown two responses at each turn of a conversation and picked the more helpful and honest one. The interface also let them say how strong the preference was, and only comparisons stronger than the weakest option were kept (helpful and harmless assistant paper, Anthropic).

NVIDIA’s HelpSteer3 gave each annotator seven choices: response 1 much better, better or slightly better; response 2 slightly better, better or much better; or neither response valid. Each choice came with a justification of one to two sentences (HelpSteer3-Preference paper, NVIDIA).

OpenAI’s InstructGPT labelers first scored each output for overall quality from 1 to 7, then ranked all the outputs for the prompt, and were encouraged to mark ties where two outputs seemed equally good (InstructGPT paper, OpenAI).

At the expert end, OpenAI’s GDPval asked people from the relevant occupation to rank two or more unlabeled work deliverables against a request and its reference files. A single comparison took over an hour on average, and graders wrote detailed justifications (GDPval paper, OpenAI).

What matters most: the guidelines’ order

InstructGPT’s instructions told labelers that for most tasks, being truthful and harmless matters more than being helpful (InstructGPT paper, OpenAI).

There was one exception. If one output was much more helpful, only slightly less truthful or harmless, and the task was not high-stakes, with loan applications, therapy, and medical or legal advice given as high-stakes examples, the more helpful output should be rated higher (InstructGPT paper, OpenAI).

When two outputs were similarly helpful but failed in different ways, the instructions asked which one was more likely to harm the person using it, and to mark a tie if that was unclear (InstructGPT paper, OpenAI).

How much better

HelpSteer3 took the average of the three annotators who agreed most closely and rounded it to get the final label. Items where even those three were more than two points apart were dropped, on the reasoning that the task was too subjective or the annotators had missed something important (HelpSteer3-Preference paper, NVIDIA).

Teams can calibrate strength differently. HelpSteer3’s authors saw one vendor’s annotators choose strong judgments more often while another vendor’s more often chose “slightly better”. They suggested, without testing it, that the first vendor may have trained its annotators toward stronger judgments, or that the second set of prompts was easier (HelpSteer3-Preference paper, NVIDIA).

Equal is not the same as acceptable

It is easy to read a tie as a verdict on quality, but “about the same” and “good enough” are different questions. A study of chat-model responses to Russian-language and coding instructions kept them apart. Judges could say that the candidate was better, that the baseline was better, that both responses were equally good, or that both were equally bad, and they did not see which model wrote which (side-by-side evaluation paper, MWS AI).

Looks right versus follows the instruction

LLMBar’s authors curated 419 pairs in which one output follows the instruction and the other does not, but may have deceptive qualities such as a more engaging tone (LLMBar paper, Princeton). In one example the authors give, the right-hand output follows the instruction, yet both LLM evaluators and humans are often biased toward the left-hand one because of its more engaging tone (LLMBar paper, Princeton).

Biases a careful comparison corrects for

Longer is not better by default. A study of reward models found their scores strongly correlated with response length, a bias it traced back to the preference data the models learned from (length correlations in RLHF paper).

Anthropic did not expect its crowdworkers to fact-check, and reports that they often preferred responses containing links that did not work (helpful and harmless assistant paper, Anthropic).

In the same study, the preference model’s three confidently wrong outliers on one evaluation set all involved the same trade: an answer in which the model admitted it did not know, against a thorough, sophisticated-sounding one containing misleading information. The paper calls them failures of honesty and accuracy (helpful and harmless assistant paper, Anthropic).

Blind comparisons are not always blind. GDPval’s authors note that experts may still have recognized which model wrote a deliverable from its style, such as one model’s habit of using em dashes or another’s first-person phrasing (GDPval paper, OpenAI).

What AI judges get wrong about order

Position bias is an LLM’s tendency to favour certain positions over others (MT-Bench and Chatbot Arena paper, UC Berkeley). A study of 15 LLM judges separates three things that are easy to run together: whether a judge repeats itself on the same query, whether it prefers the same solution after the order is swapped, and whether it favours one position (position bias in LLM judges paper, Dartmouth). The authors report that position bias is not due to random chance, that it varies significantly across judges and tasks, and that it is strongly affected by the quality gap between the solutions (position bias in LLM judges paper, Dartmouth).

Agreement figures depend on which votes they count. On MT-Bench’s first-turn questions, GPT-4 judging pairs agreed with humans 66% of the time when ties and verdicts that changed with the order were counted as ties, and 85% when only non-tie votes were counted (MT-Bench and Chatbot Arena paper, UC Berkeley). The second figure covers fewer votes, 859 against 1,343, and human-human agreement on the same two measures was 63% and 81% (MT-Bench and Chatbot Arena paper, UC Berkeley).

How often careful raters agree

Expect to disagree with other careful raters some of the time. InstructGPT’s training labelers agreed with each other about 72.6% of the time (InstructGPT paper, OpenAI), and GDPval’s expert graders 71% of the time (GDPval paper, OpenAI).

When the preference itself is clear, experts agree more. LLMBar kept only pairs with an objective preference about instruction following and reports an expert agreement rate of 94%, which its authors say is significantly higher than any of the earlier benchmarks they compare with (LLMBar paper, Princeton).

Prompt: Explain this notice to a ten-year-old in one sentence: “From June, the library will close at 5 p.m. on Fridays because of staff shortages. Weekend hours are unchanged. Books can be returned through the outside drop box at any time.”

Response 1: Starting in June, the library will close early on Fridays, at 5 p.m., because there aren’t enough workers. You can still drop off books any time.

Response 2: Due to staffing constraints, Friday operating hours will be curtailed to 17:00 from June, with weekend hours and the external return facility unaffected.

Verdict: Response 1 is slightly better.

Why: Response 2 keeps to one sentence and covers all three points, but “curtailed”, “staffing constraints” and “17:00” are hard going for a ten-year-old, so it misses the reader the prompt names. Response 1 uses plain words but runs to two sentences and leaves out the unchanged weekend hours. Each breaks one instruction; Response 1 breaks the one that matters less for this reader, so the gap is small.