Skip to content
HumanSourcer

For researchers · Free tool

Side-by-side annotator

Load a JSONL file of response pairs, label them from the keyboard, and export results your training code can read. Load several annotators' results and get an agreement report with an adjudication queue. Nothing leaves your browser.

Built for measurement

  • The A/B order is randomised per annotator and item, recorded, and undone in the export, so position bias can be measured.
  • 7-point scale, reasons by dimension, confidence, and evidence spans taken from your text selection with exact offsets.
  • Agreement across files: Krippendorff's alpha with a bootstrap interval, Gwet's AC1, pairwise weighted kappa, Dawid-Skene annotator accuracy, and the items most worth adjudicating.

Private by construction

  • Files are read with your browser's file API. Progress autosaves in this browser only; results leave as files you download.
  • The offline copy carries a Content Security Policy that forbids network requests, and its build fails if a network call appears in the code.
  • Exports: hs.result/1 JSONL (with a published JSON Schema) and TRL-style preference pairs, ties dropped and margins kept.

The annotator

Annotate a file

Load a JSONL file of side-by-side items (hs.item/1, or lines like {"id","prompt","response_a","response_b"}). Nothing you load or write leaves this browser; you download your results when you are done.

Compare annotators

Load two or more results files from this annotator. If you also loaded the items file above and it has reference answers, each annotator is scored against them too.

Practice engine 0.1.0, sbs-ui-2. MIT licensed.

Use it offline

Download the annotator as one HTML file (about 75 KB). Open it from your disk in any modern browser; it needs no installation and no connection.

SHA-256 3cbdaad8eb9a55320a973829ec7bfa3a0638f05bf9db154de183102f2948a60c · practice engine 0.1.0

Item files can be hs.item/1 records or plain lines of {"id", "prompt", "response_a", "response_b"}. Want to try it first? Use the free samples on the side-by-side practice page.

How the statistics are checked

Each statistic is tested against published worked examples and against the standard Python packages for it (krippendorff, statsmodels, irrCAC, SciPy, crowd-kit) on fixed inputs. As an end-to-end check, the engine reproduces the judge-agreement table of the MT-Bench paper (Zheng et al. 2023, Table 14) from its released votes: every vote count exactly, and every rate to the published percentage. The engine is MIT licensed.