Skip to content
HumanSourcer

HumanSourcer-Scraper

HumanSourcer-Scraper is the crawler behind this site. It collects public job listings from AI training and data-labeling platforms, so people looking for that work can compare them in one place. This page explains what it does, so you can recognise it and decide whether to allow it.

How to recognise it

Every request carries this User-Agent:

HumanSourcer-Scraper/1.0 (+https://www.humansourcer.com)

A few sites only show their listings to a web browser. For those, it uses headless Chromium, and the User-Agent is Chromium's own string with the identifier above appended, so the request still names us.

It runs mostly on GitHub-hosted servers, whose IP addresses change, so there is no fixed IP list to publish. To make our requests verifiable anyway, we are rolling out Web Bot Auth signatures. Our public key directory is at https://www.humansourcer.com/.well-known/http-message-signatures-directory.

What it collects

  • Public job listings only: title, description, stated pay, location and the link to apply.
  • Every listing shown on HumanSourcer links back to the provider's own application page. Applications happen on the provider's site, not ours.
  • It never logs in, never creates accounts, and never reads anything behind a sign-in.
  • It does not collect personal data about workers or applicants.
  • Collected listings are not used to train AI models.

How often, and how hard

  • Normally once a day per site. A failed collection is sometimes retried the same day.
  • Sites are visited one after another, not all at once.
  • Most sites take one request, or a few pages of results. Where a site needs one request per listing to show the full description, it makes at most 8 of those at a time.

What it will not do

  • Solve CAPTCHAs or click through bot challenges.
  • Rotate through proxy networks, mask its browser fingerprint, or leave its identifier out of a request.
  • Collect from paths your robots.txt disallows. We check it before adding a site as a source.
  • Work around a block. If a site blocks us, we stop and, where it makes sense, ask the provider for a feed instead.

Stopping it, or sending us a feed

The quickest way to stop it is to block the User-Agent above at your server or CDN. That takes effect on the next run, and we will not try to get around it.

We review robots.txt when we add a site, not on every run. If you change yours to exclude us, or want your listings removed, please tell us through the contact form for providers. The same form is the way to send us a listings feed, which is more reliable for both of us than crawling.