What AI-Crawler Blocking Changes for AI-Training Job Data

Corrected 14 October 2026. This note first described the change as Cloudflare blocking AI crawlers by default for everyone, and called the 374 roles on providers’ own domains “genuinely exposed” to a default-on change. Cloudflare’s announcement is narrower: the default is for new domains, on pages that display ads. The measurements below are unchanged; the framing is corrected.

On 15 September 2026, Cloudflare sets new defaults for AI traffic. Read closely, its announcement says that “for all new domains onboarding to Cloudflare,” Training and Agent crawlers are blocked by default on pages that display ads, while Search stays allowed. It does not say existing domains are switched, and a careers page or an applicant-tracking posting API does not display ads. It is a reasonable change - site owners should get to decide who trains on their content - and it still matters to any service built on reading the open web. This site is one of those: we track live AI-training and data-labeling listings across dozens of providers so the Pay Index and Market Data pages describe a real market rather than one platform’s marketing.

So we did the obvious thing before the date arrived: probed every source we read, from its own endpoint, and measured exactly how exposed the corpus is.

What we found

Twelve of the 34 sources sit behind Cloudflare, carrying 770 of the roles we index - 11.9%. But that headline number is the wrong one to worry about. The split underneath it is what matters:

• 396 of those roles are on third-party applicant-tracking platforms - Workable, BambooHR, Workday, Ashby. These are business-to-business posting systems whose entire purpose is pushing their customers’ listings out to aggregators. Turning on AI/agent blocking there would break the distribution their own customers pay for, so it is unlikely to flip.
• 374 roles are on a provider’s own domain, where the provider controls the Cloudflare dashboard directly. Any blocking here would be the provider’s own choice, so this is the set to watch - Mercor is the largest single source in it, followed by Outlier and OneForma.

The other 22 sources are on infrastructure the September change does not touch: plain origins, applicant-tracking APIs, or a different vendor’s edge entirely.

We ask nicely, and it still works

Every one of the 34 sources returns a normal 200 OK to our crawler, which sends a plain, self-identifying User-Agent - HumanSourcer-Scraper/1.0 - rather than pretending to be a browser. Not one endpoint gates on the User-Agent string: the honest identifier is served exactly the same bytes a browser gets. We keep it that way on purpose, and our robots.txt argues the same good-citizen position.

That is the real point of measuring this. If a source starts blocking an honest, clearly-labeled crawler, the right response is not to disguise it as Chrome - it is to ask the provider for a direct feed of their own listings, which is more accurate for them and for us than scraping ever is. A blocked honest crawler is a signal to start that conversation, not to evade the block.

The honesty caveat, stated up front

The first version of this probe ran from a home internet connection, while the service itself runs from a datacenter. Those two network paths do not always agree - one source in our fleet answers a residential address and returns a firewall 403 to datacenter IPs, which is a real, IP-based block that has nothing to do with the User-Agent. So we re-ran the entire probe from a datacenter (GitHub Actions) address.

The verdicts held: 32 of 34 sources answered normally, and both exceptions were already known - the IP-based block just mentioned, and the one source that already sits behind a Cloudflare challenge page (which we read with a real headless browser, not a plain request). Nothing new broke on the path that actually matters.

What happened after the date

This note was written as a dated baseline, so that the identical probe could be re-run afterward and the two diffed. We did that on 26 September 2026, from a datacenter and from a residential connection. Eleven of the 12 Cloudflare-fronted sources answered our crawler exactly as before, which fits a default that was never aimed at career pages. One source, Stellar (5 roles), began serving a Cloudflare challenge page to every client, browsers included, between 18 and 19 September. From outside we cannot tell whether that was a deliberate choice, and it is the kind of case where we would open a feed conversation rather than fight the block.

For anyone doing similar work: the reassuring finding is that the sky did not fall on 15 September. The alarming-sounding “11.9% of listings are behind Cloudflare” collapses, on inspection, to a much smaller set on domains the provider controls - and the honest way to read a market does not depend on pretending to be something you are not.