User-agent: * Allow: / # Machine-readable site summary for agents/crawlers: /llms.txt # /go/ is the outbound redirect, not content. Once role pages started routing # applies through it, crawlers walking the ~6,000 role pages followed those # links too: 356 of 359 hits in one 12-hour window were referrer-less, across # 290 distinct roles. rel="nofollow sponsored" is only a hint and was plainly # being ignored. Every such hit costs a serverless invocation and a database # write, and lands a junk row in the table that measures apply intent. Disallow: /go/ # The three *-data.json files are client-side filter payloads, NOT content. # Between them they are the entire normalized dataset - every active role with # its pay, classification, and provider - served as ~4.3MB of static JSON that # anyone could fetch in three unauthenticated GETs. That is the site's core # differentiator (normalized cross-provider pay) available for bulk cloning # without so much as writing a scraper, which is strictly worse than the # per-page crawl these same files exist to make unnecessary for real visitors. # # Deliberately NOT the same call as the HTML above: the pages stay fully open # to every crawler here, including the AI ones, because being read and cited # is the goal. What's withdrawn is only the convenience of taking the whole # database in one request instead of reading the site. # # DO NOT ADD /market-data-summary.json TO THIS LIST. It looks like a fourth # member of the set and is not one: it carries aggregates only - the published # findings, histogram bins and the platform table, i.e. exactly what # /market-data/ already renders as HTML - and no per-listing rows at all. It is # crawlable on purpose, and the Dataset node on /market-data/ names it as its # `distribution`. Disallowing it would point schema.org at a URL crawlers are # told not to fetch, which is the self-contradiction that kept that field off # the node until 2026-08-10. The test for whether something belongs in the # disallow list below is whether it could reconstruct individual listings. # # SEO-safe, verified before adding: every /roles/* page server-renders its own # bounded role list, and RolesFilter.tsx treats a failed fetch of this file as # a no-op (filters simply stay scoped to the server-rendered page). No indexed # content depends on these files, and none of them appear in the sitemap. GSC # may report them as blocked page resources - that is expected and benign. # # Advisory only, and known to be: robots.txt stops well-behaved bulk ingestion, # not a determined scraper. Raising the cost past "one clean GET" needs a real # access boundary (see docs/roadmap.md, "Defensive posture"). Disallow: /roles-data.json Disallow: /listings-analytics-data.json Disallow: /pay-index-data.json # Answer-engine crawlers, stated explicitly rather than left to inherit from *. # Being cited by them is a deliberate goal here, not something merely tolerated: # this site's differentiator is cross-provider pay data that is computed from # live listings and refreshed daily, which is exactly the kind of thing these # crawlers should be reading rather than guessing at. # # NOTE: a named group does NOT inherit the rules above it. Every Disallow that # should apply to these agents has to be repeated in each group - so if /go/ or # anything else is ever disallowed for *, add it here too. User-agent: GPTBot Allow: / Disallow: /go/ Disallow: /roles-data.json Disallow: /listings-analytics-data.json Disallow: /pay-index-data.json User-agent: OAI-SearchBot Allow: / Disallow: /go/ Disallow: /roles-data.json Disallow: /listings-analytics-data.json Disallow: /pay-index-data.json User-agent: ChatGPT-User Allow: / Disallow: /go/ Disallow: /roles-data.json Disallow: /listings-analytics-data.json Disallow: /pay-index-data.json User-agent: ClaudeBot Allow: / Disallow: /go/ Disallow: /roles-data.json Disallow: /listings-analytics-data.json Disallow: /pay-index-data.json User-agent: Claude-Web Allow: / Disallow: /go/ Disallow: /roles-data.json Disallow: /listings-analytics-data.json Disallow: /pay-index-data.json User-agent: PerplexityBot Allow: / Disallow: /go/ Disallow: /roles-data.json Disallow: /listings-analytics-data.json Disallow: /pay-index-data.json User-agent: Google-Extended Allow: / Disallow: /go/ Disallow: /roles-data.json Disallow: /listings-analytics-data.json Disallow: /pay-index-data.json User-agent: Applebot-Extended Allow: / Disallow: /go/ Disallow: /roles-data.json Disallow: /listings-analytics-data.json Disallow: /pay-index-data.json # ONE sitemap is listed here on purpose, and a second one deliberately is not. # # /sitemap-roles.xml carries the role detail pages, which are currently # withdrawn from GOOGLE's index only (see src/lib/google-index-policy.ts) while # the domain is re-evaluated on its original pages. They stay fully open to # Bing, ChatGPT's crawler, Perplexity and everyone else - nothing here # disallows them, and each page carries a `googlebot`-scoped noindex rather # than a site-wide one. # # A `Sitemap:` line cannot be addressed to one crawler: every engine reads # every line in this file. Listing the role sitemap here would hand Googlebot # the exact list being withheld from it, so that file is submitted by hand in # Bing Webmaster Tools instead, and IndexNow keeps pushing new roles as they # are scraped. Sitemap: https://www.humansourcer.com/sitemap-index.xml