crawl · crawl.robots.disallowAll
robots.txt blocks all crawlers
A blanket Disallow: / removes the entire site from search. Correct on staging, catastrophic on production.
Why it matters
User-agent: * with Disallow: / blocks Google completely. If this is a staging environment that's fine; on production it deletes you from search.
Example fix
# Blocks everything — remove on production\nUser-agent: *\nDisallow: /\n\n# Allow crawling instead\nUser-agent: *\nAllow: /
Authoritative reference
https://developers.google.com/search/docs/crawling-indexing/robots/robots_txt
Run a full audit of your site
Free, no signup, no crawl cap. Check this and 90+ other factors across every public page.
Audit my site →