crawl · crawl.robots.missing
No robots.txt found
robots.txt is the standard way to tell crawlers which paths to skip and where to find sitemaps.
Why it matters
Without robots.txt, you can't reference your sitemap or block crawlers from /admin, /api, etc.
Authoritative reference
https://developers.google.com/search/docs/crawling-indexing/robots/intro
Run a full audit of your site
Free, no signup, no crawl cap. Check this and 40+ other factors across every public page.
Audit my site →