Starts a rule group for one crawler name or * for a broad group.
Web diagnostics
Robots.txt checker
Fetch a public site's robots.txt file and inspect crawler groups, allow/disallow rules and sitemap declarations.
Quick guide
How robots.txt works
robots.txt is a crawler instruction file at the root of a host. It can suggest which paths compliant crawlers should or should not request and can advertise sitemap locations.
robots.txt is not an access-control mechanism. A disallowed URL can still be reachable by anyone who knows the address, and search engines may sometimes know that a URL exists even when crawling is blocked.
Describe paths a compliant crawler should avoid or may crawl.
Points crawlers at one or more XML sitemap files.
IPGetter fetches the public robots.txt file, parses user-agent groups, Allow and Disallow directives, and any Sitemap declarations it contains.
- Checking whether crawlers are blocked
- Finding declared sitemap URLs
- Reviewing robots.txt after a deployment
Terminal shortcut
Prefer the command line?
Fetch and summarize a public robots.txt file.
> robots example.com
Run in Terminal ->