Free Robots.txt Tester
Quick answer: enter a website address to see its robots.txt, which crawlers are allowed or blocked on a path, and which rule decided it. The table covers Googlebot, Bingbot and AI crawlers such as ChatGPT, Claude and Perplexity. It is free and needs no sign-up.
What the Tester Shows
- The robots.txt file itself, with line numbers
- Allowed or blocked for 19 crawlers on the path you choose, and the exact rule and line that decided it
- Checks for common mistakes: a blanket Disallow, no Sitemap line, unknown directives and rules without a User-agent
How the Rules Are Applied
The tester follows Google’s rules. A crawler uses its own group if the file has one and the * group otherwise. The longest matching pattern wins, and Allow wins a tie. The * wildcard and the $ end marker work as in Google’s documentation. If /robots.txt returns a 4xx error, crawlers treat the whole site as allowed. A 5xx error is treated as everything blocked until it is fixed.
Should You Block AI Crawlers?
It depends on what you want. Search crawlers such as OAI-SearchBot, Claude-SearchBot and PerplexityBot fetch pages so AI assistants can show and cite them, so blocking them removes you from those answers. Training crawlers such as GPTBot, ClaudeBot and CCBot collect content to train models. Blocking them is a legitimate choice and does not change how your pages rank in Google Search.
Limits
- Some crawlers, such as Perplexity-User, do not follow robots.txt. The table marks them.
- robots.txt is a request, not a security measure. Do not use it to hide private pages.
- The tester reads the file as published. It does not check what your firewall does with a crawler. The free GEO audit does that.
Frequently Asked Questions
What is a robots.txt file?
A robots.txt file sits at the root of a website and tells crawlers which paths they may fetch. Well-behaved crawlers read it before they visit your pages.
How do I test whether a URL is blocked by robots.txt?
Enter the website address and the path you want to test, for example /blog/. The table shows for each crawler whether the path is allowed and which rule decided it.
Does robots.txt remove a page from Google?
No. It stops crawling, but a blocked page can still appear in results if other sites link to it. To keep a page out of results, use a noindex meta tag or an X-Robots-Tag header and leave the page crawlable.
Does robots.txt stop AI companies from training on my content?
Only crawlers that obey robots.txt, and only from the moment they read the new rules. It does not remove content that was already collected.
What data do you save?
We save only the domain, the date and your country (worked out from your IP address, which is not stored), so we can see how the tool is used. We do not ask for your email. For the hourly limit we keep a temporary hash of your IP address for one hour.
Need help fixing what the tools find? See our SEO & GEO service or contact MaslonLabs.
