SEO tool
Robots.txt Tester
Find out whether a page is open to crawlers, and which line of your robots.txt makes the decision.
Runs entirely in your browser. Nothing you paste is uploaded.
Things to check in this file
How to use the Robots.txt Tester
- Paste the contents of your robots.txt file into the left box.
- Enter the URL or path you want to check and choose a crawler.
- Read the verdict and the matching rule, and review any warnings about the file.
How robots.txt rules are applied
A robots.txt tester, also called a robots.txt validator or checker, reads your robots.txt file the way a search engine crawler does. It tells you whether a URL is allowed or blocked and warns about lines that are not valid.
A crawler first picks the group that names it most specifically. Googlebot-Image uses a User-agent: Googlebot-Image group if there is one, then falls back to Googlebot, then to *. Rules in other groups are ignored completely, which is why adding a group for one bot can accidentally release it from your general rules.
Within that group, the longest matching path wins, and Allow beats Disallow when both are the same length. * matches any run of characters and $ anchors the end of the URL, so Disallow: /*.pdf$ blocks every PDF.
Remember that robots.txt controls crawling, not indexing. A blocked page can still appear in results if other sites link to it. To keep a page out of search, allow crawling and use a noindex robots meta tag.
Common questions
How do I test my robots.txt file?
Paste the contents of the file, enter the URL you want to check and choose the crawler. The tester shows whether the URL is allowed or blocked and the exact line that decided it.
Is there still a robots.txt tester in Google Search Console?
Google retired its old robots.txt Tester. Search Console now has a robots.txt report that lists the files Google found and their errors, but it has no box for testing a URL against rules you have not published yet. This tool does that.
Does Disallow remove a page from Google?
No. It stops Google from crawling the page, but the URL can still be indexed from links. Use a noindex meta tag to keep it out of results.
Which rule wins when Allow and Disallow both match?
The rule with the longer path. If they are the same length, Allow wins.
Is the file uploaded or fetched?
No. The tester works on the text you paste and runs entirely in your browser.
Why is Crawl-delay flagged?
Google ignores the Crawl-delay directive. Bing and Yandex respect it, so it is a warning rather than an error.
Guides for the Robots.txt Tester
- Robots.txt Guide: Rules, Examples and How to Test a URLRobots.txt tells crawlers which URLs they may fetch. Learn the rules, see working examples, and test whether a URL is blocked and by which line.
- AI Crawlers and Robots.txt: How to Allow or Block AI BotsAI crawlers and robots.txt: the main AI bots and what each does, how to allow search bots while blocking training, and copy-ready rules to test.
- Robots.txt vs Noindex: Which One Should You Use?Robots.txt vs noindex explained: one controls crawling, the other controls indexing. When to use each, why combining them fails, and a decision table.
- llms.txt Explained: What It Is and Whether Your Site Needs Onellms.txt explained: what the proposed standard is, how it differs from robots.txt and sitemaps, what goes in the file and whether to add one.