Skip to content

SEO tool

Robots.txt Tester

Find out whether a page is open to crawlers, and which line of your robots.txt makes the decision.

Runs entirely in your browser. Nothing you paste is uploaded.

Robots.txt Tester by YegappanFree tool. Open the full page for the guide.

How to use the Robots.txt Tester

  1. Paste the contents of your robots.txt file into the left box.
  2. Enter the URL or path you want to check and choose a crawler.
  3. Read the verdict and the matching rule, and review any warnings about the file.

How robots.txt rules are applied

A robots.txt tester, also called a robots.txt validator or checker, reads your robots.txt file the way a search engine crawler does. It tells you whether a URL is allowed or blocked and warns about lines that are not valid.

A crawler first picks the group that names it most specifically. Googlebot-Image uses a User-agent: Googlebot-Image group if there is one, then falls back to Googlebot, then to *. Rules in other groups are ignored completely, which is why adding a group for one bot can accidentally release it from your general rules.

Within that group, the longest matching path wins, and Allow beats Disallow when both are the same length. * matches any run of characters and $ anchors the end of the URL, so Disallow: /*.pdf$ blocks every PDF.

Remember that robots.txt controls crawling, not indexing. A blocked page can still appear in results if other sites link to it. To keep a page out of search, allow crawling and use a noindex robots meta tag.

Common questions

How do I test my robots.txt file?

Paste the contents of the file, enter the URL you want to check and choose the crawler. The tester shows whether the URL is allowed or blocked and the exact line that decided it.

Is there still a robots.txt tester in Google Search Console?

Google retired its old robots.txt Tester. Search Console now has a robots.txt report that lists the files Google found and their errors, but it has no box for testing a URL against rules you have not published yet. This tool does that.

Does Disallow remove a page from Google?

No. It stops Google from crawling the page, but the URL can still be indexed from links. Use a noindex meta tag to keep it out of results.

Which rule wins when Allow and Disallow both match?

The rule with the longer path. If they are the same length, Allow wins.

Is the file uploaded or fetched?

No. The tester works on the text you paste and runs entirely in your browser.

Why is Crawl-delay flagged?

Google ignores the Crawl-delay directive. Bing and Yandex respect it, so it is a warning rather than an error.

Guides for the Robots.txt Tester

Related tools