Robots.txt and Sitemap Checker

Two small files tell search engines where they may go and what to index: robots.txt and sitemap.xml. Enter a website and we read both, flag rules that block crawlers by mistake, validate the sitemap and let you test whether a specific page is allowed.

What robots.txt does

A robots.txt file sits at the top of a website (for example example.com/robots.txt) and asks crawlers which paths they may fetch. It is a request that well-behaved crawlers such as Googlebot and Bingbot follow, not a lock. It does not hide pages from people, and it is not a security measure. A page blocked in robots.txt can still appear in search results if other sites link to it, just without a description. To keep a page out of results, use a noindex tag and leave the page crawlable so the tag can be read.

What we check in robots.txt

• The file exists and is readable: a missing file means everything may be crawled. A server error (5xx) is serious, because Google can pause crawling the whole site while robots.txt fails.
• The whole site blocked: Disallow: / under User-agent: * is the classic mistake left over from a staging site. We tell you straight away if it applies to all crawlers or to Googlebot alone.
• Syntax problems: lines without a colon, unknown directives, rules before any User-agent line, paths that do not start with a slash and relative Sitemap addresses.
• An HTML page returned instead of text: some servers answer /robots.txt with their home or error page, so crawlers cannot read any rules.
• Sitemap lines: whether the file points crawlers to your sitemap.

How Google decides between rules

If several rules match a path, the most specific one wins, which means the one with the longest matching pattern. If an Allow and a Disallow match equally, Allow wins. The star (*) matches any run of characters and a dollar sign ($) marks the end of the address, so Disallow: /*.pdf$ blocks every address ending in .pdf. A crawler that has its own group, such as Googlebot, follows only that group and ignores the rules for everyone else. The path tester above applies exactly these rules, so you can try an address before you publish a change.

What we check in the sitemap

We read the first sitemap listed in robots.txt, or the usual address /sitemap.xml if there is none, and check that:

• the file is valid XML with the standard sitemap namespace (one unescaped ampersand is enough to break it);
• it is a URL list (urlset) or an index of sitemaps (sitemapindex), and for an index we also check its first child;
• every address is a full http or https URL, listed only once, and all on one host;
• lastmod dates are valid and not in the future, and priority values are between 0 and 1;
• the file stays within the limit of 50,000 URLs (and 50 MB uncompressed) per sitemap.

Sitemaps help search engines discover pages, but they do not guarantee indexing. List only the final, canonical, indexable addresses. Check how your pages redirect with the redirect checker and what is on them with the SEO meta tag checker.

About this tool

Our server fetches robots.txt (over HTTPS first, then HTTP) and the sitemap, with size limits of 500 KB and 2 MB, compressed sitemaps included. The reading and checking run in your browser. The tool only reaches public websites on the standard web ports, is rate limited and stores nothing. Large sitemaps are read in part, and we say so when that happens.

Frequently asked questions

Where is the robots.txt file?

Always at the root of the site, for example https://example.com/robots.txt. Crawlers do not look for it anywhere else, and each subdomain needs its own file.

Does Disallow in robots.txt remove a page from Google?

No. It stops crawlers from fetching the page, but the address can still be listed if other sites link to it. To remove a page from results, use a noindex tag or the removal tools, and let the page be crawled so the tag is seen.

What does Disallow: / mean?

It blocks the entire site for the crawlers in that group. Under User-agent: * it blocks all well-behaved crawlers. It is meant for sites that should not be indexed yet and is a common mistake when it is left in place after launch.

Do I need a sitemap?

Small, well linked sites can be crawled without one, but a sitemap helps search engines find new and deep pages faster. It is especially useful for large sites, new sites and pages with few links pointing to them.

How many URLs can one sitemap have?

At most 50,000 URLs and 50 MB uncompressed. Larger sites split their URLs over several sitemap files and list them in a sitemap index file.

Should the sitemap list redirected or blocked pages?

No. List only final addresses that return a 200 status, are allowed by robots.txt and can be indexed. Listing redirects, errors or noindex pages sends mixed signals to search engines.

Related tools and guides

• SEO meta tag checker: title, description, Open Graph.
• Redirect checker: trace every hop.
• Is it down? checker: status and response time.
• Security headers checker: grade HSTS, CSP and more.

← Run the speed test · All tools · All guides