Free tool · no sign-up · Reads one public page on our server. Nothing is stored.
Robots.txt and AI crawler checker
One line in robots.txt can hide your whole site from Google or from AI search. Check what each crawler is allowed to read, and see the rule that decides it.
How it works
How to use the robots.txt and AI crawler checker
- Enter your domain. Add a page path if you want to test one page.
- PLUTO fetches the robots.txt your server actually sends.
- Each crawler gets allowed or blocked, plus the exact rule and group that decided it.
- Any sitemaps the file declares are listed, so you can check them next.
Questions
Frequently asked questions
Where does robots.txt live?
Only at the root of the domain, like yourbusiness.com/robots.txt. Crawlers do not look for it anywhere else.
How does a crawler decide if a page is allowed?
It uses the group with its own name if there is one, otherwise the * group. Inside that group the longest matching rule wins, and Allow wins a tie. This tool follows the same rules.
Should I block GPTBot or ClaudeBot?
It is your call. GPTBot and ClaudeBot control model training, separately from search. Blocking them does not stop you appearing in AI search. OAI-SearchBot serves ChatGPT search; Claude-SearchBot serves Claude search.
Does blocking Google-Extended hurt my Google ranking?
No. Google-Extended only controls whether your content is used for Gemini. Googlebot is the one that matters for Google Search.
Can robots.txt hide private pages?
No. The file is public and only asks crawlers politely. A noindex tag only asks search engines not to list a page; it does not keep anyone out. Private pages need a login or other access control.
More free tools
Other free SEO tools
Check your website
- Meta title and description checkerSee the title and description Google shows for any page, and what to fix.
- XML sitemap checkerFind your sitemap, count the pages in it and catch the common mistakes.
- AI search readiness checkCheck whether ChatGPT, Claude and Perplexity can read your page and quote it.
Write and plan
- Google search result previewType a title and description and see how they look on Google, desktop and phone.
- LocalBusiness schema generatorFill a short form, get the JSON-LD code that tells Google your business details.
- Keyword grouper (English and Malay)Paste a messy keyword list and get it sorted into topics and search intent.