Test if a URL is allowed or blocked by robots.txt for Googlebot and other bots.
The robots.txt tester downloads the robots.txt of a website and tells you whether a URL is allowed or blocked for the bot you choose – Googlebot, Bingbot, AI crawlers like GPTBot and ClaudeBot, or any bot. It shows the user-agent group and the exact rule that decides.
Paste a modified robots.txt to test it before publishing. The matching follows Google's rules (RFC 9309): the most specific user-agent group applies, the longest matching rule wins, and * and $ wildcards are supported.
A clear answer with the matching rule and its line.
Google, Bing, Yandex, DuckDuckGo and AI crawlers.
Longest match, Allow on ties, * and $ wildcards.
Paste a draft robots.txt to try it.
No, it stops crawling. A blocked page can still be indexed from links. Use a noindex tag to keep a page out of Google.
Everything can be crawled. A server error on robots.txt, however, makes Google stop crawling the site.
Add a group for each bot, e.g. User-agent: GPTBot followed by Disallow: /.