Skip to content
AEOtic
ToolsCompareFree beta

Free tools

Free robots.txt generator for AI crawlers

robots.txt is a small file that tells web crawlers which parts of your site they may visit. AI companies run several different crawlers, and some are used to build answers while others collect content to train models.

Pick a starting point, adjust individual bots if you like, then copy or download the file. Nothing leaves your browser.

Starting point

Fine-tune by bot

Changing a bot here overrides the starting point. Picking a new starting point resets these choices.

AI search and assistant bots

AI training bots

Paths to keep crawlers out of

Paths to keep crawlers out of

Optional. These apply to every crawler that is allowed, for example /admin/ or /cart.

Preview: robots.txt

# AI crawlers: allowed
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: Claude-SearchBot
User-agent: Claude-User
Allow: /

# AI crawlers: blocked
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: CCBot
User-agent: Applebot-Extended
User-agent: Meta-ExternalAgent
User-agent: Amazonbot
User-agent: Bytespider
Disallow: /

# Everyone else
User-agent: *
Allow: /

Where does this go?

Save the file as robots.txt and place it at the top level of your domain, so it opens at https://www.example.com/robots.txt. Each subdomain needs its own file.

If your site already has a robots.txt, compare before you replace it, so you do not lose rules you rely on. Firewalls and bot protection services can also block crawlers regardless of this file.

Check how your page scores

Once the file is live, run a free scan to see how readable and citable a page is to AI answer engines, with a plain-language list of fixes.

Scan a page