Free Robots.txt Generator with AI Bot Rules

Quick answer: set your rules, choose which AI bots to allow or block, and copy or download a ready robots.txt file. It lists the main AI crawlers from OpenAI, Anthropic, Perplexity, Google, Apple and more. It is free, runs in your browser and needs no sign-up.

Quick presets:

AI bots

Choose a rule per bot, or leave it on Default to follow the rules above. Purposes follow each company’s published bot documentation, which can change.

BotCompanyPurposeRule

    Everything runs in your browser, nothing is sent to us or saved. Save the file as robots.txt in the root of your site and test it with our robots.txt tester.

    What the Generator Does

    • Builds a robots.txt with your disallow and allow paths and your sitemap URLs
    • Lets you allow or block each major AI bot, with presets for blocking training bots only or all AI bots
    • Repeats your disallow paths for AI bots you explicitly allow, because a bot with its own group ignores the general rules
    • Warns when a choice could hurt your visibility, such as blocking all crawlers or an AI search bot

    Training, Search and User Bots

    AI companies run different bots for different jobs. Training bots collect pages to train models, search bots index pages so an AI search product can cite them, and user bots fetch a page when a person asks the assistant about it. Many sites block training bots and allow search bots, so they can still be cited in AI answers. This is a business choice, and the tool leaves it to you.

    Limits

    • robots.txt is a request, not a lock. Reputable crawlers follow it, others may not.
    • It blocks crawling, not indexing. To keep a page out of search results, use a noindex tag.
    • The list of bots and their purposes follows each company’s published documentation at the time of writing and changes often.
    • Always test the file before you publish it, because a wrong rule can block your whole site.

    Frequently Asked Questions

    What is a robots.txt file?

    A robots.txt file is a text file at the root of your site (example.com/robots.txt) that tells crawlers which paths they may or may not fetch. Well-behaved crawlers such as Googlebot and the AI bots listed here follow it.

    Does robots.txt hide pages from Google?

    No. It stops crawling, not indexing: a blocked URL can still appear in results if other sites link to it. To keep a page out of search results use a noindex tag or password protection instead, and do not block that page in robots.txt, or crawlers will not see the tag.

    Should I block AI bots?

    It depends on what you want. Training bots such as GPTBot and ClaudeBot collect content to train models. Search bots such as OAI-SearchBot and PerplexityBot are used to find and cite pages in AI answers, so blocking them can reduce your visibility there. Many sites block training bots and allow search bots.

    What is the difference between training, search and user bots?

    Training bots crawl pages to build model training data. Search bots index pages so that an AI search product can cite them. User bots fetch a page when a person asks the assistant about it. The labels in the table follow the companies’ own documentation and can change.

    Is Google-Extended a crawler?

    No. Google-Extended is a robots.txt token. Crawling is done by Google’s normal crawlers, and the token only tells Google whether your content may be used for Gemini training and grounding. Blocking it does not affect your Google Search ranking.

    Do all bots obey robots.txt?

    Reputable crawlers do, but robots.txt is a request and not a lock. Bots that ignore it have to be stopped at the server or firewall.

    Do you save my file?

    No. The file is generated in your browser and nothing is sent to our server.

    Already have a robots.txt? Check it with the robots.txt tester or see whether AI bots can reach your site with the AI crawler checker.

    Need help fixing what the tools find? See our SEO & GEO service or contact MaslonLabs.

    Related Tools

    How useful was this post?

    Click on a star to rate it!

    Average rating 0 / 5. Vote count: 0

    No votes so far! Be the first to rate this post.

    We are sorry that this post was not useful for you!

    Let us improve this post!

    Tell us how we can improve this post?