SearchLaneRobots.txt generator
Pick what Google and AI crawlers may read, add the paths to keep private and copy a ready robots.txt — with separate rules for AI search and AI training.
| User-agent: * | Disallow: /admin |
| OAI-SearchBot, PerplexityBot | Allowed |
| GPTBot, ClaudeBot | Blocked |
| Sitemap | sitemap.xml |
Example result, not real data
What it shows
- AI searchOAI-SearchBot, ChatGPT-User, PerplexityBot and Claude crawlers. Allow them to be cited in AI answers.
- AI trainingGPTBot, ClaudeBot, Google-Extended, Applebot-Extended and CCBot. Blocking them does not affect search.
- Private pathsAdmin, cart or search pages you don't want crawled, repeated for every crawler group.
How to use the Robots.txt Generator
- Choose AI accessKeep AI search allowed if you want to be cited by ChatGPT, Perplexity and Claude. Decide separately about AI training.
- Add private pathsOne per line: /admin, /cart, /search. Leave empty to let crawlers read everything.
- Copy and uploadSave the file as robots.txt at the root of your site, then test it with the robots.txt checker.
Why it matters
robots.txt is the first file every crawler reads. A missing or wrong rule can keep Google away from a whole section, or block AI search crawlers you actually want.
Since 2023 the file has a new job: separating AI crawlers that cite you in answers from those that only collect training data. A generated file with both groups spelled out avoids the most common mistake — blocking everything AI by accident.
Guide
AI crawlers and robots.txt: GPTBot, OAI-SearchBot, ClaudeBot and more
Questions
Why are the paths repeated for each group?
A crawler follows only the most specific group that names it. If GPTBot has its own group, the rules under User-agent: * no longer apply to it.
Does blocking GPTBot remove me from ChatGPT?
No. ChatGPT search uses OAI-SearchBot and ChatGPT-User. GPTBot only collects training data.
Where do I put the file?
At the root of the domain as plain text: https://example.com/robots.txt. Each subdomain needs its own.
Does robots.txt keep a page out of Google?
It stops crawling, not indexing. A blocked page can still appear in results if other sites link to it. Use noindex to hide a page.
More free tools
| Googlebot | Allowed |
| GPTBot | Allowed |
| PerplexityBot | Blocked |
| Google-Extended | Blocked |
| # Brew Lab | title |
| > Coffee grinders and guides | summary |
| ## Main pages | 4 links |
| ## Blog | 8 links |
| High Broken internal links | Search |
| Medium No author on articles | AI answers |
| Medium Duplicate titles | Search |