Generate a correct robots.txt: allow or block all crawlers, exclude specific paths, optionally block AI training bots like GPTBot, and declare your sitemap.
robots.txt generator
Build a valid robots.txt: allow or block crawlers, exclude specific paths, optionally block AI training bots, and point to your sitemap. Copy or download the result.
How to use robots.txt generator
- Pick a base policy: allow everything (live sites) or block everything (staging).
- List any paths to exclude, one per line.
- Optionally add your sitemap URL and tick the AI-bot block.
- Copy the result or download robots.txt and put it at your site root.
Example: A live shop might allow everything except /cart/ and /admin/, plus a Sitemap line.
Frequently asked questions
- Where does robots.txt go?
- At the root of your domain: https://www.example.com/robots.txt. Crawlers only look there—a robots.txt inside a subfolder does nothing.
- Does robots.txt block pages from Google’s index?
- Not reliably. It stops crawling, but a blocked URL can still be indexed if other sites link to it. To keep a page out of the index, use a noindex meta tag or remove the page.
- Should I block AI training bots?
- That is a policy choice. Blocking GPTBot, CCBot, and similar stops your content being used for model training, but does not affect your Google or Bing search rankings.