By Hridoy Ahmed | Last updated: October 2026 | 5 min read
Short answer: To create a robots.txt file for WordPress, pick the WordPress preset, add your sitemap URL, block only the areas crawlers do not need, and upload the file to your site root. Robots.txt controls crawling, not indexing. To keep a page out of Google, use a noindex tag.
Why use a robots.txt generator?
Short answer: One wrong line in robots.txt can block your whole site from search. A generator builds the file from presets and options, so you avoid syntax mistakes.
Real use case: You launch a new site. You pick the WordPress preset, add your sitemap URL and download the file. It blocks the admin area and leaves your content open.
Benefits
- Safe presets for WordPress, allow all and block all
- Sitemap line added for you
- Optional rules for AI crawlers
- Copy or download in one click
Why use this tool
- Free, no signup.
- Private: the file is built in your browser
- Beginner-friendly, with no syntax to remember
If your SEO plugin already manages a good robots.txt, you may not need it.
Create robots.txt in 4 steps
- Open the tool. Use my free Robots.txt Generator.
- Pick a preset: WordPress, allow all or block all.
- Add your sitemap URL and any paths to disallow. Optionally add rules for AI crawlers.
- Copy or download the file. Upload it to your site root, or paste it into your SEO plugin’s robots editor.
Where it goes
The file must load at the root of your domain, for example https://example.com/robots.txt. A file in a folder does not work.
What to block and what not to
- Usually block: admin areas and internal search pages you do not want crawled
- Do not block: your content, CSS or JavaScript files
- Add: a line that points to your XML sitemap
Common mistakes
- Using “block all” on a live site. It stops crawling of everything. Use it only while the site is in development.
- Trying to remove a page from Google with robots.txt. Use noindex and keep the page crawlable.
- Blocking CSS or JavaScript, so Google cannot see the page properly
If a page is not showing in Google, see why a page is not indexed.
FAQ
Does robots.txt remove a page from Google?
No. It controls crawling, not indexing. Use a noindex tag instead.
Where does robots.txt go?
In the root of your domain, such as https://example.com/robots.txt.
Should I block AI crawlers?
It is a business choice. See my AI search readiness checklist.
Try the tool: Generate your robots.txt. Want it checked as part of a full review? See my technical SEO service.
About the author: Hridoy Ahmed is an SEO specialist and WordPress security expert based in Dhaka, Bangladesh. He has 8+ years of experience and works with clients worldwide.