WordPress creates a virtual robots.txt file for you at yourdomain.com/robots.txt if no physical file exists. You can change it by uploading your own file or by using the editor in an SEO plugin. A safe starting point blocks the admin area, allows the AJAX endpoint and lists your sitemap, without blocking scripts, styles or images.
Where the file is
Open yourdomain.com/robots.txt in a browser. If you see output but find no file in your hosting folder, WordPress is generating it on the fly. Uploading a real robots.txt file to the site root overrides that virtual version.
A sensible default
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Sitemap: https://www.example.com/sitemap_index.xmlUse your own sitemap URL. Build and adjust the file with my free robots.txt generator.
Not sure if this is hurting your site?
Get the fixes that matter most, in priority order.
How to edit it
- Use the file editor in your SEO plugin, if it offers one.
- Or create a plain text file named robots.txt and upload it to the site root through your host.
- Reload the address in a browser to confirm the change.
- Check specific URLs in the robots.txt tester.
What not to block
Do not block theme or plugin folders that deliver CSS and JavaScript, and avoid blocking the uploads folder. Google needs these files to render pages the way a visitor sees them.
The search visibility setting
WordPress has a setting under Reading that asks search engines not to index the site. It is useful during development, but if it stays on after launch your pages can disappear from search. Always check it when a site goes live.
Robots.txt is not a privacy tool
The file is public, and listing a folder in it can point curious people to it. To keep content out of search, use noindex or password protection, as explained in noindex vs canonical vs robots.txt. For the fundamentals, read what robots.txt is and why it matters for SEO.
Frequently asked questions
Should I block my search results pages?
Internal search results often create near-endless URLs, so many sites block them. Check your own crawl data before deciding.
Can a plugin change my robots.txt without me noticing?
Yes. Recheck the live file after installing or updating SEO, security or caching plugins.
Next steps
If your WordPress site has indexing problems, contact me here for a technical SEO review.
More in this series (robots.txt questions): start with Free Robots.txt Generator, then explore:
- How to Add a Sitemap to Robots.txt
- Robots.txt Wildcards: How * and $ Work With Examples
- Indexed, Though Blocked by Robots.txt: Why It Happens and How to Fix It
- Does Google Respect Crawl-Delay in Robots.txt?
Request received
Thank you! Saad will contact you within the next 24 hours.
Keep an eye on your phone and inbox.