A robots.txt file is a plain-text file that gives web crawlers instructions about which URLs they may or may not request from a website. Search engines such as Google use it to understand a site’s crawling preferences before requesting pages. For website owners, it is an important part of technical SEO because a small rule can affect how search engines access large parts of a website.
However, robots.txt is often misunderstood. It controls crawling, not indexing. It does not protect private information, guarantee that a URL will disappear from search results, or force every bot to follow its instructions. This guide explains how robots.txt works, how to write common rules, how to use it safely on WordPress websites, and how to test changes before publishing them.
What Is a Robots.txt File?
Robots.txt is a text file that follows the Robots Exclusion Protocol. It is normally placed in the root directory of a website and is available at a URL such as https://example.com/robots.txt.
When a compliant crawler visits a website, it can request this file and use its rules to decide which paths it should crawl. The file is public, so anyone can open it in a browser. It is not a password, firewall, or access-control system.
Why Does Robots.txt Matter for SEO?
- Crawl management: It can discourage crawlers from repeatedly requesting low-value URL patterns.
- Server load: On large websites, carefully chosen rules can help reduce unnecessary crawler requests.
- Content discovery: A sitemap line can point crawlers to the XML sitemap location.
- Technical control: It helps website owners communicate crawling preferences for specific sections or bots.
Robots.txt does not directly improve rankings. Its value is in helping crawlers access a site in a way that matches the site’s technical and content needs.
Where Is Robots.txt Located?
For most websites, the file is located at the root of the host:
https://example.com/robots.txthttps://www.example.com/robots.txt
The protocol, host, and port matter. A robots.txt file for the main domain does not automatically control a separate subdomain. For example, blog.example.com can have its own robots.txt file.
Open your domain followed by /robots.txt in a browser to see the publicly served file. If the file does not exist, the server response and crawler rules still matter; do not assume that a missing file means your pages are blocked.
How Does Robots.txt Work?
Robots.txt is made up of groups of rules. Each group begins with a User-agent line that identifies the crawler. The rules below it apply to that crawler until another user-agent group begins.
For example:
User-agent: *
Disallow: /private-area/
Sitemap: https://example.com/sitemap.xmlThis example asks all crawlers that follow the protocol not to request URLs under /private-area/ and provides the location of the XML sitemap. It does not make the folder private or guarantee that URLs in it will be absent from search results.
Important Robots.txt Directives
1. User-agent
User-agent identifies the crawler a group of rules targets. The wildcard * is commonly used for all crawlers that support the protocol.
User-agent: *
Disallow: /staging/You can also create a group for a named crawler, such as Googlebot:
User-agent: Googlebot
Disallow: /temporary/2. Disallow
Disallow tells the matching crawler not to request a specified path.
User-agent: *
Disallow: /test/
Disallow: /internal-search/A rule such as Disallow: / blocks crawling across the entire host for the matching crawler. Use it only when that is genuinely your intention, such as on a private staging site that is separately protected. Accidentally publishing it on a live website can prevent search engines from crawling important pages.
3. Allow
Allow can create an exception within a disallowed path. Search engines that support these rules generally use the most specific matching rule when deciding whether a path is allowed.
User-agent: *
Disallow: /resources/
Allow: /resources/public-guide.pdfCheck the behaviour of the target crawler before relying on complex combinations of rules.
4. Sitemap
The Sitemap field provides the absolute URL of an XML sitemap or sitemap index. It is not restricted to a particular user-agent group.
Sitemap: https://example.com/sitemap_index.xmlUse the sitemap URL that your website actually serves. A sitemap reference helps crawlers discover the sitemap, but it does not guarantee that every listed URL will be crawled or indexed.
5. Wildcards and URL endings
Some crawlers, including Google, support pattern characters such as * and the end-of-URL marker $. For example:
User-agent: *
Disallow: /*?sort=
Disallow: /*.pdf$These patterns can affect many URLs. Their meaning and support can vary between crawlers, so avoid copying wildcard rules without testing the exact URL patterns on your site.
Robots.txt Examples for Common Websites
Basic Allow-All File
If you do not need to restrict crawling, a simple file can identify the sitemap and leave paths accessible:
User-agent: *
Allow: /
Sitemap: https://example.com/sitemap.xmlFor most crawlers, omitting the allow-all rule has the same default effect. Always confirm the sitemap path before using the example.
Restricting a Test Folder
User-agent: *
Disallow: /staging/
Sitemap: https://example.com/sitemap.xmlThis is a crawling instruction, not a replacement for password protection. A staging website should be protected from public access at the server or application level.
WordPress Robots.txt Example
A WordPress website may use a setup similar to this, but it should be adapted to the site’s plugins, URL structure, and technical requirements:
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Sitemap: https://example.com/wp-sitemap.xmlWordPress installations may use a core sitemap, a sitemap index generated by an SEO plugin, or another sitemap location. Use the sitemap URL that is active on your site. Do not automatically block /wp-content/, CSS, JavaScript, images, or other resources that search engines need to render and understand pages.
The admin area is not made secure by robots.txt. Authentication and proper server permissions provide security.
Robots.txt vs Noindex: What Is the Difference?
Robots.txt and noindex solve different problems. Confusing them is one of the most common technical SEO mistakes.
| Robots.txt | Noindex |
|---|---|
| Controls whether a compliant crawler may request a URL. | Asks supported search engines not to include a crawled page or resource in search results. |
| Usually applies to URL paths for a crawler on a host. | Can be applied to an individual HTML page or resource using a meta tag or HTTP header. |
| Does not reliably remove a URL from search results. | Can remove a page from search results after the crawler accesses and processes the directive. |
For example, a page that should be removed from Google Search needs to remain crawlable so Google can read its noindex instruction. If robots.txt blocks the page, Google may not see the noindex tag. A blocked URL can still be shown in search results if Google discovers it through links or other signals, sometimes without a descriptive snippet.
To prevent access to confidential information, use authentication or server-level access controls. Neither robots.txt nor noindex is a security measure.
How to Create or Edit Robots.txt in WordPress
There are several ways to manage robots.txt on a WordPress website. The correct method depends on your hosting setup and plugins.
- Check the current file: Visit
https://yourdomain.com/robots.txtand save a copy of the existing rules. - Check your SEO plugin: Some WordPress SEO plugins provide a file editor or settings for robots.txt.
- Check hosting or server access: Depending on the setup, the file may be managed through the hosting file manager, SFTP, or server configuration.
- Make one small change: Add only the rule needed for a clearly identified crawling issue.
- Test the result: Confirm that important pages and resources remain crawlable.
- Keep a backup: Save the previous version so you can restore it if a rule causes problems.
WordPress can generate a virtual robots.txt response when a physical file is not present. A physical file at the site root may take precedence. After making changes, check the public URL rather than relying only on what an editor displays.
How to Test and Audit Your Robots.txt File
Do not publish robots.txt changes without checking their effect. A short review can prevent a site-wide crawling problem.
- Open the live file: Visit yourdomain.com/robots.txt and confirm that the expected version is served.
- Review important URLs: Check the homepage, service pages, product pages, blog posts, category pages, and important resources.
- Check Google Search Console: Use URL Inspection to understand whether Google can access a specific URL and to review the page’s indexing status.
- Test affected patterns: Use an appropriate robots.txt tester or crawler to check representative URLs against the rules. Google Search Console’s former standalone robots.txt Tester is no longer generally available, so do not rely on old tutorials that tell you to find it.
- Check rendered resources: Make sure important CSS and JavaScript files are not blocked in a way that prevents Google from understanding the page.
- Recheck after deployment: Verify the live file again after plugin, theme, migration, or hosting changes.
URL Inspection is useful for checking how Google sees a particular URL, but it is not a substitute for reviewing every rule and URL pattern in the robots.txt file.
Common Robots.txt Mistakes to Avoid
- Blocking the whole website: A broad
Disallow: /rule can stop search engines from crawling the entire host. - Using robots.txt to remove pages from Google: Use a crawlable
noindexdirective when the goal is to keep a page out of search results. - Trying to hide sensitive information: The file is public and does not prevent direct access.
- Blocking CSS and JavaScript: Important resources may be needed to render and understand a page.
- Blocking important WordPress paths without checking: Do not blindly disallow uploads, themes, plugins, or admin-related resources.
- Using unsupported directives: Rules such as
Noindexin robots.txt are not supported by Google. - Copying rules from another website: URL structures, plugins, and server configurations differ.
- Forgetting subdomains: Each host may need its own robots.txt file.
- Failing to test after a migration: A staging restriction can accidentally remain on the live domain.
Does Robots.txt Affect Crawl Budget?
Robots.txt can help manage crawler requests by excluding URL patterns that do not need to be fetched. This can be useful for very large websites with extensive faceted navigation, generated search URLs, or duplicate URL patterns.
However, crawl budget is not a problem every website needs to solve. For small and medium websites, clear navigation, useful internal links, reliable hosting, canonical URLs, and high-quality content are often more important priorities. Blocking URLs does not automatically make other pages rank better or get indexed faster.
Robots.txt and Website Migrations
Robots.txt deserves special attention during a website launch or migration. Development websites are often restricted from crawling, and those restrictions can cause serious problems if they are copied to the production domain.
- Check the live domain’s robots.txt before and after launch.
- Remove staging-only restrictions from the production version when appropriate.
- Confirm that important landing pages, assets, and sitemap URLs are accessible.
- Review redirects, canonical tags, and noindex settings separately.
- Use Search Console URL Inspection to check key migrated URLs.
Robots.txt SEO Checklist
- File location: The file is available at the root of the correct host.
- HTTP response: The live URL returns the expected response and file content.
- Important pages: Key landing pages, articles, and products are not unintentionally blocked.
- Resources: Required CSS, JavaScript, and image resources remain accessible.
- WordPress: Rules are compatible with the site’s plugins and URL structure.
- Sitemap: The file points to the correct, working XML sitemap.
- Noindex: Indexing requirements are handled with meta robots or X-Robots-Tag, not a robots.txt directive.
- Security: Private areas are protected with authentication or server controls.
- Testing: Representative URLs and patterns have been checked after every change.
- Monitoring: The file is reviewed after redesigns, migrations, plugin changes, and hosting updates.
Frequently Asked Questions About Robots.txt
1. Is robots.txt important for SEO?
Yes. It helps communicate crawling preferences to compliant search engine bots. Incorrect rules can block important pages or resources, so it should be reviewed as part of technical SEO.
2. Does robots.txt stop a page from appearing in Google?
No. Robots.txt controls crawling, not guaranteed removal from search results. Google may still show a blocked URL if it discovers it elsewhere. Use a crawlable noindex directive or access protection when appropriate.
3. Is robots.txt a security file?
No. Anyone can view it, and crawlers are not required to obey it. Protect private pages with authentication and server-level access controls.
4. Should I block the WordPress admin area?
Many WordPress robots.txt examples disallow the /wp-admin/ path while allowing /wp-admin/admin-ajax.php. This is a crawling preference only; it does not secure the admin area. Check your site’s actual requirements before applying the rule.
5. Can I add noindex to robots.txt?
No. Google does not support a noindex directive in robots.txt. Use a robots meta tag for HTML pages or an X-Robots-Tag HTTP header for other resources when you need to control indexing.
6. Does every website need a robots.txt file?
Not necessarily. A website can be crawlable without a custom robots.txt file. It is useful when you need to provide specific crawling rules or point crawlers to a sitemap.
7. Can robots.txt block images and CSS files?
It can restrict crawling of matching resource URLs, but blocking resources needed to render a page can make it harder for search engines to understand the page. Review resource dependencies before adding restrictions.
8. How often should I check robots.txt?
Review it after website migrations, redesigns, hosting changes, major plugin updates, and changes to URL structures. Also include it in routine technical SEO audits.
Final Thoughts
Robots.txt is a small file, but its rules can affect how search engine crawlers access a website. The safest approach is to keep the file simple, restrict only clearly identified URL patterns, and test every important change on the live host.
Remember the key distinction: robots.txt manages crawling; noindex manages search indexing; authentication protects private information. When these tools are used for the right purpose, they help keep your website technically clear and easier for search engines to process.