What is a robots.txt file?
Before search engines crawl your website, they check the robots.txt file for instructions. Robots.txt is a very powerful file, so tread lightly when updating it. Double-check that you are indeed communicating your intended goals.
You can view your robots.txt file at www.mywebsite.com/robots.txt.
WordPress & Robots.txt
WordPress websites in development typically use the robots.txt file to block crawlers from indexing the website. The code looks like:
User-agent: *
Disallow: /
“User-agent:” followed by an asterisk addresses all crawlers. “Disallow:” followed by a forward slash instructs crawlers to not index any page on the website.
To update your robots.txt file, navigate to the Reading settings in your WordPress website. When you’re ready for search engines and people to visit your website, go to Settings > Reading in the left-hand menu, and uncheck the box for “Discourage search engines from indexing this site”.
Then, your robots.txt file will read:
User-Agent: *
Disallow:
We recommend a very clean robots.txt file since, in addition to its pages, search engines also like to crawl your website’s JavaScript, CSS, and Ajax.
Should I add my XML sitemap to the robots.txt file?
Rather than adding your XML sitemap to the robots.txt file, submit it directly to Google Search Console and Bing Webmaster Tools.
How do I update my robots.txt in WordPress?
An easy way to update your robots.txt in WordPress is in the Yoast SEO plugin.
- Log into your WordPress website.
- In the left-hand navigation, go to Yoast > Tools and click on File Editor.
- Update the robots.txt file.
What are robots meta directives?
Robots meta directives provide firm, detailed instructions on how search engines should crawl and index an individual page’s content.
<meta name=“robots” content=“[parameters]”>
The most common content values are:
- Nofollow – Don’t follow the links in the page’s content.
- Follow – Follow the links in the page’s content.
- Noindex – Don’t index the page.
- Index – Index the page, please.
If you want to exclude a page from search engine results, but you want crawlers to follow the links on that page, use:
<meta name=“robots” content=“noindex,follow”>
How do you add a robots meta directive in WordPress?
Yoast SEO makes it easy to update the robots meta directives at the content type and taxonomy levels.
- Log into your WordPress website and go to the Yoast SEO plugin in the left-hand navigation.
- Select Search Appearance.
- To change a content type’s meta directive, slide the bar from Yes to No in the “Show [content type] in search results?” section. Save your changes.
In Closing
Robots meta directives can be a powerful SEO tool. Learn how to use them properly and you will be able to better manage how Google and Bing interact with your site.
Want to learn more about WordPress SEO, check out our Field Guide to SEO.
