WordPress Robots.txt & Robots Meta Directives Explained

Erik Wardell

Erik is an SEO coach that loves breaking down complex SEO topics into understandable instructions anyone can follow. He thrives on helping people do what it takes to see their businesses succeed in search. Stay in touch on Twitter — @wanderinwardell.

What is a robots.txt file?

Before search engines crawl your website, they check the robots.txt file for instructions. Robots.txt is a very powerful file, so tread lightly when updating it. Double-check that you are indeed communicating your intended goals.

You can view your robots.txt file at www.mywebsite.com/robots.txt.

WordPress & Robots.txt

WordPress websites in development typically use the robots.txt file to block crawlers from indexing the website. The code looks like:

User-agent: *
Disallow: /

“User-agent:” followed by an asterisk addresses all crawlers. “Disallow:” followed by a forward slash instructs crawlers to not index any page on the website.

To update your robots.txt file, navigate to the Reading settings in your WordPress website. When you’re ready for search engines and people to visit your website, go to Settings > Reading in the left-hand menu, and uncheck the box for “Discourage search engines from indexing this site”.

robots.txt

Then, your robots.txt file will read:

User-Agent: *
Disallow:

We recommend a very clean robots.txt file since, in addition to its pages, search engines also like to crawl your website’s JavaScript, CSS, and Ajax.

Should I add my XML sitemap to the robots.txt file?

Rather than adding your XML sitemap to the robots.txt file, submit it directly to Google Search Console and Bing Webmaster Tools.

How do I update my robots.txt in WordPress?

An easy way to update your robots.txt in WordPress is in the Yoast SEO plugin.

  • Log into your WordPress website.
  • In the left-hand navigation, go to Yoast > Tools and click on File Editor.
  • Update the robots.txt file.

What are robots meta directives?

Robots meta directives provide firm, detailed instructions on how search engines should crawl and index an individual page’s content.

<meta name=“robots” content=“[parameters]”>

The most common content values are:

  • Nofollow – Don’t follow the links in the page’s content.
  • Follow – Follow the links in the page’s content.
  • Noindex – Don’t index the page.
  • Index – Index the page, please.

If you want to exclude a page from search engine results, but you want crawlers to follow the links on that page, use:

<meta name=“robots” content=“noindex,follow”>

How do you add a robots meta directive in WordPress?

Yoast SEO makes it easy to update the robots meta directives at the content type and taxonomy levels.

  • Log into your WordPress website and go to the Yoast SEO plugin in the left-hand navigation.
  • Select Search Appearance.
  • To change a content type’s meta directive, slide the bar from Yes to No in the “Show [content type] in search results?” section. Save your changes.
Yoast No Index

In Closing

Robots meta directives can be a powerful SEO tool. Learn how to use them properly and you will be able to better manage how Google and Bing interact with your site.

Want to learn more about WordPress SEO, check out our Field Guide to SEO.

Erik Wardell

Erik is an SEO coach that loves breaking down complex SEO topics into understandable instructions anyone can follow. He thrives on helping people do what it takes to see their businesses succeed in search. Stay in touch on Twitter — @wanderinwardell.

Field Guide to SEO

An approachable and actionable guide to SEO for beginners to experts

Wordpress SEO

You Will Learn

WordPress is our favorite CMS for SEO. Learn why and how to leverage WordPress to grow your traffic.

Scroll to Top