WordPress Robots.txt: What Should You Include?

Modest file robots.txt Often quietly sits in the background of WordPress websites, but the default setting is a bit basic outside the frame and, of course, it does not contribute to any adapted directives that you might want to adopt.

No more introduction is needed – we dive right into what you can still include to improve it.

(A small note to add: This post is only useful for WordPress installations only at the root directory of domain or underdogs, eg homein.com or example.domain.com.)

Where exactly is the WordPress robots.txt file?

WordPress generates the virtual file of robots.txt according to the default settings. You can see it by visiting /robots.txt your installations, for example:

https://yoursite.com/robots.txt

This default file exists only in memory and not presented with a file on your server.

If you want to use a custom file robots.txt file, all you have to do is transfer one to the root folder of the installation.

You can do this either using a FTP application or accessory, such as Yoast SEO (SEO → Tools → File editor), this includes a editor of robots.txt to which you can access within the WordPress admin area.

Default WordPress robots.txt (and why not enough)

If you do not create a Robots.txt file, a default WordPress output looks like this:

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

Although this is certain, it is not optimal. Let’s move on.

Always include your XML SiteMap (s)

Make sure all XML SiteMaps are explicitly stated, as this helps search engines discover all relevant URL.

Sitemap: https://example.com/sitemap_index.xml
Sitemap: https://example.com/sitemap2.xml

Some things I can’t block

There are now proposals to disable some basic directors of WordPress such as/WP-Includes/,/WP-CONTENT/PLUGINS/, or even/WP-CONTENT/upload/upload/. Not!

Here’s why you should not block them:

1. Google is smart enough to neglect unimportant files. Blocking CSS -ai JavaScript can harm the adoption and cause the problem to index.
2. You can inadvertently block valuable images/videos/other media, especially those loaded from/WP-CONTENT/uploads/, which contain all the transmitted media that you definitely want to crawl.

Instead, let the creepers bring CSS, Javascript and the images they need to show properly.

Web site management

It is advisable to ensure that websites are not released in SEO and for general security purposes.

I always advise To disable the whole page.

You should still use Noindex Meta Tag, but to ensure that the second layer is covered, it is still advisable to do both.

If you move up to Settings> ReadingYou can indicate the “Break Search Sciences from Indexing this website”, which works the following in the Robots.txt file (or you can add it to yourself).

User-agent: *
Disallow: /

Google can still index the pages if it detects connections elsewhere (usually caused by calls for styling from production when migration is not perfect).

IMPORTANT: When you move to production, ensure that you re -check this setting to ensure that you return any disabling or noidexing.

Clean some irrelevant WordPress paths

No need to block everything, but many default paths do not add a SEO value, such as below:

Disallow: /trackback/
Disallow: /comments/feed/
Disallow: */feed/
Disallow: */embed/
Disallow: /cgi-bin/
Disallow: /wp-login.php
Disallow: /wp-json/

To disable certain inquiries parameters

Sometimes you will want to prevent search engines from crawling URLs with known small value parameters, such as monitoring parameters, answers to comments or print versions.

Here’s an example:

User-agent: *
Disallow: /*?replytocom=
Disallow: /*?print=

You can use the Google Search Console URL Parameters to monitor the Indexing Forms aimed at parameters and decide if additional disabled people are worth adding.

Designing taxonomy and SERP with low values

If your WordPress Website includes the archives of tags or internal pages of search results that do not offer added value, you can also block them:

User-agent: *
Disallow: /tag/
Disallow: /page/
Disallow: /?s=

As always, weigh this against your specific Content strategy.

If you use taxonomy pages with tags as part of the content you want to index and crawl, ignore it, but in general, they do not add any advantage.

Also, make sure your internal connection structure supports your decision and minimizes any internal connection with areas that do not intend to intend to indexing or creep.

Creep statistics monitor

Once your Robots.txt has been established, follow the Crawl statistics via Google Console for search:

• See crawling statistics in the settings to see if the bots will spend resources.
• Use the URL Inspection tool to confirm whether the blocked URL is indexed or not.
• Check the sitemaps and make sure they are just a reference page you actually want to crawl and index.

In addition, some server management tools, such as Plesk, Cpanel and Cloudflare, can provide an extremely detailed crawl statistics outside Google.

And finally, use Screaming Frog’s Configuration Override to simulating changes and re -inspect the cramping optimization features Yoast SEO, some of which resolve the above.

Final thoughts

While WordPress is a great CMSnot set up with the most ideal default robots.txt or set up with the optimization of crawling in mind.

Only a few lines of code and less than 30 minutes of your time can save you thousands of unnecessary crawl requirements to your web site that are not worth identifying at all, as well as to ensure a potential scales problem in the future.

More resources:

---

Sepaled picture: Sklyareek/Shutterstock