If you’ve poked around in the WordPress backend, you might have noticed a setting labelled “Discourage search engines from indexing this site” and wondered what it meant. 🤨
Or perhaps you’re looking for a way to protect your site from unwanted visitors and are unsure whether this one small checkbox is sufficient to keep your content secure? What does this choice imply? What effect does it have on your website? And why should you avoid it — even if you’re attempting to conceal or hide your content?
What Does “Discourage Search Engines from Indexing This Site” Mean?
Have you ever wondered how search engines index your website and determine its search engine optimization (SEO) ranking? They accomplish this through the use of an automated programme known as a spider, also known as a robot or crawler. Spiders “crawl” the web, visiting and logging all of your content 🕷🕸 They are used by Google to determine how to rank and position your website in search results, to extract excerpts from your articles for the search results page, and to pull your images into Google Images.
✅ When you select “Discourage search engines indexing this site,” WordPress modifies the robots.txt file (a file that gives instructions to spiders on how to crawl your site). Additionally, it can add a meta tag to your site’s header instructing Google and other search engines not to index (not to show) any of your site’s content.
🕷 While web crawlers will still be able to find your site, properly configured crawlers will read your robots.txt file and exit without indexing or displaying the content. 🕷
More for You: How to Bulk Rename files in Windows 10 and 11
Why A Person Might Not Want to Index His Site?
Websites are designed to be viewed by other people. If you want users to read your articles, purchase your products, and consume your content, why would you deliberately attempt to block search engines?
There are a few reasons why you might want to conceal a portion or the entirety of your website:
- Your website is under construction and is not yet ready for public viewing.
- You’re using WordPress to manage your content but want to keep it private.
- You’re attempting to hide sensitive data.
- You want your site to be accessible only to a select few people via a link or via invitations, not via public search pages.
- You want to protect some content, such as newsletter-exclusive articles, behind a paywall or other gate.
- You want to redirect traffic away from old, out-of-date articles.
- You want to avoid being penalised for duplicate content or test pages.
While there are better solutions for some of these — such as using an offline development server, making your articles private, or encrypting them — there are legitimate reasons to deindex a portion or all of your site. 😎
How to Discourage Search Engine Indexing in WordPress
If you’d rather skip the extra steps and return to the default setting, here’s how to enable or disable the “Discourage search engines” option in WordPress.
Navigate to Settings > Reading in your WordPress dashboard. Locate the option for Search Engine Visibility, which includes a checkbox labelled “Discourage search engines from indexing this site.”
If this is already checked and you want your site to be indexed, uncheck it. If you’re going to prevent your site from being indexed, double-check it (and make a mental note to turn it off later!).
Now click Save Changes to finish. Your site may take some time to be reindexed or removed from search results. If your site remains deindexed, you can also remove the noindex code from your header file or manually remove the “Disallow” flag from robots.txt.
That is straightforward, but what are some reasons to avoid this option, or at the very least not entirely rely on it?
Disadvantages of Using the Discourage Search Engines Option
It appears straightforward — simply check a box and no one will be able to view your site. Isn’t that sufficient? Why would you want to avoid using this option alone?
When this setting or a similar one is enabled, it simply adds a tag to your header or robots.txt. As demonstrated by the fact that older versions of WordPress still allow your site to be listed in search results, a minor glitch or other error can result in people seeing your ostensibly hidden pages.
Additionally, it is entirely up to search engines to abide by your request for them to refrain from crawling your site. Although major search engines such as Google and Bing typically do, not all search engines use the same robots.txt syntax, and not all spiders crawling the web are sent out by search engines.
For instance, the Wayback Machine is one service that makes use of web crawlers. And once your content is indexed by such a service, it is permanently archived on the web.
You may believe that simply because your brand new website has no links to it protects it from spiders, but this is not the case. Existing on a shared server, sending an email containing a link to your website, or even visiting your website in a browser (particularly Chrome) may expose your site to crawling.
If you want to hide content, adding a parameter and hoping it works is simply not a good idea. 😎
To be clear, if the content you’re deindexing is sensitive or personal in nature, you absolutely should not hide it using robots.txt or a meta tag. This option will completely deindex your site from search engines, whereas you may only want to deindex specific pages.
There are numerous reasons to hide content on your website, but relying on the “Discourage search engines from indexing this site” option is not the best way to ensure that such content is not seen.
Unless you want to completely hide your website from the internet, you should never click this option, as it can have a bad effect on your SEO if accidentally toggled. And even if you do wish to conceal your website, the default method is insecure. It should be used in conjunction with password protection or other forms of blocking, particularly if you’re dealing with sensitive data.
Do you use any other techniques to conceal your site or portions of it? 😮 Kindly inform us in the comments section.
More for You: Update!! Windows 11 is arriving soon