Clean-Param: utm_source&utm_medium&utm_campaign

Mimaktsm10 · Post by **Mimaktsm10** » Thu Jan 30, 2025 5:16 am

Source: shutterstock.com

How to close a site from indexing using meta tags
The robots meta tag serves as an alternative to the text file of the same name. It is written in the source code of the web resource (in the index.html file), placed in the <head> container. It is necessary to specify who is not allowed to index the site. If the ban is general, robots is entered; if entry is prohibited to only one crawler, its name must be specified (Google - Googlebot, Yandex - Yandex).

There are two possible options for recording a meta tag.

First:

<metaname=”robots” content=”noindex, nofollow”/>

Second:

<meta name=”robots” content=”none”/>

The following values can be used for the “content” attribute:

none — prohibit indexing (including noindex and nofollow);

noindex — prohibits indexing of content;

nofollow - prohibits indexing of links;

follow - permission to index links;

index—enable indexing of content;

all—allow indexing of content and links.

For different cases, you need to use combinations of values. For example, when disabling content indexing, you need to allow bots to index links: content=”noindex, follow”.

How to close a site from indexing using meta tags

Source: shutterstock.com

By closing a website from ecuador email list search engines via meta tags, the owner does not need to create robots.txt at the root.

It is important to remember that in the matter of indexing, much depends on the "politeness" of the spider. If it is "well-mannered", then the rules prescribed by the master will be relevant. But in general, the validity of robots directives (both the file and the meta tag) does not mean 100% compliance with them. Even for search engines, not every ban is ironclad, and there is no need to talk about various kinds of content thieves. They are initially set up to bypass all bans.

Read also!

"Robots.txt Check: Common Errors and How to Fix Them"
Read more
In addition, not all crawlers are interested in content. For some, only links are important, for others - micro-markup, others check mirror copies of sites, etc. At the same time, system spiders do not crawl around the site like viruses, but remotely request the necessary pages. Therefore, most often they do not create any problems for resource owners. But if errors were made in the design of the robot or some external non-standard situation arose, the crawler can significantly load the indexed portal.