How to close a site from indexing in search engines

[gafimiv406]

In general, indexing is a useful and extremely important process for promotion. Only by getting into the search engine index, a site can count on an influx of organic traffic from search. But sometimes indexing individual pages - for example, service pages containing technical information or personal client profiles - is detrimental to the overall optimization of the resource and its ranking. There are also situations when indexing is required to completely close the entire web project: during development, restructuring, testing.
Every webmaster needs to be able to communicate with search engine robots, including to set restrictions for them. In this article, we will tell you about the main ways to close a site from indexing.

Close a site from indexing using Robots.txt
This text document is checked by search robots first. Robots.txt is located in the root directory of the site and contains instructions for crawlers: which pages are allowed to be indexed, which are prohibited, and for which robots these prohibitions are set. Usually, it is in robots.txt that webmasters close a web resource from indexing - completely or partially.

How to close a site from indexing in search engines.Image by storyset on Freepik.
If there is no file with such a name in the root of your project, it means that absolutely all web pages are open for indexing. As we have already said, a complete lack of restrictions is not good, so you need to create "robots" yourself and write the rules in it. To do this, open an empty text file on your computer (in .txt format) and name it "robots" (all letters in lowercase). Now let's get to setting it up.

To close the entire website from indexing in robots, enter the code norway whatsapp number database there:

User-agent: *

Disallow: /

The user-agent directive shows the name of the bot to which the next command is addressed, in this case it applies to all search engines. In disallow (prohibit) the relative address of the pages to be closed from indexing is specified. The slash sign means a ban for the entire site.

If you need to close indexing for only one search engine, in the first line instead of the asterisk write the name of its crawler: for example, "Yandex" for Yandex. Lists of search robot names can be found on the Internet.

Sometimes you need to close access to all bots except one. Let's say we want to open scanning only for Google. Then we'll write in robots.txt:

User-agent: *

Disallow: /

User-agent: Googlebot

Allow: /

The allow directive, as you might guess, permits an action. Thus, we will set a ban for everyone except Googlebot. After you write the commands, the robots.txt file must be uploaded to the server in the root folder. It should be available via a link like domain.ru/robots.txt.