The term "site indexing" is familiar to every webmaster and almost every online business owner today. It affects the position your resource occupies in search engines, and, of course, when it is higher, users who are looking for the necessary information, goods, and services will be more likely to be among your customers, because they will see you first.
Accordingly, indexing inaccuracies will negatively affect the ranking in the TOP results, or will completely lead to the fact that the company's offer will be outside the user's intended queries. Idea Digital Agency Company has compiled a short guide for online business owners. We will tell you how to check the indexing of your site, and what to do if it has not been seen by bots.
What is site indexing?
This is a procedure that is carried out by algorithms using the principles of artificial intelligence, in order to determine the relevance of data, its importance for users, and relevance. It is carried out automatically, and the robot performs a certain sequence of actions during the procedure:
- Finds you through SEO promotion or using external links;
- Clicks on the link and sends a request to the server to obtain content;
- Studies and evaluates the importance of the content, then makes a decision on indexing;
- Sends a report.
In the event of a positive decision, the rating increases. If the data is recognized as unimportant, you will not have to wait for a second procedure until the shortcomings that prevented the robot from doing its job are corrected.
Why do you need to index sites in Google, you ask. The answer is simple: only the scanned content that received an index from the bot is displayed in the user-defined output in the positions you need. If for some reason the bot analytics passed by or the URLs were partially analyzed, no matter how great the content was or how useful the products were, customers will not know this and will receive links to competitors' stores in the output of their queries.
It is wrong to confuse the concepts of scanning and indexing. The bot performs both procedures. The robot can scan quite quickly, but indexing is a longer process.
We will consider in detail how to speed up the indexing of a site later, but the main thing in the principle of spider bots' work is scanning the HTML code. Therefore, it is important not only to prepare relevant content for the page, but also to correctly write the main tags, including headings of different levels, Title, and Description meta tags. It is better to entrust this task to professional optimizers if 100% results are important.
How to check site indexing
If you are not sure whether a URL has been crawled, you notice low traffic or are looking for reasons for low conversion, it is worth running a URL check in Google to make sure that it is properly processed and available. There are several ways to do this, and each method has its own advantages. Let's consider each in more detail.
Check in the webmaster panel
Indexing a site in Google is possible using a standard tool — Search Console. Select the “Overview” section in the console, and open the “Coverage” graphical report. On the graph, you can see the number of objects indexed without errors (green graph) and those with which an error occurred (red graph). You can see a detailed report on the main files by going to the adjacent tab with the report.
If Google's site check shows poor results, you should make sure that it is not closed to their attention and that the basic problems have been fixed.
Material on the topic: Google Search Console: how to add a site and SEO life hacks
Checking through operators in search queries
The indexing of sites in Google is also displayed in the search engine itself. To study the amount of indexed data in general, you can use the operator site :. The algorithm in this case is simple: you need to add an exact link or domain name to Google, putting the site : in front of the address. After that, copy the full link without spaces. Look at the output.
To get a more detailed analysis in Google, you can go to the "Search Tools" section: this way you can see how many links were processed with index assignment over a certain period of time.
In parallel, let's look at how to check the indexing of the site by operators for each section.
- The standard scheme is the same site :. Insert the full link after it and view the result. The absence of a result is a sure sign that the page is not indexed.
- In this case, you can use a special operator - inurl :. When processing a query with this operator, a positive result means that the section is still indexed, and a negative result means that it is not in the PS analytics.
For the same purpose, you can perform such a simple action as viewing the Google cache. To do this, while on the selected page, in the search bar, write the cache operator before the URL and press Enter.
Checking via plugins and bookmarklets
You can find out information without directly interacting with search engines and their consoles. It is enough to use plugins or bookmarklets - small applications that are installed directly in the browser. You can use plugins such as RDS bar or SEO Magic. In addition to them, there are other micro-programs with which mass checking of page indexation will be a matter of minutes.
Analysis through services for checking indexation
Some software developers offer ready-made solutions for finding out the status of the PS. Some of them are offered on a paid basis and allow you to extract a lot of data for analysis at once. There are also simpler, shareware services. They may have a limit on the number of links, or they may have limited functionality, but they can provide you with basic data.
Paid versions of services are offered by companies such as Serpstat, Netpeak, and Se Ranking (the latter offer a free service, but only if you are already subscribed to one of the paid plans).
How to check the indexation of a specific page
As with analytics in general, you can analyze each page separately through Google Search Console. It's very easy to do: copy the link and paste it into the Console search bar. If the answer is positive, a corresponding information window will appear.
If the answer is negative, you can resend it using the "Request indexing" button.
You can also test certain URLs through a search engine with the operator site:, url / inurl. And, of course, such an option is present in paid services or plugins for analysis.
Why the site is not indexed, main errors
Forewarned is therefore armed, we believe. Therefore, it is important to know about the main reasons why search bots ignore you and do not index your entire site or individual sections.
- Often the problem is trivial: an online project has just been created, and search spiders have not yet reached it. According to the statistics of creating new objects on the Internet, such a situation is not uncommon. Therefore, sometimes you should not panic, just be patient and regularly check to see if your pages have appeared in the index.
- The absence of a sitemap is not critical, but not the best option. If you have not provided the robots with a ready-made guideline, they can selectively index a few random pages, and skip or ignore the rest.
- Errors in pages: Google Search Console reports should not be ignored. They show current page errors, which, in turn, can lead to processing failures by search robots.
- The lack of content is a reason for rejection. Content is not only text but also all kinds of media objects located on the page.
- Problems with robots.txt. If the meta tag was accidentally assigned the index attribute, search spiders will not be able to find it. Similarly, the X-Robots tag could have been set as Noindex during testing or set by accident. In general, the robot's file should be carefully examined for the presence of unnecessary characters or attributes.
- Low uniqueness of content also provokes rejection. Even if you are the original source of content, but your competitors “licked” it and indexed their resource quickly, they will get priority, not you.
- The presence of duplicate pages for Search robots is also an alarming signal.
You can also look at statistics to track the reasons for ignoring bots: perhaps the reason is low loading speed.
Instructions for accelerating site indexing
Indexing by search engines is an urgent issue. And it is worth speeding it up in all possible ways. And there are not so few of them. Here are the most effective ones:
- Be sure to create a sitemap — it’s a guide for search engine robots.
- If you already have a map, don’t forget to update it regularly so that new pages and updated content are also indexed.
- Monitor the uniqueness of the content. Remove the one that shows low uniqueness.
- Monitor whether there are any duplicate pages left after development.
- Failures when following links and erroneous redirects only confuse the “spiders”.
- Constant monitoring of robots.txt is practically preventive maintenance for the health of your online business.
- Adjusting pages for speed will help speed up the process. So do this too.
- A smart distribution of links on donor resources and within the resource itself will allow algorithms to reach you quickly and accurately. To do this, announce the updated sections on social networks, choose appropriate donors, and logically relink on the resource.
- Regular updates are a signal to robots that you are alive and developing. Remember this.
Material on the topic: How to check and increase the loading speed of a site?
How to block a site from indexing?
There are situations when it is necessary to perform the opposite action - to prevent you from being indexed. You can remove a page from Google search and block it from bots by adding the appropriate commands to robots.txt.
For this, the Disallow command is used.
With its help, you can close both all and individual directories, images, or media content.
Similarly, you can set a ban through the webmaster's office. In different CMSs, this can be implemented in different menu positions, but in general, it is quite feasible.
And, of course, you can request password access at the server settings level.
Conclusion
The mysterious procedure we described is not as scary as it seems at first glance. It is useful for all resources, except for those that have not yet been launched or are already outdated. Therefore, tracking the absence of files is very important. We hope that our article will be useful to you and that your online project will be accurately indexed and displayed at the very top positions in search engine results.
