Why Your Pages Are Not Listed By Google – Confirmed Indexing Issues
Google is facing serious indexing problems these days that are affecting countless companies around the world. Despite official announcements that the indexing issues have been fixed, the issues seem to persist.
With great power comes great responsibility. We can therefore only hope that these problems will be resolved as soon as possible for all webmasters and company owners who are already facing a difficult year for their company.
Google is without a doubt one of the largest and clearest monopolies in the world. Bing, its closest search competitor, only has 2 percent of the market – hardly a significant threat to Google's 90 percent. So what happens if Google's SERP doesn't work properly?
Google indexing issue – Confirmed October 1, 2020
It all started in late September when a number of complaints surfaced related to deleting pages from Google search index. Lots of people have started reporting issues with index coverage and crawling bugs in forums, Facebook groups, etc.
As usual, Twitter has been the most popular channel for these type of issues as it is the only place there is a chance a Google representative could actually act on the issue.
The same thing happened in this case with the mention that this time Google's representative John Mueller was not aware of the problem, as indicated in the screenshot below.
This, of course, leaves a lot of room for speculation on the matter, but we prefer to leave those thoughts with you and just present you with the facts for now.
There have been many other discussions on the same topic everywhere, Search Console Help and Black Hat World to name a few. Below is an idea of the sentiment Twitter had on the matter.
On October 1st, 24 hours later, Google officially announced that there were some indexing issues and they are working to resolve them.
We're currently working on fixing two separate indexing issues that were affecting some URLs. One of them is mobile indexing. The other is canonicalization of how we recognize and treat duplicate content. In either case, pages may not be indexed.
– Google SearchLiaison (@searchliaison) October 1, 2020
For the next three weeks, they continued to provide updates about the issues and how to fix them. The latest and greatest updates on this topic are as follows:
- 9th October: Mobile indexing was effectively fixed yesterday, with about 99% of the URLs being restored.
- the 20th of October: The canonical issue was effectively fixed last Wednesday (October 14th) with about 99% of the URLs being restored. We expect the remaining fringe cases to recover within a week or two.
In the meantime, Google communications revolved around these types of messages:
The website owners cannot take any action on these issues.
We apologize for the problems here and are working quickly to resolve them.
We will update this thread as soon as everyone is fixed.
It can take days for both issues to be completely resolved. However, we have already restored many URLs and are working quickly to process more.
All good in the hood, one might try to say. Google had a bug that they fixed three weeks later.
Just that things are not exactly like that. Aside from the fact that three weeks in which new content is not indexed or old content is de-indexed takes a long time in Internet days, the problem persists, as reported by several webmasters.
We have two examples of our own that confirm the problem still exists. I'm not sure you know, but the same hard working team that created cognitiveSEO also created the brand monitoring tool, BrandMentions.com. Both examples are from this website.
The first is really cool research on the best time to post on Instagram (I may be subjective here, but you can check it out for yourself).
It doesn't appear that the URL has a crawling problem, but it is not being indexed by Google. It is a unique piece of content with no specific information (there is no no-index directive).
The article was posted on the BrandMentions blog on October 13th. I don't think it's Tuesday the 13th. Bad luck. Most likely, Google failed to fix the issues for 99% of the alleged URLs. And yes, there is a chance the article is within 1% of the unset URLs. However, this is not the only example.
The second example is a brand tracking information page that is also part of brandmentions.com. This time the site seems to have been crawled (on the same unfortunate Tuesday 13th), but today, October 20th, there is nothing to be found in the SERP.
We hope this changes, of course, and by the time you read this article the page will be indexed and ranked pretty well. For now, however, the problem with Google indexing seems to persist.
What's the problem with google canonicalization?
In a tweet on October 2, Google stated, “(…) If it is a canonical problem, the URL Inspector may show the URL as a duplicate and the canonical document that Google selected is different. The problem with canonicals impacted approximately 0.02% of our index starting around September 20th and ending at 4:30 p.m. PT late yesterday. Since then we have recovered about 10% of these URLs and are still processing them. "
What does that mean exactly?
It's kind of a duplicate problem. Let's say you publish a new article and soon after a scraper copies your entire article and publishes it anyway.
The canonical tag added on your website tells Google that your article is the original article and that this article should come first. In principle, you can tell Google that this page should be displayed instead of this page. So if you have a ranking on Page B for a keyword, you can basically tell Google to display Page A instead.
However, once the issue of canonicalization occurs, Google will no longer display the original content but instead display the syndicated (the one that is a copy of the original content).
Canonical tags are Google's recommendation for different situations as you can see on the dedicated page or in the picture below.
There are billions of websites out there and it's Google's job to rate them. Ideally, all websites should have unique pages. Each page should contain original content. In the real world, however, duplicate pages are more common than you think.
Google will crawl your website and find 3 pages trying to rank for the exact same keyword. Not only does Google have to choose between billions of different websites, it also has to choose between duplicate pages on the same website. That seems difficult indeed. That's why there is the canonical tag to tell Google which original piece should be classified.
And when you have a problem of this size, remember how important it is for the search engine to do its job correctly, but also for the webmasters to set everything up correctly.
Yes, it seems that even if everything is okay, problems can arise.
At the time of reading this post, these issues should no longer occur as Google has stated that they have been fixed. Given the current situation and the fact that webmasters are still complaining about it, we hope for the best and expect the worst.
What's the problem with Google Mobile indexing?
Due to the problem of mobile indexing, it took a long time for new web pages to be indexed and shown in the Google index. Some of the newly published content is not yet indexed.
Even the big websites were affected by these problems. Below are some screenshots from Ewdison Then. He reported this issue on September 29th because he found it really strange that all of these websites posting news every few minutes hadn't indexed anything from the past hour.
Google stated, "If a previously indexed page disappears, it may be the mobile indexing issue where we don't select a page to index at all."
As you could see in the screenshots above, many webmasters have complained that they just disappeared from the index. As you can see in the screenshot below, Google stated that the problem only affected a few websites (around 0.2%).
This may sound like a small number, but keep in mind that the Google index is huge and 0.2% of that is still a large number. The Google Search Index contains hundreds of billions of web pages and is well over 100,000,000 gigabytes in size. Therefore, 0.2% of this enormous amount of data doesn't feel so small anymore.
The mobile indexing issue was first fixed almost two weeks before canonicalization, according to Google. Hopefully, this issue will no longer occur in the future as it affects people's search and businesses.
Google has had problems with indexing in the past as well. For example, a similar problem has arisen over the past year, where people faced indexing problems from all sectors and countries. Let's hope that this time the solution is final and that these types of problems do not arise again.
We are aware of indexing issues affecting some websites as of Friday. We believe the problems are largely resolved and do not require any special efforts on the part of the website owners. We will provide another update when the issues are considered fully resolved.
– Google SearchLiaison (@searchliaison) April 7, 2019
How is the Google indexing problem affecting your business?
The short answer to that would be: GREAT TIME.
Homepages, articles and product pages were suddenly de-indexed. Imagine your company's website disappearing from searches at all.
The errors in Google search lead to a massive drop in organic traffic. Your content marketing strategy is no longer helpful
It's been a difficult year already, and the indexing problem is making the business lose even more.
It's easy to forget that Google is ultimately just a private company like any other. One that has a worldwide monopoly on search engines, with a staggering 92.27% market share that we all depend on, and that has managed to get its namesake in the dictionary as a common word. But still a private company. While we would all like to understand what has happened and is happening and move on to our virtual lives, it may not be that simple.
However, with great strength comes great responsibility. At least it should be.
Who should the companies that lose their income for more than two weeks blame? Yes, errors happen all the time, and all apps, big or small, have them. And yes, we tend to misuse the Google engine as a “reality interface”. But Google also has its responsibilities.
For example, Google affects the economy through the way ads are currently placed: the more a company pays, the more often the add will be visible. Google responses resulting from queries are also rated during the search. And Google is responsible for a company's value (to a certain extent, of course). Google does that for ads, of course, but it does it for the entire SERP.
A sudden disappearance or, in general, a dramatic and sudden change event is a frequently used narrative tool in much of literature and film. It is sometimes left unexplained, but always manages to evoke a strong emotion. This is the mass disappearance of 2% of the world population in the television series "The Leftovers" or the sudden stopping of a death for an entire country in Jose Saramago's novel "Death with Interruptions".
In real life, such events are usually much smaller and less dramatic, but they are not without consequence.
We will likely never know the actual number of companies affected.
What we do know is that in a year that is already difficult enough for so many, a problem of this magnitude translates into very stressful situations for entrepreneurs and their employees, for the general welfare and profits of companies.
Is there anything you can do about this situation?
The short answer is No. We just have to wait and hope that things go back to normal. We're sure you've heard this far too often lately.
On September 30th, Google announced that its webmaster tool has temporarily suspended request indexing.
There was no detailed explanation as to why, just a vague mention of infrastructure changes.
As the days progressed, the company's staff insisted that everything was okay, while complaints about Google indexing issues persisted as more and more companies appeared to be affected.
Google is so ubiquitous these days that most of the time we don't even consider it a private company. Both the Oxford English Dictionary and the Merriam-Webster Dictionary added “to google” as a transitive verb in 2006. It's so common that it's just … there. With a market share of over 90%, this mindset is no surprise.
The problem with all of this is that for some reason we consider common nouns to be more trustworthy. Like Google, they were no longer a company but a public utility whose sole intent is to help its users rather than try to make up for it and make a profit as a market-leading private company.
And companies are sometimes more interested in profit than in other things like fairness or truth. That's not to say that Google has deliberately lied in the past or made insufficient effort to correct their mistakes. But it has told it in a profitable way at times: not telling the whole truth about the implications of links from press releases, not disclosing anything about the Hummingbird algorithm update except long after its impact was already on, or not mentioning the current Submit -URLs and indexing errors from the start.
If all of these are true, that's what companies do. They keep a little to themselves, undercut or undercut, as it suits their purpose. However, because of Google's quasi-monopoly, they can also improve the playing field, not just accuracy or truth. Of course, the search engine giant isn't required to play nice. As a private company, you don't have to tell anyone how to update your algorithms if you have errors, what impact your actions should have or what strategy you are pursuing. And yet there is an entire industry trying to find out exactly these things.
In the hope that this article will be indexed and that you will find it easily within the SERP, we look forward to your opinion.