A client came to us recently with an interesting problem that evolved around Google’s Smartphone Googlebot. It was an interesting problem due to the fact that it was occasional and didn’t effect the Desktop Googlebot.
Their issues became apparent when they noticed that certain pages were being dropped out of the search results (on mobile). They also received errors when submitting their pages to Google via the Search Console.
How Did We Find The Problem
After gaining access to the client’s website, I was able to replicate their issue by performing a number of live tests in Search Console. As shown below, the page in question couldn’t be crawled because the” no index” tag was detected on it.
Our first thought was that the client had accidentally added a “no index” tag to their website. However, after analysing it, we found that this wasn’t the case, and even if it had, it would’ve caused issues with the desktop version of the website, too.
The next step was to find out what Google’s Smartphone Googlebot was seeing, and this was achievable via its Mobile Usability Tester. After performing multiple tests (as this was an occasional error), we came across the exact problem!
Siteground was blocking the Smartphone Googlebot, as shown in the image below:
As well as Google viewing the following page on mobile, within the code, the “noindex, nofollow” tag was there in black and white.
Identifying The Issue With Google
After finding that Siteground was indeed blocking the Smartphone Googlebot, we showed this to @JohnMu, who then praised us for finding the error sooner rather than later.
Shortly after speaking to John, he Tweeted about this particular topic:
As you can see from the tone of this Tweet, he wasn’t particularly pleased with Siteground! He also provided a link to Google’s Documentation that details information about verifying Googlebot’s for hosting companies to refer to.
What Was The Fix?
To prevent the “you might be a bot” page from appearing on the Smartphone Googlebot, we had to request that Siteground’s Anti-Bot AI be removed from their website via a support ticket. The client couldn’t have done anything on the website to prevent this.
Conclusion
This isn’t the first time Siteground has caused mayhem within the SEO community. Only a few months back, they had issues with their DNS that were causing issues with Googlebot.
To conclude this post, ensure that you are paying attention to your website’s indexing. This can be achieved by checking the coverage report and keeping an eye on certain statistics (e.g., Valid Core Web Vital pages dropping with no explanation).
Luckily, this client didn’t see any severe impact from this, but it’s quite a worry that an error from your hosting company can cause your pages to drop entirely out of Google!
Experiencing Similar?
If you are experiencing similar issues with the Robots.txt file, our Cardiff SEO team at Webspection will be on hand to diagnose any issues straight away.
