A client came to us recently with an interesting problem that evolved around Google’s Smartphone Googlebot. It was an interesting problem due to the fact that it was occasional and didn’t effect the Desktop Googlebot.
Their issues came apparent when they noticed that certain pages were being dropped out of the search results (on mobile). On top of this, they were also receiving errors whilst submitting their pages to Google via the Search Console.
How Did We Find The Problem
After gaining access to the client’s website, I was able to replicate their issue by performing a number of live tests in Search Console. As shown below, the page in question couldn’t be crawled due to the “noindex” tag being detected on the page.
Our first thought was that the client had accidently added a “noindex” tag to their website. However, after analysing their website, this wasn’t the case and even if it was, it would’ve caused issues with the desktop version of the website too.
The next step was to find out what Google’s Smartphone Googlebot was seeing and this was achievable via their Mobile Usability Tester. After performing multiple tests (as this was an occasional error), we came across the exact problem!
Siteground was blocking the Smartphone Googlebot as shown in the image below:
As well as Google viewing the following page on mobile, within the code, the “noindex, nofollow” tag was there in black and white.
Identifying The Issue With Google
After finding that Siteground were indeed blocking the Smartphone Googlebot, we showed this to @JohnMu, who then praised us on finding the error sooner rather than later.
Shortly after speaking to John, he Tweeted about this particular topic:
If you run a hosting service, and have a “you might be a bot” interstitial, make sure it uses a non-200 HTTP result code (perhaps 503), and that it doesn’t have a noindex on it. Serving a noindex+200 will result in pages being dropped from search, if search engines see it. pic.twitter.com/LFGQcq2dzf
— 🐄 John 🐄 (@JohnMu) January 17, 2022
As you can see from the tone of this Tweet, he wasn’t particularly pleased with Siteground! He also provided a link to Google’s Documentation that details information about verifying Googlebot’s for hosting companies to refer to.
What Was The Fix?
To stop the “you might be a bot” page appearing to the Smartphone Googlebot, we had to request for Siteground’s Anti-Bot AI to be removed from their website via a support ticket. There was nothing the client could’ve done on the website to stop this from happening.
Conclusion
This isn’t the first time Siteground have cause mayhem within the SEO community. Only a few months back they had issues with their DNS that was causing issues with Googlebot.
We’ve received reports that a number of Google bot networks cannot crawl some websites using our DNS service.
After thoroughly audit of our DNS, we can confirm there is no blocking on our end that prevents such crawling, nor we see any logs suggesting an issue with our systems.— SiteGround (@SiteGround) November 10, 2021
To conclude this post, ensure that you are paying attention to the indexing of your website. This can be achieved by checking the coverage report as well as keeping an eye on certain statistics (i.e. Valid Core Web Vital pages dropping with no explanation).
Luckily this client didn’t see any severe impact from this but its quite a worry that an error from your hosting company can cause your pages to drop entirely out of Google!
Experiencing Similar?
If you are experiencing similar issues with the Robots.txt file, our Cardiff SEO team at Webspection will be on hand to diagnose any issues straightaway.
