Is Your Hosting Provider Blocking Googlebots?


A client came to us recently with an interesting problem that evolved around Google’s Smartphone Googlebot. It was an interesting problem due to the fact that it was occasional and didn’t effect the Desktop Googlebot.

Their issues came apparent when they noticed that certain pages were being dropped out of the search results (on mobile). On top of this, they were also receiving errors whilst submitting their pages to Google via the Search Console.

How Did We Find The Problem

After gaining access to the client’s website, I was able to replicate their issue by performing a number of live tests in Search Console. As shown below, the page in question couldn’t be crawled due to the “noindex” tag being detected on the page.

Google Smartphone Bot

Our first thought was that the client had accidently added a “noindex” tag to their website. However, after analysing their website, this wasn’t the case and even if it was, it would’ve caused issues with the desktop version of the website too.

The next step was to find out what Google’s Smartphone Googlebot was seeing and this was achievable via their Mobile Usability Tester. After performing multiple tests (as this was an occasional error), we came across the exact problem!

Siteground was blocking the Smartphone Googlebot as shown in the image below:

Googlebot Blocked

As well as Google viewing the following page on mobile, within the code, the “noindex, nofollow” tag was there in black and white.

Noindex Tag Googlebot

Identifying The Issue With Google

After finding that Siteground were indeed blocking the Smartphone Googlebot, we showed this to @JohnMu, who then praised us on finding the error sooner rather than later.

Shortly after speaking to John, he Tweeted about this particular topic:

As you can see from the tone of this Tweet, he wasn’t particularly pleased with Siteground! He also provided a link to Google’s Documentation that details information about verifying Googlebot’s for hosting companies to refer to.

What Was The Fix?

To stop the “you might be a bot” page appearing to the Smartphone Googlebot, we had to request for Siteground’s Anti-Bot AI to be removed from their website via a support ticket. There was nothing the client could’ve done on the website to stop this from happening.


This isn’t the first time Siteground have cause mayhem within the SEO community. Only a few months back they had issues with their DNS that was causing issues with Googlebot.

To conclude this post, ensure that you are paying attention to the indexing of your website. This can be achieved by checking the coverage report as well as keeping an eye on certain statistics (i.e. Valid Core Web Vital pages dropping with no explanation).

Luckily this client didn’t see any severe impact from this but its quite a worry that an error from your hosting company can cause your pages to drop entirely out of Google!

Experiencing Similar?

If you are experiencing similar issues with the Robots.txt file, our Cardiff SEO team at Webspection will be on hand to diagnose any issues straightaway.

© Webspection. All rights reserved. #1 SEO Agency in Cardiff - 95 Cathedral Road, Cardiff, CF11 9PG