Today, I had a developer contact me with a problem. His client was saying that Google was not crawling the website he had built for him. This developer was concerned and wanted to know if I could help. At this point, I thought it might be helpful to explain how to tell if Google is crawling your website. So that’s what this post will cover (along with a few bonuses).
How To Tell If Google Is Crawling Your Website
In order to tell if Google was crawling this person’s website, I wanted to first find out if Google had indexed it. If you want to know if Google is crawling your website, just go to google.com and type this in the search bar (we’ll use CNN.com as an example):
Here’s what you’d see if you did this (with AdBlock and the MozBar turned on):
So that is how to tell if Google is crawling your website. Google has clearly crawled www.cnn.com. They would not have indexed all of these pages had they not. Web crawlers are designed to crawl, digest, interpret, and index content from all over the Internet. You aren’t going to find indexed content that hasn’t been crawled. To find out more about how web crawlers work, click here.
Back to this developer. I emailed him back and attached screenshots that showed the website he built is indeed being crawled and indexed by Google. I also advised him to set up Google Webmaster Tools, verify his site, and upload an XML sitemap to his account. This helps the Google Web crawler to find content that might be a bit harder to find on his domain.
Clearly though, this developer’s problem wasn’t that his client’s site wasn’t being crawled. So why was he getting this email from his client?
My theory: His client wasn’t complaining that his site wasn’t being crawled. He was complaining that his site wasn’t generating new customers. So I ran some more tools to check out what was going on behind the scenes.
Right off the bat, I could see the problem. His client’s problem wasn’t that Google isn’t crawling the site. The problem is that the site has little to no authority or trust.
Search engines like Google make money by providing search results that are trustworthy and authoritative. If they don’t, searchers go somewhere else. If searchers go somewhere else, search engines don’t make their advertising money. So they aren’t going to rank a site like this at the top of page 1 for many keywords (if any at all).
Let’s break it down. This site has a domain authority of 11/100, a page authority of 23/100, a trust flow of 2/100, a citation flow of 11/100, and 5,599 backlinks from only 5 unique domains. For definitions on all of these metrics, check out our SEO terminology glossary.
For those of you that do not wish to check out the glossary, here’s the short version of this situation. This site has some really poor stats. Not only that, but the backlinks (all 5,599) are really spammy. There are only 5 domains linking back to them. That’s nearly 1,120 backlinks per domain, which just reeks of spam – even before you look at their anchor text (again, check out the glossary), which is also very spammy.
At best, this website has no trust and authority and they need to start building it.
At worst, this website has negative trust and authority and needs to start over on a new domain.
Either way, the problem isn’t that Google wasn’t crawling this website. It’s that this website had bigger issues. This developer’s client wants to rank like a rockstar, but you can’t be a rockstar with Google when you have little authority, little trust, and a spammy backlink profile. This isn’t 1998 anymore. But if it was, we’d be jamming to this:
That album is full of jams.
If you want to rank like a rockstar, you need to act like one. Learn on-page SEO. Learn how to build links. Get down with the Semantic Web. Learn how to optimize a WordPress blog post. Do something. Don’t blame it on Google not crawling (or indexing) your site. As long as you haven’t blocked a search engine via your robots.txt file, this is more than likely not the reason you aren’t ranking, not getting exposure, not getting leads, and not closing them.
So…in summary. I’ve talked about how to tell if Google is crawling your website. It’s not hard to find out. But once you do, you’re likely to realize that isn’t your problem. It’s probably something that is harder to fix. It takes a lot of work, and that’s why I love what I do. I get to help people figure it out all day (and I get to learn along the way too).