Joost de Valk
Joost de Valk is the founder and Chief Product Officer of Yoast. He’s a digital marketer, developer and an Open Source fanatic.
A question that often pops up for sites that are chasing “hot topics” is: how to get Google to crawl our site faster? This could be either because you have a big site and want to see changes faster or because you’re optimizing a news site. There are a few things you can do to improve your crawl speed. We’ll cover them in this post.
There are a couple of possible reasons why Google is slow when spidering your site. The first might seem obvious: if Google doesn’t find enough (quality) links pointing to your site, it doesn’t think your site is very important. The other reasons are technical: Google has too much to crawl on your site, your site is too slow, or it’s encountering too many errors.
Your site doesn’t have enough authority
When your site doesn’t have enough quality inbound links, Google will not crawl your site very quickly. You can imagine it doesn’t necessarily want to spend much time on a site that it doesn’t consider important. When your site is brand new, this will be the case.
So, when you need more crawl action on your site, start doing some link building. We’ve written about that topic before, I’d suggest reading these three posts. They cover what you need to know to get started, so I won’t go into it any further here:
- Link building from a holistic SEO perspective
- 5 link building DON’Ts you didn’t know about
- 6 steps to a successful link building strategy
If the above three posts are not enough for you, there’s more in our link building archive.
Technical reasons for slow crawling
The technical reasons for Google to crawl your site slowly can be divided into three groups: your site is too slow, you have too many errors, or you have too many URLs.
Your server is slow
The main reason why we see Google crawling sites slowly is when the site itself is actually slow. If your website is slow to respond to requests, or loads lots of bulky resources, Google might intentionally lower their crawl frequency and depth (or limit your crawl budget) to try to prevent things from breaking.
If that’s the case, you probably have larger issues with site speed, and should read up on how to measure your site speed, and how to speed things up!
Too many errors on your site
If you have a lot of errors on your site for Google, Google will start crawling slowly too. To speed up the crawl process, fix those errors. Simply 301 redirect those erroring pages to proper URLs on your site. If you don’t know where to find those errors: log into Google Search Console. If you have access to your site’s access logs, you can also look at those, preferably with a tool like Screaming Frog’s Log file analyzer. To prevent your site from being crawled slowly, it’s important that you regularly look at your site’s errors
This article was written by Joost de Valk and originally published on Yoast • SEO for everyone.