Tell Googlebot to Crawl Faster

Google's Webmaster Tools has changed the interface that allows a webmaster to control how fast or how slow Googlebot crawls a site.

I just noticed the change today, in the past, there were 3 radio buttons for normal, faster and slower. Many webmasters could only set it to normal and slower because faster was disabled unless Google thought your site really needed to be updated faster and could handle it.

As you can see from the screenshot, the control has now been changed to a slider and presumably everyone can set it how they wish, but it's likely that Google may override the settings. Also, setting the crawl rate faster doesn't mean Googlebot will visit your site more frequently. The crawl rate just determines how spaced out one request will be from the other. If Google thinks your site should get 200 visits a day, you're going to get 200 visits a day. If you set a faster crawl rate, that just means those request could happen within the same hour.

One neat thing about the new interface is that it shows you what Google's recommended crawl rate for your site is. One one of sites the crawl rate was faster than the other. In the past, you could only judge by seeing your crawl stats. The top chart shows the number of visits, the second the number of kilobytes downloaded and the third the average time spent downloading a page.

As you can see from the charts, this site seems to be getting slower and the recommended crawl rate was set a little slower than on other sites. Probably because Google thought there was too much strain on the server. This wasn't the case. The drop in response time had to do with the use of a third-party API but the site had plenty of capacity. In this case, I instructed Googlebot to crawl faster.

But sometimes, you really do want to slow down your crawl rate. On another site that is fairly new, Googlebot visits the site 10 times more than real visitors do. It's not a site I'm actively promoting yet and while I don't mind getting the many pages crawled, it seems a waste to waste so much bandwidth on crawling compared to the actual site traffic. In that case I slowed it down.

If you want your site to be crawled less, setting a slower crawl rate can help accomplish that. For instance, if you only want Googlebot to visit your site 100 times a day, you would set your crawl rate so that Googlebot only visits ever 864 seconds.

To help figure out what you should set your crawl rate to take 86,400 and divide it by how often Googlebot should crawl your site. 86,400 is the number of seconds in a day.

If you're not familiar with the Google Webmaster Tools interface, these are the steps to make the change. The placement has also changed:

1. Log into Google Webmaster Tools
2. Click on the website you'd like to change in the Dashboard
3. Select Settings from the left menu bar
4. Scroll down to the Crawl rate section
5. Slide the slider either towards faster or slower
6. Click the Save button

Remember that these are just recommendations and Google may decide to do what it feels is best.