More Info On When The GoogleBot Visited Your Page

Vanessa Fox posted some more information about the Google Bot, this time she explained how do you know when the Google Bot successfully crawled your site. Here are some of the highlights.

if Googlebot accesses a page and gets a 304 (Not-Modified) response to a If-Modified-Since qualified request, Googlebot doesn't download the contents of that page. This reduces the bandwidth consumed on your web server.

Which is very good to know because I had a little problem with bandwith in our forums which was the google bot over crawling the forum and taking up all my bandwith.

When you look at Google's cache of a page (for instance, by using the cache: operator or clicking the Cached link under a URL in the search results), you can see the date that Googlebot retrieved that page.

So now you know thta your cache updates every time the Google Bot crawls your site SUCCESSFULLY, because the crawler can come to update a few things but wont deep crawl sometimes.

Vanessa Fox's Entry | Google Sitemaps | Google's Add URL Page | Digg This Article