Google has been working on overhauling their algorithm in how they crawl, index and rank web content. The new algorithm is called Google Caffeine and has been in the works for quite a while. Google said they would be rolling it out in phases and just recently at SMX Advanced Matt Cutts announced that they have officially fully released Google Caffeine.
The web is continually growing in size and in the amount of fresh content that is provided by users. With the growing number of ways that content can be added to the web and the type of content that can be added on one web page from video content, real-time content, news content, images and more the average webpage is more complex to crawl.
Google has made the new algorithm, Caffeine, so that they can provide 50% fresher results over the old algorithm. What this means is that the new algorithm crawls much faster and much more sophisticated. If you were to look at the amount of pages that Google Caffeine crawls in a single second it would look like a stack of papers that would equate to 3 miles high.
Here is an example of how the new crawler looks as opposed to the old crawler:
Image courtesy of Google
With this new change of how Google crawls the internet they also change in how updated their index is. The old algorithm would consist of Google updating their algorithm every few weeks or so. Google Caffeine will analyze the web in small portions and update their index on a continual daily basis. It is also worth noting that their crawler, Googlebot, crawls from several IP locations in the United States, Googlebot does not crawl from IP addresses located in other countries.
With this new technology it will be very important for SEO’s to continually advise their clients to add unique content to their websites as often as possible. The more Googlebot is on your site the better as you can easily tell Googlebot that you constantly have good content that you add to the web and that your content should be updated on a regular basis.