In August Google launched a brand new way for site owners to submit a request for specific pages within their website to be crawled by the Googlebot. What you need to know right away is that use of this new tool will be limited, so it is important that you only use it when you need a specific page to be crawled immediately. However, it should be noted that Google does not guarantee that every page submitted via this new tool will be indexed within their popular search engine - the new crawl request feature - known simply as "Fetch as Googlebot" - will at least in some way improve the amount of time required to have your page crawled.
How Google Works
Google lists websites and Web pages within its ever-expanding index by crawling a Web page and then processing that page through a series of algorithmic processes to determine whether the page should be stored within the index - or not. Google initially finds out about various Web pages in a number of different ways and then adds those pages to its scheduling system to be crawled. The list of pages-to-be-crawled is sorted via yet another algorithm which prioritizes the pages based upon specific factors.
Those factors include: the frequency of content updates, the importance of the content (i.e. news, etc.) and the value of the page, which is based upon the ranking system used for Google's PageRank. The bottom line is that Google does not send the Googlebot to crawl every page it becomes aware of, and it doesn't add every page it crawls into the index.
How Google Finds You
There are a couple of different ways that Google can find your Web pages - aside from the new Fetch as Googlebot tool. Google mainly discovers Web pages by following links. Most Web developers know the importance of linking to every single page on their website with internal links, but few know why it's so important. Google also uses RSS feeds as a method of discovery. If your website is well-linked internally and externally on other relative websites, chances are Google already knows you exist.
Here are some other ways that Google can discover your Web pages:
- Add URL - You've probably noticed that "Add URL" link on the Google home page. Google invites anyone to request that a URL get added into their search engine index. Initially it was added for people who were looking for a specific link but were unable to find it so that it could be found easily by the next person, but as you might imagine it was soon used and abused by over-eager Web developers vying for a good listing. As a result, the requests made via this tool are ranked much lower than other discovery methods.
- XML Sitemaps - You can submit an entire list of URLs to Google - and to Bing - using this tool. While the engines won't guarantee that they will crawl each and every URL that gets submitted, they do add the URLs to their crawl to-do list to be ranked with the other discoveries.
- Fetch As Googlebot - The latest submission tool from Google. Despite the public launch in early August, this tool has actually been around much longer - it's just now "official" and out of beta. Fetch as Googlebot allows you to direct Google to crawl a specific URL that you have verified to be your own so you can see what Googlebot "sees" via the server. This data can then be used to help you de-bug any issues that you might not notice otherwise when you view the site in a browser window. Once you ask Googlebot to crawl your page and make the necessary changes to fix any issues you discover, all you need to do is hit the "Submit" button to have it added to the index.
Fetch As Googlebot: The Advantages
While some of the other methods are much more advantageous at getting your Web pages into the queue, there are some advantages associated with using this new feature. First of all, it is an ideal tool to use when you add new pages to your site. It is also helpful to use when you do a major structural update. Google states in the description that you can use the Fetch as Googlebot tool to increase the time it takes to remove URLs from the index or to update cached pages in your listings.
You now have the ability to add as many as 50 URLs a week, but you can also submit all of the crawled URLs and pages that are linked from that URL. Linked page submissions are limited to just 10 a month, so make sure to utilize this tool wisely. It is also very important that you have a solid link structure for your internal and external links if you are going to use this tool. Again, it would be a good idea to utilize this tool when you make a major update or add a new section to your website. Because you must be a verified site owner to use the Submit All URLs tool, this is a much better choice than the public Add URL or Crawl URL tool. However, it is important to note that Google's basic rules still apply: just because you submit it, doesn't mean that it will be listed in the index - it just means you will jump ahead in the crawling process.
Click for information and details, visit Google's Answer Page on Fetch as Googlebot: https://www.google.com/support/webmasters/bin/answer.py?hl=en&answer=158587"
