I realize that you understand in the general sense how search engines work: you go to a search engine to help you find something, you enter in the search query for what you're looking for and the search engine gives you links to sites that are relevant to your search query. But one question you need to ask is: how'd they do that?
Think about it… A few hundreds of millions of times a day, people ask Google questions (enter search queries), and within a fraction of a second (typically less than .4 second), Google needs to decide which among the billions of pages on the web to show them and in what order.
This is a pretty big task to accomplish successfully millions of times a day, so let's break it down to see how it's done; tackling two of the biggest misconceptions about Google.
True or False: Each time a query is entered at Google; Google sends its search-spider out to all the websites to find the most relevant websites to list in the results?
If you answered "True," please find the closest hard object on your desk and boink yourself upside the head. If you answered "False," please stand up, smile real big and take a bow — you are brilliant.
So now all of you that answered "True" are wondering how it works. According to Google, each time a query is entered, it sends the query through its index servers to find which pages contain the words that match the query, then the query is passed to their document servers to retrieve the stored documents and generate snippets to describe the results and finally return the search results to the user… all in under a half a second.
When they index a page they apply more than 500 million algorithm variables against 2 billion terms to better understand the page, so they can accurately rank and tag it within their system for immediate retrieval.
But the ranking doesn't stop there. According to Google, they also apply what they call Hypertext-Matching Analysis, where the search engine analyzes page content. However, instead of simply scanning for page-based text (which can be manipulated by publishers), their technology analyzes the full content of a page and factors in fonts, subdivisions and the precise location of each word. They also analyze the content of neighboring pages to ensure the results returned are the most relevant to a user's query.
In layman's terms, the Googlebot compiles a massive index of all the words it sees and their location on each page, working from copies of our web pages that they store on their index and document servers.
They analyze the semantic content (words) of a web page to determine the relevance of the words used in the search query, taking into account the title tags, the heading tags and the text it detects on the page. It will also check out text related contextually to what it considers to be the main keywords and then rank that page according to how relevant it calculates it to be for the main theme of the page.
Then they examine the number of other web pages that are linked to it, and regard that as a measure of how important, or relevant to the query keywords, the page is. The value of the links is regarded as peer approval of the content. All of these factors determine how high that page is listed for search queries that are similar contextually to the content of the page.
So, you see, it is computationally impossible for Google to go out directly to websites with each query and return accurate results in a fraction of a second.
True or False: Each time the Googlebot comes through a webpage it is in fact making an indexed copy (cache) of that page. If you answered "True," put the dunce hat on. If you answered "False," then stand up and do the happy dance — you are correct.
According to Google, it uses an algorithmic process to determine which sites to crawl, how often, and how many pages to fetch and store from each site.
Google's crawl process begins with a list of web page URLs, generated from previous crawl processes, and augmented with Sitemap data provided by webmasters. As the Googlebot visits of each these websites, it detects links on each page and adds them to its list of pages to crawl. New sites, changes to existing sites, and dead links are noted and used to update the Google index.
In layman's terms, keep an eye on your server stats and you will see where Googlebot will just barely 'tap' or 'ping' a page and other times it sits there for a while apparently consuming the entire page (document). It is my belief, through monitoring and testing, the quick little pings are one of two common scenarios: either they discovered a link to your page while on another site — they ping it to verify it's a good link to a live page and then list your page for an upcoming full crawl; or, they discovered a link to your page, or site, while on another site — and are taking a look at the contextual relevance between the sites to determine if the peer vote (link) is of any value for the purpose of PageRank.
I imagine there are many other reasons for them to quickly 'tap' pages, but the two listed above are the most obvious.
Remember just because you see in your stats that Googlebot has recently come through does not mean it has indexed any of your page information or updated its cache of your page. You can always check the date on Google's cached version of your page to know when they last indexed the page (date is in the upper right corner of their cached copy).
Have you noticed throughout this article (well, all of my articles), that I keep trying to hammer into you that it is words that make all of this happen? I know I sound like a broken record, but if you still don't get it, maybe this will help…
A person wants to find what you offer on your website: They go to a search engine and enter what? The search engine looks through its massive index of what? To locate the most relevant sites to serve up on the results page in what format? In order for the person to decide from the results to go to your site they have to be convinced by your what?
The answer to all of these questions, and what most of your websites are dearly missing, is simple: "WORDS."
It really boils down to how you optimize your online communication skills. You have to learn how to concisely communicate, to both the search engines and the visitors (potential customers), what you offer on your website.
Always remember your website is an information transportation system and search engines are information retrieval systems — and words are quintessential to them playing well together.
