educational

How Search Engines Work

In June, Internet monitor Hitwise.com reported that Google now holds a 69.17 percent market share based on U.S. traffic. Since they're driving nearly 70 percent of all search traffic it's important for you to understand how they work.

I realize that you understand in the general sense how search engines work: you go to a search engine to help you find something, you enter in the search query for what you're looking for and the search engine gives you links to sites that are relevant to your search query. But one question you need to ask is: how'd they do that?

Think about it… A few hundreds of millions of times a day, people ask Google questions (enter search queries), and within a fraction of a second (typically less than .4 second), Google needs to decide which among the billions of pages on the web to show them and in what order.

This is a pretty big task to accomplish successfully millions of times a day, so let's break it down to see how it's done; tackling two of the biggest misconceptions about Google.

True or False: Each time a query is entered at Google; Google sends its search-spider out to all the websites to find the most relevant websites to list in the results?

If you answered "True," please find the closest hard object on your desk and boink yourself upside the head. If you answered "False," please stand up, smile real big and take a bow — you are brilliant.

So now all of you that answered "True" are wondering how it works. According to Google, each time a query is entered, it sends the query through its index servers to find which pages contain the words that match the query, then the query is passed to their document servers to retrieve the stored documents and generate snippets to describe the results and finally return the search results to the user… all in under a half a second.

When they index a page they apply more than 500 million algorithm variables against 2 billion terms to better understand the page, so they can accurately rank and tag it within their system for immediate retrieval.

But the ranking doesn't stop there. According to Google, they also apply what they call Hypertext-Matching Analysis, where the search engine analyzes page content. However, instead of simply scanning for page-based text (which can be manipulated by publishers), their technology analyzes the full content of a page and factors in fonts, subdivisions and the precise location of each word. They also analyze the content of neighboring pages to ensure the results returned are the most relevant to a user's query.

In layman's terms, the Googlebot compiles a massive index of all the words it sees and their location on each page, working from copies of our web pages that they store on their index and document servers.

They analyze the semantic content (words) of a web page to determine the relevance of the words used in the search query, taking into account the title tags, the heading tags and the text it detects on the page. It will also check out text related contextually to what it considers to be the main keywords and then rank that page according to how relevant it calculates it to be for the main theme of the page.

Then they examine the number of other web pages that are linked to it, and regard that as a measure of how important, or relevant to the query keywords, the page is. The value of the links is regarded as peer approval of the content. All of these factors determine how high that page is listed for search queries that are similar contextually to the content of the page.

So, you see, it is computationally impossible for Google to go out directly to websites with each query and return accurate results in a fraction of a second.

True or False: Each time the Googlebot comes through a webpage it is in fact making an indexed copy (cache) of that page. If you answered "True," put the dunce hat on. If you answered "False," then stand up and do the happy dance — you are correct.

According to Google, it uses an algorithmic process to determine which sites to crawl, how often, and how many pages to fetch and store from each site.

Google's crawl process begins with a list of web page URLs, generated from previous crawl processes, and augmented with Sitemap data provided by webmasters. As the Googlebot visits of each these websites, it detects links on each page and adds them to its list of pages to crawl. New sites, changes to existing sites, and dead links are noted and used to update the Google index.

In layman's terms, keep an eye on your server stats and you will see where Googlebot will just barely 'tap' or 'ping' a page and other times it sits there for a while apparently consuming the entire page (document). It is my belief, through monitoring and testing, the quick little pings are one of two common scenarios: either they discovered a link to your page while on another site — they ping it to verify it's a good link to a live page and then list your page for an upcoming full crawl; or, they discovered a link to your page, or site, while on another site — and are taking a look at the contextual relevance between the sites to determine if the peer vote (link) is of any value for the purpose of PageRank.

I imagine there are many other reasons for them to quickly 'tap' pages, but the two listed above are the most obvious.

Remember just because you see in your stats that Googlebot has recently come through does not mean it has indexed any of your page information or updated its cache of your page. You can always check the date on Google's cached version of your page to know when they last indexed the page (date is in the upper right corner of their cached copy).

Have you noticed throughout this article (well, all of my articles), that I keep trying to hammer into you that it is words that make all of this happen? I know I sound like a broken record, but if you still don't get it, maybe this will help…

A person wants to find what you offer on your website: They go to a search engine and enter what? The search engine looks through its massive index of what? To locate the most relevant sites to serve up on the results page in what format? In order for the person to decide from the results to go to your site they have to be convinced by your what?

The answer to all of these questions, and what most of your websites are dearly missing, is simple: "WORDS."

It really boils down to how you optimize your online communication skills. You have to learn how to concisely communicate, to both the search engines and the visitors (potential customers), what you offer on your website.

Always remember your website is an information transportation system and search engines are information retrieval systems — and words are quintessential to them playing well together.

Related:  

Copyright © 2024 Adnet Media. All Rights Reserved. XBIZ is a trademark of Adnet Media.
Reproduction in whole or in part in any form or medium without express written permission is prohibited.

More Articles

profile

'Traffic Captain' Andy Wullmer Braves the High Seas as Spirited Exec

Wullmer networked and hobnobbed, gaining expertise in everything from ecommerce to SEO and traffic, making connections and over time rising through the ranks of several companies to become CEO Mobile at TrafficPartner.

Alejandro Freixes ·
opinion

To Cloud or Not to Cloud, That Is the Question

Let’s be honest. It just sounds way cooler to say your business is “in the cloud,” right? Buzzwords make everything sound chic and relevant. In fact, someone uninformed might even assume that any hosting that is not in the cloud is inferior. So what’s the truth?

Brad Mitchell ·
opinion

Upcoming Visa Price Changes to Registration, Transaction Fees

Visa is updating its fee structure. Effective April 1, both the card brand’s initial nonrefundable application fee and annual renewal fee will increase from $500 to $950. Visa is also introducing a fee of 10 cents for each transaction, and 10 basis points — 0.1% — on the payment volume of certain merchant accounts.

Jonathan Corona ·
opinion

Unpacking the New Digital Services Act

Do you hear the word “regulation” and get nervous? When it comes to the EU’s Digital Services Act (DSA), you shouldn’t worry. If you’re complying with the most up-to-date card brand regulations, you can breathe a sigh of relief.

Cathy Beardsley ·
opinion

The Perils of Relying on ChatGPT for Legal Advice

It surprised me how many people admitted that they had used ChatGPT or similar services either to draft legal documents or to provide legal advice. “Surprised” is probably an understatement of my reaction to learning about this, as “horrified” more accurately describes my emotional response.

Corey D. Silverstein ·
profile

WIA Profile: Holly Randall

If you’re one of the many regular listeners to Holly Randall’s celebrated podcast, you are already familiar with her charming intro spiel: “Hi, I’m Holly Randall and welcome to my podcast, ‘Holly Randall Unfiltered.’ This is the show about sex, the adult industry and the people in it.

Women In Adult ·
trends

What's Hot Now: Leading Content Players on Trending Genres, Monetization Strategies

The juggernaut creator economy hurtles along, fueled by ever-ascendant demand for personality-based authenticity and intimacy — yet any reports of the demise of the traditional paysite are greatly exaggerated.

Alejandro Freixes ·
opinion

An Ethical Approach to Global Tech Staffing

One thing my 24-year career as a technologist working to support the online adult entertainment industry has taught me about is the power of global staffing. Without a doubt, I have achieved significantly more business success as a direct result of hiring abroad.

Brad Mitchell ·
opinion

Finding the Right Payment Partner

Whenever I am talking with businesses that are just getting started, one particular question comes up a lot: “How do I get a merchant account?” It’s a simple question, but it has a complicated answer.

Jonathan Corona ·
opinion

The Taxman Cometh for Every Business

February may be the month of romance, but it is also a time when we need to think about something that inspires very little love: taxes. April is not far away, and the taxman is always waiting. This year, federal and most state income taxes are due Monday, April 15.

Cathy Beardsley ·
Show More