h2g2 The Hitchhiker's Guide to the Galaxy: Earth Edition

How do search engines work? What criteria do they use to decide which results come where in their results?

Editorial Note: This conversation has been moved from 'Talking About the Guide - the h2g2 Community' to 'Ask the h2g2 Community'.

Someone always knows the answer here.

Mina

I'm not an expert, but as I understand it, it works like this. The search engines use automated algorithms called 'spiders' (I think) to search the web periodically. The results of these searches are used to compile a database of the 'contents' of the web. When you do a web search, the search engine uses your search string to access its database and find the websites that form the best match for your search string. In other words, the search is not done on the web at the time you enter the search string. Instead, it is done on information in a database that may not be up-to-date. This is why you may sometimes find that the URL you click on in the search results no longer exists.

As I say, this is my understanding of how it works. Hope this helps.

That's half the story... The s make a database containing the page URLs and the words and phrases appearing on them. Then each page is given a "rating" for each word or phrase, depending on how often it appears on the page, and how near the start of the page it appears. The better search engines (e.g. Google) also keep a tally of how many other pages link to the page, on the basis that a page with a lot of incoming links is probably a good source of information. All of this is used to decide the order in which the pages are listed in your results.

These criteria can be overridden in some engines if the owner of the page is prepared to bid the most cash for a particular "keyword", placing their page at the top of every search containing that keyword until someone comes along and outbids them. (See Altavista for an example of this...)