SEARCH ENGINES
The vast majority of search engines use spiders. These ingenious software programs have only one task — to crawl the Web 24 hours a day, finding and indexing web pages.
These spiders (also called “bots” or “crawlers”) visit a web page, read it, and then follow links to other pages within the site. The spider revisits websites on a regular basis (e.g. every month or two) to look for changes.
Everything a spider finds is put in the search engine’s index. The index, sometimes called the catalog, is a giant database that contains a copy of every web page that the spiders find.
When a web page changes, the index is updated with the new information, but it doesn’t happen immediately — it can take a while (as much as six weeks) for new pages or changes to be added to an index.
So although a web page may have been “spidered” it may not have been “indexed,” and until such indexing occurs, that new page is not available to those searching with that search engine.
Search engines do not index the entire Web (although it may seem like they do). Most also don’t include dynamically created web pages like library web catalogs or other data behind CGI-walls.
And none index the entirety of every website, nor do they share a common search language (i.e. the algorithm used to determine what is searched and ranking of the searched item varies depending on the search engine).
However, there are three important elements that are common to all search engines:
1. The database operates on the same principles as your website’s database. The database consists of indexed descriptions of web pages including a link list with a small description for each link. When a search request is received from a surfer, these databases utilize special search algorithms, using keywords, to find needed web pages.
2. Search engines give each page they find a ranking as to the quality of the match to the surfer’s search query. Relevant scores reflect the number of times a search term appears, if it appears in the title, if it appears at the beginning of the page or HTML tags, and if all the search terms are near each other.
Some engines allow the user to control the relevance score by giving a different weight to each search word.
A search term used too many times within a page can be considered web spamming (for which search engines penalize) so don’t overdo the use of a keyword or phrase on a page (don’t exceed the 15-25 count range).
Each search engine has its own peculiar ranking method.
For example, if there are no links to other sites or pages within a website (a single page website) some search engines will not list that website.
The vast majority of search engines use spiders. These ingenious software programs have only one task — to crawl the Web 24 hours a day, finding and indexing web pages.
These spiders (also called “bots” or “crawlers”) visit a web page, read it, and then follow links to other pages within the site. The spider revisits websites on a regular basis (e.g. every month or two) to look for changes.
Everything a spider finds is put in the search engine’s index. The index, sometimes called the catalog, is a giant database that contains a copy of every web page that the spiders find.
When a web page changes, the index is updated with the new information, but it doesn’t happen immediately — it can take a while (as much as six weeks) for new pages or changes to be added to an index.
So although a web page may have been “spidered” it may not have been “indexed,” and until such indexing occurs, that new page is not available to those searching with that search engine.
Search engines do not index the entire Web (although it may seem like they do). Most also don’t include dynamically created web pages like library web catalogs or other data behind CGI-walls.
And none index the entirety of every website, nor do they share a common search language (i.e. the algorithm used to determine what is searched and ranking of the searched item varies depending on the search engine).
However, there are three important elements that are common to all search engines:
1. The database operates on the same principles as your website’s database. The database consists of indexed descriptions of web pages including a link list with a small description for each link. When a search request is received from a surfer, these databases utilize special search algorithms, using keywords, to find needed web pages.
2. Search engines give each page they find a ranking as to the quality of the match to the surfer’s search query. Relevant scores reflect the number of times a search term appears, if it appears in the title, if it appears at the beginning of the page or HTML tags, and if all the search terms are near each other.
Some engines allow the user to control the relevance score by giving a different weight to each search word.
A search term used too many times within a page can be considered web spamming (for which search engines penalize) so don’t overdo the use of a keyword or phrase on a page (don’t exceed the 15-25 count range).
Each search engine has its own peculiar ranking method.
For example, if there are no links to other sites or pages within a website (a single page website) some search engines will not list that website.