designing engines

Designing for Search Engines But, you might ask, how do crawler-based search engines go about determining relevancy, when confronted with hundreds of millions of web pages to sort through? They use algorithms, i.e. a set of rules that govern their spiders’ crawling techniques, indexing...