To keep up with the growth of the web, search engine technology has had to scale dramatically. The World Wide Web Worm (WWWW) one of the first web search engines, in 1994, had an index of 110,000 web pages and web accessible documents. As of November, 1997, the top search engines claim to index from 2 million (WebCrawler) to 100 million web documents.

A comprehensive index of the Web will contain over a billion documents, foreseeable by the year 2000. At the same time, the number of queries search engines handle has grown remarkably also. In March and April 1994, the World Wide Web Worm received an average of about 1500 queries per day. In November 1997, Altavista claimed to have handled about 20 million queries per day.

By the year 2000, with the growing number of users on the web and automated systems which query search engines, it seems likely that top search engines will deal with hundreds of millions of queries per day. The goal of our system is to address the various problems, both in quality and scalability, introduced by scaling search engine technology to such extraordinary numbers.