Updated October 9, 2010Doc Sheldon
In this series, we’ll be looking at search engine technology as it was, and is now. We’ll also look at how the optimization efforts employed by site owners can affect the user’s experience. Eventually, we’ll work our way up to what search engine technology needs to be to satisy John Q. User’s needs. Finally, we’ll take an objective look at how likely those necessary changes are, and how they’ll affect the site owners, the SEOs and most importantly, the users.
The first search engines, in the early 1990s, had much less of an internet to deal with than is the case today. Those that didn’t simply buy access to the index of another search engine, often limited their indexing to smaller, manageable chunks of the ‘net. Their searches were pretty well limited to their ability to find a verbatim reference on a page, that corresponded to that entered in the search query.
In 1993, Excite was created, and became the best-known search engine to date, although it, too, matched only verbatim terms.
In October of that year, Aliweb made its appearance. It used no bots, but crawled meta data of the sites that were manually submitted to it.
Just before Christmas of ’93, the World Wide Web Worm showed up. It would return indexed titles and URLs, but in the order in which they were found, rather than in any ranked order.
The following month was born infoseek, which allowed webmasters to submit their sites in realtime.
Almost simultaneously, altavista was launched, bringing several innovations with it. Natural language queries were possible, URLs could be added or deleted within 24 hours, advanced search techniques could be employed and several other new features were introduced.
In April of ’94, both WebCrawler and the Yahoo Search Directory appeared. WebCrawler was the first crawler to index entire pages, but unfortunately it could only be used at night, due to its popularity and limited resources.
Lycos was launched in July of 1994, actually ranking relevance in their retrievals, and introducing word proximity to their search. Originally launching with a catalog of 54,000 documents, they increased that to 1.5 million by the following January.
By January of 1996, the internet was ready for something new. Larry Page and Sergey Brin heard the call and began working on BackRub, a prototype search engine that utilized backlinks for establishing an “authority” ranking for a site.
In May of ’96, Inktomi’s Hotbot was introduced, which pioneered paid inclusion.
Nearly a year later, in April of ’97, Ask Jeeves launched their natural language search engine, powered by DirectHit, which ranked links by popularity, and hence, was very easy to spam.
1998 saw the launch of two search giants… MSN Search and Google. MSN was powered by Overture, Looksmart and Inktomi initially, while they watched to see if Google’s backlinks model would prove viable. Google, meanwhile, initially struggled with the fact that noone wanted to purchase their PageRank technology.
Around the middle of 1998, the Open Directory Project was launched, a huge internet directory managed by human editors. While not a search engine, the dmoz directory was nevertheless a major player in internet development.
Next, in May of ’99, AllTheWeb appeared, with a sleek new interface and some advanced features that raised a few eyebrows.
October of 2005 brought us Snap, but it was too complicated for the average user, and never really caught on.
Then in Sept. of 2006, MSN gave us LiveSearch.
June of 2008 saw cuil hit the internet.
And MSN rebranded LiveSearch as Bing in June of 2009.
All the foregoing certainly doesn’t name all the search engines that came and went since 1990. There were many players, but these were the majors.
As you can see, we progressed from manual submittals of title and URL, through crawling of submitted pages’ meta data, on to automated crawling of entire pages, to what we have today.
It’s been an interesting two decades, during which time many fortunes were made and lost, much fame was gained, and some lost, and much progress was made in the capability of searching hundreds of billions of pages, and returning a result for our query in almost no time at all.
At least one major player would say, “in an Instant“.
In the next installment, we’ll look at where search is today.