A Short History of Search Engines and SEO

From Archie and Yahoo! to Google's PageRank and major ranking updates, how search engines and SEO evolved, and what lasts.

How to Redesign a Website · Last reviewed October 19, 2026 · 6 min read

Today, most people find websites by typing a few words into a search box. It is easy to forget that the early web had no search engines at all. CERN, where the web was invented, notes that the first web's search facility relied on keywords and that there were no search engines in the early years.

As the web grew from a few hundred servers to millions of sites, finding things became a real problem, and a whole industry grew up to solve it. Alongside it grew another industry: search engine optimization, or SEO, the practice of making websites easier for search engines to find, understand and rank. The history of the two is closely linked, and it explains a lot about what works in SEO today and what does not.

Before the Web Had Search

The idea of searching the internet predates the web. Around 1989 and 1990, Alan Emtage, then at McGill University in Montreal, created Archie, which the Internet Hall of Fame calls the world's first internet search engine. Archie did not search web pages. It collected lists of files stored on public file-sharing servers so people could look up a file by name.

Once the web took off, two different approaches emerged for finding pages:

Directories, where people sorted websites into categories by hand, like a library catalog
Crawler-based search engines, where software automatically visited pages, followed their links and built a searchable index

Directories and Early Search Engines

The best-known directory began in January 1994, when two Stanford graduate students, David Filo and Jerry Yang, started a list of their favorite sites called "Jerry and David's Guide to the World Wide Web." It was soon renamed Yahoo!, and according to the Centre for Computing History it was a human-edited directory rather than a search engine, letting people browse categories to find sites.

Crawler-based search arrived at around the same time. WebCrawler, created by Brian Pinkerton at the University of Washington, went live in April 1994 and is generally credited as the first web search engine to index the full text of pages, letting people search for any word on any page. Others soon followed, including AltaVista, which launched in December 1995.

The First Era of SEO

Early search engines mostly judged a page by what was on it: the words in its title and text, and special hidden labels called meta tags. That made them easy to game. Some site owners repeated popular words over and over, a practice known as keyword stuffing. Others hid text in the same color as the background so search engines could see it but visitors could not.

The keywords meta tag was a common target. Google later explained that because site owners could stuff it with often-irrelevant words that visitors never saw, the tag was so often abused that Google had stopped using it in ranking many years before its 2009 statement on the subject.

Google and the Power of Links

Larry Page and Sergey Brin met at Stanford University in 1995. By 1996 they had built a search engine called BackRub that, as Google tells it, used links to determine the importance of individual pages. It was renamed Google, and Google Inc. was formally founded in 1998. That year the pair also published a research paper, "The Anatomy of a Large-Scale Hypertextual Web Search Engine," describing how their system made heavy use of the link structure of the web.

The key idea, called PageRank, treated a link from one page to another as a kind of vote. Pages linked to by many important pages were probably important themselves. Results improved dramatically, and Google grew into the world's most widely used search engine. Google's own guide to its ranking systems says PageRank was one of its core systems when it launched and is still part of them, though it has evolved a lot.

There was a catch. Once people understood that links mattered, a new wave of manipulation began: buying links, swapping links in bulk and creating networks of low-quality sites purely to link to each other.

The Update Era

Google responded with a long series of changes to its ranking systems, some of which became famous by name. Google's documentation describes several that are now retired or folded into its core systems:

Panda, announced in 2011, was designed to better ensure high-quality and original content appeared in results. It became part of Google's core ranking systems in 2015.
Penguin, announced in 2012, was designed to combat link spam. It was integrated into the core systems in 2016.
Hummingbird, in August 2013, was a major improvement to Google's overall ranking systems.
The helpful content update, announced in 2022, aimed to reward content written by people, for people, rather than content made mainly to attract search traffic. In March 2024 it became part of the core ranking systems.

Google also introduced the mobile-friendly update in April 2015 and, in October 2019, announced it was using a language-understanding system called BERT to better grasp the meaning of searches. Google says it continues to improve its ranking systems regularly and gives notice of updates when that might be useful to site owners and content creators.

A Short Timeline

1989–1990: Archie lets people search lists of files on the internet.
January 1994: Yahoo! begins as a hand-built directory.
April 1994: WebCrawler launches with full-text search.
December 1995: AltaVista launches.
1998: Google Inc. is founded, and the PageRank paper is published.
2011–2012: Google's Panda and Penguin target thin content and link spam.
2022: Google announces the helpful content update.

What This Means for Your Website

The long arc of search history points in one direction. Tricks that exploit a search engine's weaknesses tend to work for a while and then stop working, sometimes with penalties. What has lasted is content that genuinely answers people's questions, a site that is easy for both visitors and search engines to navigate, and links that other sites choose to give because your pages are useful.

For a small business, that means clear page titles, honest descriptions of your products and services, pages that load quickly and work on phones, and helpful information written for your customers rather than for an algorithm. If an SEO offer promises shortcuts, it is worth remembering how many shortcuts Google has already closed.

Key takeaways

The early web had no search engines; directories like Yahoo! and crawlers like WebCrawler appeared in 1994.
Early SEO relied on on-page tricks such as keyword stuffing and stuffed meta tags, which search engines learned to ignore.
Google's use of links as signals of importance, described in 1998, changed search and led to a wave of link manipulation.
Updates such as Panda, Penguin and the helpful content system steadily rewarded quality content over shortcuts.
Lasting SEO today means useful, people-first content on a fast, mobile-friendly, well-organized website.

Sources

Spotted something out of date? Let us know.