Beyond the SEO Monopoly: Inside the Quest to Build a Universal ‘Library of Websites’

In an era dominated by algorithmic search engines, social media silos, and hyper-optimized commercial content, a fundamental question is resurfacing among developers and internet historians: What happened to the open, discoverable web?

Recently, developer Manan Tandon raised this issue on the developer community platform Dev.to, proposing a conceptual project called the "Library of Websites." Unlike Google or Bing, which index the web to rank pages based on search engine optimization (SEO) and commercial relevance, Tandon’s proposed platform would serve as a continuously updated, structured, and democratic directory of every active website on the internet.

The proposal has sparked a wider debate about the "death of the organic web" and whether a decentralized, voluntary registry could break the monopoly of modern search giants. By looking back at the history of web discovery, analyzing the technical hurdles of mapping the modern internet, and examining community feedback, we can understand both the promise and the steep challenges of building a digital library for the modern age.

1. The Core Proposal: A Democratic Registry for the Internet

At its heart, the "Library of Websites" is envisioned not as a search engine, but as a universal, structured catalog. Rather than relying on autonomous web crawlers that scour the internet to rank pages, the project proposes a voluntary, verified registry.

+-------------------------------------------------------------+
|                  PROPOSED VERIFICATION FLOW                |
+-------------------------------------------------------------+
|  1. Website Owner registers domain on the "Library"         |
+-------------------------------------------------------------+
                              |
                              v
|  2. System generates a unique TXT record / Meta Tag         |
+-------------------------------------------------------------+
                              |
                              v
|  3. Owner deploys snippet to their site                     |
+-------------------------------------------------------------+
                              |
                              v
|  4. "Library" verifies site, categorizes by tech & niche    |
+-------------------------------------------------------------+
                              |
                              v
|  5. Site is indexed & continuously monitored for uptime      |
+-------------------------------------------------------------+

Key Features of the Proposed System:

Voluntary Verification: Similar to Google Search Console or domain verification processes, website owners would install a lightweight script, meta tag, or DNS TXT record to prove ownership and active status.
Discovery-First Indexing: The directory would categorize websites by industry, niche, and underlying technology stack, allowing users to browse them like books on a library shelf.
Uptime Tracking: The platform would periodically ping registered sites to ensure the database remains free of dead links ("link rot").
Democratized Visibility: Small, independent, and non-commercial websites would stand on equal footing with major corporations, bypassing the algorithmic biases of modern SEO.

The ultimate goal is to create a living, searchable map of the internet that prioritizes exploration and curiosity over commercial transaction and search intent.

2. Chronology: The Rise, Fall, and Monopolization of Web Discovery

To understand why a "Library of Websites" is being proposed today, we must look at how the experience of finding information on the internet has evolved over the last three decades.

  1991-1998: The Curated Web Era
  (Yahoo! Directory, DMOZ, Webrings connect users manually)
            |
            v
  1998-2010: The Algorithmic Boom
  (Google's PageRank dominates; automated crawling scales search)
            |
            v
  2010-2020: The Era of SEO Hegemony
  (Commercial optimization & social media silos centralize traffic)
            |
            v
  2020-Present: The Indie Web Renaissance
  (Search quality degrades; users seek alternative discovery models)

Phase 1: The Curated Web Era (1991–1998)

In the earliest days of the World Wide Web, search engines did not exist. Instead, users relied on human-curated directories.

1994: Jerry Yang and David Filo founded Yahoo!, which began as "Jerry and David’s Guide to the World Wide Web"—a hierarchical directory of websites organized by category.
1998: The Open Directory Project (ODP), also known as DMOZ (Directory Mozilla), was launched. It relied on a global network of volunteer editors to manually review and categorize websites. For nearly two decades, DMOZ was the closest equivalent to a universal library of the web.
The Webring Movement: Independent websites during this era often joined "webrings"—circles of related sites linked together sequentially, allowing users to click "Next" or "Previous" to travel through a niche community of pages.

Phase 2: The Algorithmic Shift (1998–2010)

As the web grew exponentially, human curation became impossible to scale.

1998: Google launched PageRank, an algorithm that evaluated websites based on the number and quality of links pointing to them. This automated approach revolutionized web discovery, making directories obsolete.
The Death of Directories: Yahoo! slowly phased out its directory, and DMOZ struggled to keep pace with the sheer volume of new domains being registered daily.

Phase 3: The SEO Hegemony and Enclosure (2010–2020)

Over the last decade, search engines transformed from discovery tools into highly monetized advertising platforms.

Is There a "Library of Websites" for the Entire Internet?📚

The SEO Industry: A multi-billion-dollar search engine optimization industry emerged, dictating how websites must be written and structured to rank on the first page of search results.
The Invisible Web: Small, personal, and hobbyist websites that did not optimize for search algorithms were pushed to the outer fringes of the internet, effectively becoming invisible to the average user.
Siloing: Social media platforms (Facebook, X, Reddit) enclosed user activity, discouraging external links and further reducing the organic discoverability of the open web.

Phase 4: The Indie Web Renaissance (2020–Present)

Today, widespread frustration with the "enshittification" of search engines—characterized by pages of sponsored ads, AI-generated content farms, and SEO-optimized clickbait—has triggered a resurgence of interest in alternative discovery. Projects like the IndieWeb movement, personal blogrolls, and Tandon’s "Library of Websites" proposal reflect a growing desire to return to a human-scale internet.

3. Technical Challenges and Supporting Data

While the concept of a "Library of Websites" is philosophically appealing, implementing it at scale presents massive technical, operational, and security hurdles.

The Scale of the Modern Web

To understand the scope of the challenge, we must look at the data surrounding the size of the internet today:

Metric	Estimated Value	Source / Context
Total Registered Domains	~360 million	Verisign Domain Industry Brief
Active Websites	~18-20% of registered domains	The remainder are parked domains, redirects, or dead links
Pages Indexed by Google	Over 100 billion	Google Search corporate reports
Common Crawl Database	~3 to 4 billion pages per crawl	Open-source web crawl data repository

To map even a fraction of the truly active, unique websites, a "Library of Websites" would need to manage metadata for tens of millions of domains.

The Verification Snippet: A Security and Performance Friction

Tandon’s proposal suggests using a client-side verification script. However, web security experts point out several drawbacks to this approach:

Performance Overhead: Developers are highly protective of their website load times. Adding an external Javascript snippet can negatively impact Core Web Vitals, which affects user experience.
Security Vulnerabilities: Executing third-party scripts on a site introduces potential vectors for Cross-Site Scripting (XSS) attacks. If the Library’s servers were compromised, malicious code could theoretically be injected into thousands of verified websites.
The Alternative Standard: A safer, industry-standard approach already exists. Instead of a script, websites can use a static verification file (e.g., library-verification.txt placed in the root directory) or a custom DNS TXT record, similar to how Let’s Encrypt or Google Search Console verify domain ownership.

Alternative Verification Options:
[Option A: DNS TXT Record] -> yourdomain.com IN TXT "library-verification-code-12345"
[Option B: Static File]     -> https://yourdomain.com/.well-known/library-websites.json

The Menace of Web Spam and Automation

If a registry becomes popular, it immediately becomes a target for bad actors. Without strict curation, the library would quickly be flooded by:

Link Farms: Networks of low-quality websites built solely to generate backlinks and boost SEO.
Domain Squatters: Parked domains containing nothing but ads and affiliate links.
Malware and Phishing Sites: Malicious actors looking to legitimize their scams by obtaining a "verified" badge in a trusted directory.

4. Community and Expert Perspectives

The developer community has reacted to the "Library of Websites" proposal with a mixture of nostalgia, technological skepticism, and constructive alternatives.

The Critique of Voluntary Registration

Many experienced developers argue that relying on voluntary registration is a flawed strategy. "The people who would take the time to register and install a script are already a self-selecting, highly technical crowd," noted one community member. "This means you would build a library of developer blogs, rather than a true representation of the entire internet."

Furthermore, non-technical website owners—such as local businesses, artists, and community organizations—would likely lack the technical know-how or motivation to install a verification tag, leaving the directory incomplete.

Existing Alternatives and Precedents

Several projects already attempt to solve this problem, though they approach it from different angles:

+-----------------------------------------------------------------------------+
|                         THE WEB DISCOVERY SPECTRUM                          |
+-----------------------------------------------------------------------------+
|  ARCHIVAL            |  RAW DATA            |  INDIE SEARCH  |  PROPOSED    |
|  The Wayback Machine |  Common Crawl        |  Marginalia    |  Library of  |
|                      |                      |  Search        |  Websites    |
+----------------------+----------------------+----------------+--------------+
|  Preserves historic  |  Provides raw crawl  |  Custom index  |  Voluntary,  |
|  pages; not built    |  data; requires data |  penalizing    |  categorized,|
|  for live discovery. |  science skills.     |  modern SEO.   |  democratic. |
+-----------------------------------------------------------------------------+

Marginalia Search: Developed by Viktor Loftqvist, Marginalia is an independent search engine designed specifically to index the "small web." It uses a custom search algorithm that actively penalizes modern SEO patterns, javascript-heavy frameworks, and commercial monetization, surfacing old-school, text-heavy hobbyist websites instead.
The Wayback Machine (Internet Archive): While the Internet Archive does catalog the web, it is designed as an archival tool to preserve historical versions of pages, rather than a discovery engine for active, modern sites.
Common Crawl: This non-profit organization crawls the web and provides its dataset to the public for free. However, accessing and searching this data requires significant technical expertise and data-science infrastructure.
Curlie (The Successor to DMOZ): After DMOZ shut down in 2017, a group of volunteer editors branched off to create Curlie. While it still exists, it suffers from a lack of visibility and a dwindling editor base.

5. Implications: Redesigning the Future of Web Discovery

The discussion surrounding a "Library of Websites" is more than a technical debate; it is a philosophical inquiry into how we want to interact with digital information in the future.

Restoring Serendipity to the Internet

In the modern search paradigm, we only find what we are explicitly looking for. We type a query, and an algorithm serves a direct answer. This utility-driven approach has stripped the internet of serendipity—the joy of stumbling across an interesting, weird, or beautiful website that we didn’t know existed.

A categorized library would reintroduce the "browsing" experience. Much like walking down the aisles of a physical library, users could explore categories like "Handmade Woodworking," "Amateur Astronomy," or "Personal Philosophy Journals," discovering independent creators who have no interest in playing the SEO game.

The Role of AI in Automated Curation

To make a project like the "Library of Websites" viable without relying on a massive army of human editors (which led to the collapse of DMOZ), developers could leverage modern Artificial Intelligence.

Instead of manual categorization, an AI model could analyze a registered website’s homepage, extract its core themes, identify its tech stack, and assign it to appropriate categories. This hybrid approach—combining voluntary domain submission with automated, AI-driven classification—could finally solve the scalability issue that doomed early web directories.

Conclusion: A Call for a More Human Web

The proposal for a "Library of Websites" highlights a deep-seated cultural fatigue with the corporate, optimized, and centralized internet. While the technical execution of a universal, verified directory remains incredibly complex, the hunger for alternative discovery mechanisms is undeniable.

Whether through a voluntary registry, algorithmic counter-movements like Marginalia, or decentralized webrings, the community’s response to Manan Tandon’s proposal proves one thing: the desire for a open, diverse, and explorable internet is very much alive. The tools to map it may change, but the quest to preserve the web as a public library—rather than a shopping mall—continues.