Aggregators Rule; Search Engines Drool

This article is from the archive of our partner .

The Internet is a big place, and, as Paul Kedrosky explains in a recent blog post, our methods of navigating it have changed over the years. In those heady early days of the late '90s--back when Eve 6 was on the radio and anything seemed possible--there were the aggregators, sites like Denis Dutton's Arts & Letters Daily. These sites, explains Kedrosky, offered handpicked guides to the best of the 'net: "People found interesting things on the web, created directories of those things, and then you found what you were looking for inside those curated lists."

But gradually there got to be more and more stuff, until, in a classic John Henry situation, the machines overtook the man. "The web got bigger faster than anyone could keep track," writes Kedrosky. "Curation steadily gave way to algorithmic search, which at first was just spidering of the web, and then more intelligent spidering with keywords. And then it became Google"--Google! You've heard of Google--" with ranking algorithms that placed websites into a hierarchies of keyword-related relevance based on things like authoritativeness, as defined, in part, by links from other sites--by those original hand-curated lists, ironically enough."

And now it's 2011, and Google's not the secure fortress it once was. "Any algorithm can be gamed; it's only a matter of time," writes Kedrosky. "The Google algorithm is now well and thoroughly gamed... It no longer has lists to draw and, on its own, it no longer generates the same outperformance -- in part because it is, for practical purposes, reverse-engineered, well-understood and operating in an adaptive content landscape." Kedrosky goes on to say, memorably, that "it has turned search back into something like it was in the dying days of first-generation algorithmic search, like Excite and Altavista: results so polluted by spam that you often started looking at results only on the second or third page--the first page was a smoking hulk of algo-optimized awfulness."

What's the answer? Oh, God, what will we do? Kedrosky suggests that to see the future, we may have to look to the past; that all of this has happened before and all of it will happen again; basically, that the beginning is the end is the beginning. Specifically, he says: "We could get better algorithms, which is happening to some degree, with search engines like Blekko and others. Or, we could head back to curation, which is what I see happening."

He goes on:

The re-rise of curation is partly about crowd curation -- not one people, but lots of people, whether consciously (lists, etc.) or unconsciously (tweets, etc) -- and partly about hand curation (JetSetter, etc.). We are going to increasingly see nichey services that sell curation as a primary feature, with the primary advantage of being mostly unsullied by content farms, SEO spam, and nonsensical Q&A sites intended to create low-rent versions of Borges' Library of Babylon. The result will be a subset of curated sites that will re-seed a new generation of algorithmic search sites, and the cycle will continue, over and over.

In short, curation is the new search. It's also the old search. And it's happening again, and again.

In the interest of added value, here's that last excerpt again, this time with helpful links provided by The Atlantic Wire.

The re-rise of curation is partly about crowd curation -- not one people, but lots of people, whether consciously (lists, etc.) or unconsciously (tweets, etc) -- and partly about hand curation (JetSetter, etc.). We are going to increasingly see nichey services that sell curation as a primary feature, with the primary advantage of being mostly unsullied by content farms, SEO spam, and nonsensical Q&A sites intended to create low-rent versions of Borges' Library of Babylon. The result will be a subset of curated sites that will re-seed a new generation of algorithmic search sites, and the cycle will continue, over and over.

In short, curation is the new search. It's also the old search. And it's happening again, and again.

This article is from the archive of our partner The Wire.