As the ‘long tail’ of the web grows, rigid categories are becoming less and less useful for finding things. Websites that are locked into a walled-garden approach – such as eBay, Craigslist, and most job boards – still tend to embrace category searching and browsing. Search engines that aim to index as much content as possible on the web, in contrast, are eschewing categories in favor of free-text search. Yahoo came to prominence as a category-driven directory, but comparing its 2001 home page to today’s shows how its focus has shifted to free-text search. Just about all of Google’s vertical search services have no categories at all.
This migration away from categories can be understood in terms of the ‘long tail’, a concept recently discussed by Searchblog and Bnoopy. As Bnoopy puts it, search engines experience a handful of extraordinarily common queries and millions of far less popular queries. Because the millions of less popular queries cannot be satisfactorily shoe-horned into a limited number of fixed categories, the rationale for categories itself becomes questionable.
Is there any way to salvage categories, given most people’s intrinsic desire to categorize things? In our view, dynamic clustering is one of the more interesting roads ahead. Instead of using predetermined categories, this approach generates category-like clusters on the fly, enabling people to refine their searches intuitively after running a search. Clusty is revolved entirely around this concept, while Indeed is also using this approach.