Howlers: New Automation and Human Models Challenge Traditional Indexing

December 8, 2010  
Posted in Access Insights, Autoindexing, indexing, Taxonomy

by John Blossom
reprinted from www.shore.com/commentary

The National Federation of Abstracting and Information Services (NFAIS) is an interesting collection of institutions and publishers joining to tackle issues in aggregating content through indexing, abstracts and leading content technologies. While there are certainly a good number of indexing experts in this set there are also many executives in businesses and other publishing institutions who are trying to wrestle indexing issues to the ground that are critical to their success as publishers. NFAIS as an organization sees clearly how The New Aggregation is changing the way in which content value is being created, and many of its members do also. But at a recent daylong workshop on “Automated Indexing & Abstracting: Current Status and Future Trends” it was clear that new takes on indexing are needed to succeed in today’s content marketplace. Content products long centered on the value of traditional indexing and abstracting are struggling, daunted by the rise of automated indexing and search systems implemented by their clients and open Web search engines. Content indexing and categorization tools and taxonomy specialists are thriving in and of themselves, but the value that they provide has a hard time keeping up with the expectations of a generation of users trained by Web search interfaces.

Who’s really controlling the future of content indexing value? Increasingly it’s the “howlers,” as one person described them at the NFAIS session, active electronic content users who are involved in improving our ability to find it. Traditionally this has meant users providing feedback to editorial or support staffs on how documents get sifted into well-established categories, catching what indexers perceive as errors or “noise.” Professional indexers test and try new concepts and terms over time before adding them to controlled vocabularies. This leads to thinking that the “noise” of “misplaced” content is missing the point when in fact it may be the indexers who are missing the point. Increasingly the needs and interests of users trying to keep ahead of competitors and rivals move faster than established taxonomies and categories, with victories going to those who can divine the profits in new patterns before the next outfit, especially in rapidly blending global markets. The “howlers” at the edge of these changes who point to content with links and act to organize it themselves are pointing the way towards a new future for content organization.

Here are a few quick thoughts about where indexing’s power is moving in a world of “howlers”:

  • Indexing as a social phenomenon is still on the edges of the indexing profession. A lot of the excitement surrounding services such as del.icio.us is lost on professional indexers, with some good reason. Collaborative indexing is hardly a replacement for well-structured indexing methodologies when you’re trying to create content that can respond to authoritative needs of fact-based content systems. But all humans want to categorize things, to make sense out of chaos in a context that has social value as well as logical structure. This is not really so much a battle between absolutism and relativism as it is a struggle to have indexers accept that a community’s view of what creates that structure may be less relativistic than a small cadre of people not tuned into the latest thinking of the users who they serve. Socially-driven indexing’s output may be oftentimes trendy and narrow in scope, but oftentimes that’s a virtue in helping people find content that’s in the mainstream of a community’s thinking. Score one for social networks of “howlers.”
  • The value in content indexing is moving from “recall” to “precision.” Being able to have an indexing system capture nothing but “good stuff” on a given topic—”recall,” as the experts call it—has been the cornerstone of publishing value through indexing for a long time. Yet as highlighted at the NFAIS session by Craig Emerson, Vice President of Editorial Operations at CSA, publishers’ business models based on this kind of indexing expertise are not paying the bills like they used to. Scientific publishers in particular are challenged both by enterprise search engines and open Web search engines such as The National Library of Medicine’s PubMed and Google Scholar that emphasize “precision”—the ability to find all the “good stuff”, even if it means a little of the “noise” that may turn out to be useful after all in the eyes of a searcher. Precision-based systems that can use the input of subject matter expert “howlers” to refine rapidly shifting bodies of knowledge are winning the battle for content value in many instances.
  • Automated indexing is creating a powerful new generation of indexers. Far from eliminating indexing as a profession and an art, the proliferation of powerful software tools to aid indexers in automated indexing is empowering them to focus on the most critical aspects of indexing to improve performance. While in the transition to these tools redundancies increase as efficiencies improve, those who are sharp enough to take advantage of powerful programmable indexing systems such as Access Innovation’s Data Harmony suite will thrive. But it also means that new types of “experts” can enter the indexing fray using tools such as Verity’s K2 Profiler to allow distributed groups closer to the needs of specific communities of users to adapt indexing and taxonomies to more localized needs. This allows both corporations and publishing organizations such as the IEEE to create content more usable according to local and specialized terms cost-effectively in parallel with indexing more in line with broad industry standards. Blending the best of both rules-based indexing tools and content categorization driven by frontline users is a key bridge towards getting the “howlers” leveraging the best of existing indexing techniques and the power of Web-based collaborative thinking.

There is still a lot of mileage in traditional indexing approaches, but increasingly indexing systems are inputs into much more sophisticated content organization tools needed to support a wide variety of content retrieval and organization needs. Indexing will continue to be a highly profitable endeavor when it responds to a user base that is highly empowered by today’s content technologies. As long as they’re all howling off of the same sheet of music, that should make for some pretty handsome noise.

- John Blossom