Our capacity to produce information is exploding. Our capacity to consume information however, remains relatively static.
A recently published study by scientists at University of California estimates that the annual amount of business-related information processed by the world's computer servers is equivalent to a 5.6-billion-mile-high stack of books from Earth to Neptune and back to Earth, repeated about 20 times. In other words, it’s a hell of a lot of stuff.
“The study estimated that enterprise server workloads are doubling about every two years,” according to a report in PHYSORG.COM. “This means that by 2024 the world's enterprise servers will annually process the digital equivalent of a stack of books extending more than 4.37 light-years to Alpha Centauri, our closest neighboring star system in the Milky Way Galaxy.”
Generally, such an explosion is a good thing. In poor, primitive, repressive societies, information production is low and deliberately kept that way. The more open a country’s society and economy, the more information it produces. The challenge is to manage all this information and make it truly useful and productive.
Part of the information management challenge is to determine whether the information we produce has any use to begin with. A 2008 study from NetApp Inc. and the University of California found that 90 percent of the data analyzed was not being used.
“Moreover, among the files that were opened, 65 percent were only opened once,” a Government Computer News report stated. “And most of the rest were opened five or fewer times, though about a dozen files were open 100,000 times or more.”
This is a classic example of the long tail (low demand or no demand stuff) and the long neck (high demand stuff). The long tail has been seen as a major opportunity, but it can become a major threat for the following reasons: 1.It takes time and energy to create and store long tail information. If there is no demand for this information why are we incurring these costs? 2.Long tail content can wrap itself around the long neck and choke it. It does this by cluttering the menus, links and search results.
As Dusty Horwitt pointed out in a 2008 Washington Post article, if you want to hide really important information, publish it with lots of long tail information. “Lawyers are familiar with this phenomenon,” Horwitt stated. “In fact, they use it to their advantage: They know that if you want to hide damaging information about a case, there's nothing like a document dump to do the trick. You make the facts freely available–along with so much irrelevant data that no one will ever find them.”
Government websites are often document dumps. It’s usually not a deliberate attempt to misinform. Rather, it involves an obsession with “freedom of information.” And, of course, publishing everything is always easier and politically less troublesome than selecting what to publish.
In modern societies, the challenge is not freedom of information but rather freedom from the unimaginable quantities of low level, useless, distracting and confusing information that is being produced at ever increasing speeds. As the long tail grows it becomes harder and harder to find the long neck.