Haystacks of Needles

Our problem isn't too much noise. It's too much signal.

Before starting New Cartographies, I wrote a blog called Rough Type for nearly twenty years. While I’m getting my footing here on Substack, I thought I’d supplement my new posts with some of my favorites from the Rough Type archives — ones that might still have some relevance to our situation. So, on the occasional Sunday, I’ll publish an old post here, as what I’m calling a Sunday Rerun. This first installment, from 2011, looks at information overload.

---

Van Gogh, Haystacks in Provence

“It’s not information overload. It’s filter failure.” That was the theme of an influential talk that NYU professor Clay Shirky gave at a technology conference a few years back. It’s an idea that’s easy to like. It feels intuitively correct. It’s also reassuring: Better methods of filtering information will help alleviate information overload, and better filters are things we can actually build. Information overload, and the distractedness it entails, isn’t an inevitable side effect of digital media. It’s a problem that has a solution. So let’s roll up our sleeves and start coding.

There was one thing that bugged me, though, about Shirky’s idea, and it was this: The quality and speed of our information filters have been improving steadily for a few centuries, and have been improving extraordinarily quickly for the last two decades, and yet our sense of being overloaded with information is stronger than ever. If improved filters are going to reduce overload, then why haven’t they done so up until now? Why don’t we feel that information overload is subsiding as a problem rather than getting worse? The reason, I’ve come to believe, is that Shirky’s formulation gets it precisely backwards. Better filters don’t mitigate information overload; they amplify it. It would be more accurate to say: “It’s not information overload. It’s filter success.”

But let me back up a little, because it’s more complicated than that. One of the traps we fall into when we talk about information overload is that we’re usually talking about two very different things as if they were one thing. Information overload takes two forms, which I’ll call situational overload and ambient overload. They need to be treated separately.

Situational overload is the needle-in-the-haystack problem: You need a particular piece of information — in order to answer a question of one sort or another — and that piece of information is buried in a bunch of other pieces of information. The challenge is to pinpoint the required information, to extract the needle from the haystack, and to do it as quickly as possible. Filters have always been pretty effective at solving the problem of situational overload. The introduction of indexes and concordances — made possible by the earlier invention of alphabetization — helped solve the problem with books. Card catalogues and the Dewey decimal system helped solve the problem with libraries. Train and boat schedules helped solve the problem with transport. The Reader’s Guide to Periodicals helped solve the problem with magazines. And search engines and other computerized navigational and organizational tools have helped solve the problem with online networks and databases.

Whenever a new information medium comes along, we quickly develop good filtering tools for sorting and searching its contents. That’s as true today as it’s ever been. In general, I think you could make a strong case that, even though the amount of information available to us has exploded in recent years, the problem of situational overload has continued to abate. Yes, there are still frustrating moments when our filters give us the hay instead of the needle, but for most questions most of the time, search engines and other digital filters, or software-based, human-powered filters like email or Twitter, are able to serve up good answers in an eyeblink or two.

Situational overload is not the problem. When we complain about information overload, what we’re usually complaining about is ambient overload. This is a different beast. Ambient overload doesn’t involve needles in haystacks. It involves haystack-sized piles of needles. We experience ambient overload when we’re surrounded by so much information that is of immediate interest to us that we feel overwhelmed by the neverending pressure of trying to keep up with it all. We keep clicking links, keep refreshing screens, keep opening tabs, keep checking email in-boxes, keep glancing at social-media feeds and notifications, keep scanning Amazon and Netflix recommendations — and yet the pile of interesting information never shrinks.

The cause of situational overload is too much noise. The cause of ambient overload is too much signal.

The great power of modern digital filters lies in their ability to make information that is of inherent interest to us immediately visible to us. The information may take the form of personal messages or updates from friends or colleagues, broadcast messages from experts or celebrities whose opinions or observations we value, headlines and stories from writers or publications we like, alerts about the availability of various other sorts of content on favorite subjects, or suggestions from recommendation engines — but it all shares the quality of being tailored to our particular interests. It’s all needles. And modern, algorithmic filters don’t just organize that information for us; they push the information at us as alerts, updates, streams. We sometimes point to spam as an example of information overload. But spam is just an annoyance. The real source of information overload, at least of the ambient sort, is the stuff we like, the stuff we want, the stuff we crave. And as filters get better, that’s exactly the stuff we get more of.

This is not an indictment of modern data-filtering mechanisms. They’re doing what we want them to do: find interesting information and make it visible to us. But it does mean that if we believe that improving filtering technology will save us from information overload, we’re going to be very disappointed. The technology that creates the problem is not going to make the problem go away. If you really want a respite from information overload, pray for filter failure.