If the editors of a magazine—NEWSWEEK, for instance—want to know what interests their readers, their resources are limited. They can count cover sales, but that only tells them about one story a week. They can convene a focus group, but that's a cumbersome and costly way to assess the tastes of 3 million subscribers. Online, by contrast, that information is available for the asking—not just the number of readers, but how long they spent with a given story and what else they read. So as journalism increasingly migrates to the Web, the job of figuring out what readers want becomes almost automatic—thereby raising the question, how much do we really need editors anyway?
Just kidding! But according to a new book by Ian Ayres, an econometrician and law professor at Yale, this is a microcosm of a powerful trend that will shape the economy for years to come: the replacement of expertise and intuition by objective, data-based decision making, made possible by a virtually inexhaustible supply of inexpensive information. Those who control and manipulate this data will be the masters of the new economic universe. Ayres calls them "Super Crunchers," which is also the title of his book, the latest attempt to siphon off a bit of the buzz that surrounds the hugely successful "Freakonomics." In fields from criminal law (where statistical projections of recidivism are taking discretion away from judges and parole boards) to oenophilia (where a formula involving temperature and rainfall is a better predictor of the quality of a vintage than the palates of the most vaunted experts), "intuitivists" are on the defensive against the Super Crunchers.
Super-crunchable data can be broadly statistical or profoundly personal. Illustrating the former, Ayres chose the title of his book by running two Google ads that appeared in random order when someone searched for phrases like "data mining." The decision was made by the plurality who clicked on the ad for "Super Crunchers" rather than the competing title, "The End of Intuition." This is both a more scientific way of making the choice than over lunch with his editor at a posh venue and a case study of how super-crunching can make the economy more efficient and productive. But the same explosion of computing power gives large companies powerful new tools with which to entice—or, in some cases, to torment—their customers. "It's going to be easier to find the products and services we want," Ayres predicts. "The sellers are doing the work for us." Amazon's computers know what we'll like even before we figure it out for ourselves; Netflix customers, says Ayres, like the movies the service recommends better than the ones they choose on their own. But auto dealers can use the same kinds of data to calculate to a fine point just how far they can push their customers on price and loan rates. When airlines cancel a flight, Ayres writes, they use an algorithm to predict which customers are most vulnerable to being lured away by a competitor and to give them, not the airline's own best customers, priority in rebooking.
And this power necessarily resides in a central computer, not with the agent at the ticket counter. Increasingly, jobs that used to call for independent judgment, especially about other people, are being routinized and dumbed down. Banks no longer care about a loan officer's assessment of whether a borrower is a good risk; everything they need to know is in the numbers. Baseball managers increasingly judge prospects by quantifiable statistics, not their "drive" or "hustle." "We are living in an age when dispersed discretion is on the wane," Ayres writes, even in such intimate settings as the doctor's office. Evidence-based medicine, the use of statistical models to guide diagnoses and treatment, is already changing how doctors practice. "Many physicians have effectively ceded a large chunk of control of treatment choice to Super Crunchers," he writes, and the trend will continue despite understandable resistance from the profession. No one wants to throw away a lifetime of specialized training and experience. Which is why the editors request that if you liked this article, or even more so if you didn't … please keep it to yourself. Just kidding.