Of Voodoo and the Brain

It is a brave, or perhaps foolhardy, scientist who uses the term "voodoo" to describe results in a high-profile field of science, and in retrospect psychologist Hal Pashler of the University of California, San Diego, concedes that the word might have been just a teensy bit incendiary. But when he, Ed Vul of MIT and colleagues scrutinized the use of brain imaging in social neuroscience— a burgeoning new field that seeks the brain basis for feelings and thoughts resulting from social interactions—there was a good reason why dolls with little needles stuck in them came to mind. Voodoo doctors think the needles bring harm to the unfortunate person represented by the doll. Social neuroscientists think the brain activity they discover can predict and explain differences among people in feelings of prejudice, moral judgments, fear of pain, how much social rejection hurts and other fascinating questions. But in about half the studies the voodoo critics looked at, they charge in a paper to be published this year in Perspectives on Psychological Science, the methods and analysis are so poor that neuroscientists should go back, do it right "and correct the scientific record."

The basis for the voodoo charge is a statistical argument so arcane (I'll take a stab at explaining it below), it can obscure a more important point. The public and the press are infatuated with brain images. It is not so much the specific findings they absorb (people don't go around saying, "Uh-oh, my anterior cingulate is really active; I must be feeling social rejection"). It is something deeper. Brain activity is perceived as more real than subjective experiences, feelings or thoughts. When scans show that addicts' brains "light up" when they see drug paraphernalia, people view addiction as more real, more biological, than if addicts just say they get a craving when they see a bong.

Brain images also convey the erroneous message (and here the press is deeply culpable for reporting the studies so uncritically) that thoughts, feelings and reactions are hard-wired and unchangeable. That profoundly shapes how we view ourselves and others—in particular how we think of free will and individual responsibility. And the day is coming when brain images will be used to make predictions about behaviors, attitudes and aptitudes. One day, will relief agencies use scans to see how active a job applicant's brain becomes when she sees images of human suffering? Will dating services offer scans showing how active a potential mate's brain becomes over the thought of breaking up? Will the military demand scans to see how anxious recruits get when thinking about pain? Unfortunately, "the kinds of data in most imaging papers are fundamentally unable to tell you how predictive imaging is," says neuroscientist Russ Poldrack of UCLA. But nonexperts, having been told how tightly correlated brain activity and thoughts or feelings are, don't get it. Last year a court in India sentenced a woman to life in prison because a brain scan supposedly indicated she knew details of a killing that only the murderer could. In the United States, at least two companies are marketing "lie detector" neuroimaging.

The reason the public is snowed gets to the nub of the attack on neuroimaging (stats-phobes can skip to the next paragraph). In a nutshell, many studies claim very high correlations between patterns of brain activity and some thought or feeling. A "1" is a perfect correlation; it means whenever one thing appears (a pattern of brain activity), the second thing (feeling distressed, say) is always present. In many studies, reported correlations reach 0.8 or 0.9 or higher, which is "too good to be true," says Andrew Gelman, professor of statistics and political science at Columbia: the underlying measurements (of personality or patterns of brain activity) are less well correlated with themselves than that, so they cannot be better correlated with one another. While many of the reported relationships probably reflect something real—at this point, it's impossible to tell—the critics argue that "a considerable number" may be "entirely illusory." I asked statisticians about this criticism, and they agreed. "Correlations of 0.9 are unbelievable in any social-science setting," said William Eddy of Carnegie Mellon.

The neuroscientists accused of voodoo aren't taking it lightly, firing back with furious blog posts and online defenses. In some cases, they have not done themselves any favors. One defense, says Gelman, "is so bad, they look like idiots because they're out of their statistical depth." But a rebuttal by Matthew Lieberman of UCLA and colleagues, to be published along with the voodoo paper, makes valid points, including that they are doing a lot more than just correlations. They concede, however, that some reported correlations are inflated.

Once the bruised egos heal, this battle might improve neuroimaging. Scientists know how to avoid voodoo correlations, says Poldrack, "but the solution can be painful": it can make the statistics fade to nothing "unless you have a superstrong effect." Scientists are understandably reluctant to use a tool so rigorous that it erases what they thought were legitimate discoveries, says Poldrack, but journals should demand it. In his lab, the voodoo paper "is getting me to rethink how we report our results. And a lot of labs, including mine, are looking at old data to be sure the conclusions we drew are valid." Who says voodoo doesn't work?

© 2009