The Problem With the Rorschach: It Doesn't Work

There is nothing ambiguous about the image in my mind. It clearly depicts two medieval wizards, with tall red hats and black cloaks. They are sitting facing one another other. They appear to be giving each other the high-five.

That's my interpretation of inkblot No. 2 of the Rorschach test, a psychological test used by clinical psychologists and other therapists to assess personality and diagnose psychopathology. I don't know if my interpretation is normal or aberrant, but I do know that most people see two human beings of some kind in inkblot No. 2. I know this because Wikipedia recently published all 10 of the inkblots that Swiss psychiatrist Hermann Rorschach first introduced in his book Psychodiagnostik back in 1921—along with the most common "answers" for each of the inkblots. Therapists use these common answers—or norms—to help them diagnose abnormal behavior and thinking.

Wikipedia's move has sparked a firestorm among psychotherapists who claim that publishing the norms could skew the test's results—or worse, allow patients to fool their therapists, to game the system. Free-speech advocates—including many other therapists—dismiss those claims as nonsense.

This shouting match escalated this week when The New York Times published a long article about the Wikipedia-Rorschach brouhaha. But this heated debate has failed to raise (or answer) the most important question of all: does the Rorschach work? The answer is no, and here is the best evidence:

The journal Psychological Science in the Public Interest published an exhaustive review of all data on the Rorschach (and other similar "projective" tests) in 2000. Such meta-analyses are major undertakings, so although this PSPI report is a few years old, it remains the most definitive word on the Rorschach. The authors—psychologists Scott Lilienfeld, James Wood and Howard Garb—find the Rorschach wanting in two crucial ways.

First, the test lacks what testing experts call "scoring reliability." Scoring reliability means than you get the same results no matter who is scoring the test. Psychotherapists look at more than 100 different variables when scoring an answer: Did the patient focus on stray splotches rather than the main blot, or the white spaces instead of the ink? Did the patient interpret the color? That kind of thing. The PSPI review found that therapists disagree on fully half of these variables, making the scores unreliable for diagnosis.

But it gets worse. The authors also looked at all the extant studies on the test's validity. This is testing jargon for: Does it measure what it claims to measure? Does it predict behavior? And again the answer is a clear no. With the exception of schizophrenia and similarly severe thought disorders, the Rorschach fails to spot any common mental illnesses accurately. The list of what it fails to diagnose includes depression, anxiety disorders, psychopathic personality, and violent and criminal tendencies. It also can't detect sexual abuse in children, even though it's used for that purpose. Finally, the test is most misleading for minorities: blacks, Native Americans and Hispanics are all likely to score abnormally on the inkblot test.

Despite this damning evidence, the most recent survey data indicates that four in 10 clinical psychologists still use the Rorschach "always or frequently" with patients. Why would that be? This isn't the first time the Rorschach has come under attack. The test was roundly criticized back in the '50s for lacking standardization and norms. Those problems were presumably corrected in the '70s, with the introduction of an elaborate system of instructions for therapists, and many newly trained therapists incorporated the revised test into their practices. Even so, it is this revised version of the Rorschach that still fails on both reliability and validity, according to the PSPI report.

The same psychological journal will in a few months be publishing another major review of clinical practice, with the goal of weeding out therapies and techniques that have no scientific evidence to back them up. This dust-up over the Rorschach could be just the beginning of a major intellectual housecleaning in a field that's drifted from its scientific roots. Does anyone else see a battlefield in that amorphous inkblot?