A conference on “anomalous cognition” features unusual claims and raises issues on the role of scientific evidence, replicability, and philosophy of science, plus another: when should one stop looking for evidence in support of an elusive effect?

AMIR RAZ

Amir Raz holds the Canada Research Chair in the Cognitive Neuroscience of Attention at McGill University and the SMBD Jewish General Hospital, where he heads the Cognitive Neuroscience Laboratory and the Clinical Neuroscience and Applied Cognition Laboratory, respectively.

---

“What exactly is anomalous cognition?” As a cognitive scientist, I wondered about this question as I was peering over an intriguing invitation to attend an exclusive Meeting of Minds (MoM) conference on this very topic.¹ I had been counting the days before the MoM, until finally in July 2007 about sixty researchers got together at the University of British Columbia in Vancouver, Canada. To avoid media coverage, the organizers targeted a select group of speakers, and attendance was by invitation only. I was surprised when the conference turned out to be a series of presentations, including reports of what are arguably the best accounts in favor of the possibility of things such as parapsychology and psychic influence, also known as psi. The meeting brought together behavioral scientists and experimental psychologists—most of the audience for the talks—a few skeptics, and a group of self-labeled psi researchers, most of the presenters. As a special treat, a handful of renowned panelists—two Nobel laureates and two distinguished professors of psychology—offered pithy summaries of their impressions following the presentations. It did not take long to realize that anomalous cognition is a new euphemism for the time-honored claims of psi, including extrasensory perception (ESP) and telekinesis.

Initially, I was not sure whether I was invited as a scientist, a skeptic, a magician, or as a friend of one of the organizers. Although I am not a parapsychologist, I am genuinely interested in what I refer to as atypical cognition and rarely shy away from investigating areas within my purview, even those considered as fringe by most of my colleagues. For example, I have been studying the brain computations that occur during planes of altered consciousness, including the cognitive neuroscience of phenomena such as sleep-deprivation, hypnosis, and meditation. At the same time, I consider myself a skeptic—of the deferentially inquisitive rather than gravely unyielding variety—who thrives on converging independent replications of rigorous empirical evidence, not on doctrinaire viewpoints. Finally, it was nice to see among the MoM guests a few fellow conjurors who are, foremost, scientists. Their presence was reassuring, if only to avoid thinking about my answer to the phrase “Are you the best magician among scientists or the best scientist among magicians?” which I have heard one too many times. In that crowd, I was neither.

Having spoken to one of the organizers a few weeks before the meeting, it was my understanding that the conference’s leadership envisaged it as an opportunity to present some of the most compelling data sets in support of anomalous cognition and to urge “mainstream” scientists to foster sufficient open-mindedness to consider a more programmatic investigation into these fields based on these findings. That approach seemed fair and appropriate. Although it was unclear to me at that time what exactly anomalous cognition is, I thought then—as I do now—that it is certainly legitimate to advocate for the possibility of anomalous cognition, including psi. The agenda at the meeting, however, went beyond asking that “mainstream” scientists consider the possibility of psi: it intimated that scientific evidence for psi was solid and replicable. Furthermore, it went on to propose that a major goal of the MoM was to consider why scientific and lay communities do not appreciate the existence of psi.

Interestingly, a number of presenters who argued for the possibility of psi were mainstream researchers, at least in the sense that they had trained and worked in some of the world’s most prestigious institutions of higher learning. While several speakers judiciously implied the possibility of psi, a few explicitly claimed that, based on rigorous data, several anomalous phenomena were veridical. It is perilous, however, to overlook the tenuous boundary between suggesting the possibility of certain phenomena and insinuating—not to mention explicitly submitting—that such anomalies actually exist. During the MoM several speakers blurred this boundary, some in letter and some in spirit, and a few unflinchingly crossed it.

As the conference unfolded, serious issues began to surface concerning the role of scientific evidence, replicability of findings, and philosophy of science. In addition, another question gradually emerged, one that scientists seldom ponder: when is it rational to end the pursuit of a hard-to-pin-down goal? In other words, when should one stop looking for evidence in support of an elusive effect?

As a matter of good practice, members of the scientific community tend to be skeptical. Science thrives on a skeptical approach, and scientists are typically conservative in what they consider a “generally accepted view.” Two types of errors, however, stand in the way of any gatekeeper of science. One pertains to how nonexistent phenomena may pass as real or generally accepted; the other pertains to how real phenomena, which should be generally accepted, may pass as nonexistent. Scientists typically pay more attention to the former trap, and some consequently tend to be overzealous or dogmatically skeptical; members of this staunch group can be skeptical of their own belly buttons. The second trap, however, is usually less explored. If psi effects are real, then the scientific establishment needs to be careful not to deny a phenomenon that may later become a generally accepted view.

Carl Sagan popularized Marcelo Truzzi’s dictum that extraordinary claims require extraordinary proof. Although Truzzi used the word “evidence” rather than Sagan’s “proof,” the former, too, had paraphrased earlier statements by great skeptics such as David Hume and Pierre-Simon Laplace. Most scientists still uphold the “extraordinary” motto; however, many of them might not realize that later in his life Truzzi recanted his own maxim. While we can speculate why he did, it remains unclear what constitutes an extraordinary claim. Does claiming to possess X-ray vision or that the sun will not shine tomorrow count as extraordinary? Deciding on what constitutes an extraordinary claim is probably related to our working knowledge—the proverbial a priori Bayesian probabilities with which we navigate the world. We typically use the inductive process to decide whether claims are extraordinary. It would be easier to accept X-ray vision, for example, if we suddenly discovered special receptors for that wavelength in the human body. The presence of such receptors is unlikely—if only because they have eluded us heretofore—but not impossible. That the sun will not shine tomorrow is perhaps a more extraordinary claim because our inductive experience, not to mention our knowledge of physics, suggests otherwise. In addition, while it may be difficult to agree on what would lend extraordinary support to a claim, scientists usually agree on what constitutes unimpressive evidence. Thus, for example, experimental results that do not replicate, effects that are very small and tenuous, flaws of design and methodology, insufficient sample size, inadequate statistical analyses, and lack of a theoretical basis may all contribute to weak evidence.

Conducting parapsychology experiments is an unprotected legal act: anyone can do it without a special license. At the conference, a few talks featured nonpsychologists, including physicists, engineers, and other professionals with little or no training in behavioral science, who nonetheless reported data from studies they conducted in experimental psychology. While at least some of these studies were markedly inadequate and contained glaring shortcomings, others consisted of more careful efforts, sometimes with intriguing results. Physicists with little training in behavioral science, however, are probably not the best professionals to conduct complex psychological experiments in the same way that experimental psychologists with little background in theoretical physics are likely suboptimal candidates to carry out empirical research in quantum mechanics. Of course, individuals who combine psychology with relevant interdisciplinary knowledge, including that from the exact, life, social, and engineering sciences, may have relative merits. In this regard, magicians—those performers who are well-versed in the art of human deception and trickery—may have especially good insights to offer. Whereas I have been an active magician and spent considerable time following claims of the paranormal, I am now a professional academic scientist, at least in the sense that a reputable university supports my research and salary. These credentials make me neither omniscient nor an authority on truth. But they do suggest at least some experience with and perhaps proficiency in assessing psi claims.

Science provides an evanescent form of truth. We never get there, but we can judge how close we are. One test that we can perform requires the convergence of evidence over multiple researchers, methods, labs, and periods. We should probably apply the same time-honored, scientific principle to the study of psi. The psi phenomena reported in the conference, however, tended to comprise very small, elusive effects that were difficult to replicate. In the few cases seemingly supported by replication or meta-analysis (a statistical method that can provide a more complete picture than individual small studies can), multiple caveats cast long shadows over the raw data and the inclusion/exclusion criteria of specific studies. Statistical analysis, however rigorous, is independent of the quality of the unprocessed information: it crunches both meaningful and less meaningful data indiscriminately. Thus, independent of the statistical methods, interpretation of the results is inconclusive at best.

It became clear that proponents of the existence of psi, who typically claim that evidence for psi is bona fide and replicable, largely base their claims on the results of several meta-analyses. It is precarious, however, to rely almost exclusively on the outcomes of meta-analyses for support. Meta-analytical studies are retrospective, not prospective, and confound exploratory with confirmatory investigation. In addition, in the known cases where more than one team of investigators have conducted a meta-analysis of the same research domain within psi, the conclusions have been strikingly different (e.g., a psi proponent reported a meta-analysis of Ganzfeld studies with an average effect size that significantly differed from zero with odds of more than a trillion to one while another meta-analysis of the Ganzfeld data concluded that the average effect size was consistent with zero). This lack of robustness is difficult to reconcile.

Scientists, including the better and smarter of them, are fallible beings prone to the entire spectrum of human behaviors and blunders. People, including scientists, often ask unscientific questions: do you believe that hypnosis can reduce pain? Do you suppose that Prozac can help depression? Pristine scientists, however, do not believe or suppose. Instead, they look at the data and ask whether the evidence supports the hypothesis. At least in theory, researchers’ beliefs should be immaterial to the results of their experiments, because science is about empirical evidence. In reality, however, the experimenter’s beliefs may introduce a substantive bias to the interpretation of data and sometimes even to more nuanced aspects. For example, beliefs and attitudes may bias participant recruitment and influence their expectations, affect feedback, and may even subtly permeate data collection and analysis. At the MoM, it quickly became evident that people had strong beliefs. “What kind of data would make you change your mind?” I asked many a colleague. While several associates danced around the answer with grace and elegance, most coy responses amounted to one troublesome sentiment: “none.”

In a short, informal gathering following the main MoM event, a few participants suggested that perhaps psi effects are not amenable to standard scientific scrutiny because the alleged effects, when they do occur, typically disappear soon after the initial experiment, thereby preventing replication. This “decline effect”—the tendency of psi phenomena to wane over time, sometimes reaching chance levels—is most peculiar. Another commonly reported outcome is the “experimenter effect”: a difference in participants’ performance as a function of the individual who is administering the experiment. It may be interesting to further pursue the latter, as it may also elucidate the therapeutic alliance we so desperately seek with our health practitioners. Nonetheless, we should heed Karl Popper, an influential twentieth-century philosopher of science, who taught us that a proposition or theory is scientific if it permits the possibility of being shown false—the falsifiability criterion. The history of science shows that many theories were not initially falsifiable not because they were not sufficiently well-operationalized in terms of measurable variables—as was the case in Freudian theories, for example—but because they were not fully developed. Such theories, however, have often served a valuable purpose. Proponents of psi may feel that they operate in a similar climate: they might not yet be ready for “prime time” but may want to use the controversy surrounding psi to generate interest and perhaps even a large body of research from which new theories and empirical findings can evolve.

Theory is important, and the life of the scientific theoretician is anything but easy because experiments are inexorable evaluators of one’s work. These unfriendly judges—the experiments—never say yes to a theory and in the great majority of cases assert a flat-out no. Even in the most favorable of situations, they suggest only a “perhaps.” Historically, rather than anchor their observations in a theoretical framework, most proponents of psi have focused on a technicality: their pivotal criterion for the presence of psi hinged on obtaining a statistically significant departure from chance. It became gradually evident, however, that in this way it was difficult to specify what properties typified psi and what criteria determined its absence. Nowadays, theories of psi abound, with most loosely brushing against quantum theory and generating no specific, testable, and falsifiable predictions. Such theories, some rather grandiose, appear especially disjointed, as they are not grounded in supporting experimental data.

“A wise man…proportions his belief to the evidence,” wrote Hume in his 1748 essay Of Miracles. Having attended all the talks at the meeting, the collective evidence that I have examined does not support the hypothesis that psi phenomena exist. Neither I nor anyone else, however, can reject this hypothesis and conclude that such phenomena do not exist. For example, based on insufficient evidence we cannot decisively conclude that the Tooth Fairy does not exist. But the burden of “proof” rests with those who make the extraordinary claim. On the one hand, when intriguing nascent evidence presents itself, further investigation should ensue. On the other hand, skeptics will probably continue to maintain that psi is unlikely, and proponents will almost certainly continue to look for new ways to demonstrate their claims.

The air was effervescent as each panelist offered an extemporaneous eight-minute summary. Peppering their comments with humor and panache, the psychologists were largely unimpressed by the evidence and pointed to a number of the abovementioned weaknesses. The Nobel laureates, however—one in physics and one in chemistry—echoed a favorable and more accepting tenor. One mentioned atmospheric science as a metaphor for the science of psi, suggesting that psi phenomena may be difficult to predict and replicate consistently in the same way that weather forecasts are nebulous. The other described his experiences with personal acquaintances whom he considered to be genuine psychics.

These last statements left me rubbing my ears in disbelief. On the one hand, albeit far from perfect, weather forecasts have gotten better over the past few decades and are certainly more reliable than outcome predictions from psi research. On the other hand, befriending individuals who claim psychic abilities is hardly firm grounds for scientific exchange.

Individuals, including intelligent persons, are infamously irrational, and one personal “psi experience” is often more compelling than multiple converging scientific accounts. Social psychologists have coined this phenomenon the “vividness” effect. Being a scientist, a prestidigitator, and a skeptic who is keenly aware of his bellybutton, I’d be curious to see compelling scientific demonstrations of psi (i.e., a string of multiple successful experiments by several independent investigators producing lawful and replicable outcomes). Alas, I have found none to date. But when do you conclude that the effect you are seeking is unlikely? When do you stop looking?

Data in support of psi have so far failed to meet the acceptable scientific standards of lawfulness, replicability, objectivity, falsifiability, and theoretical coherence. A group of dogmatically skeptical individuals seems to consistently reject psi research because of granitic prejudices, but navel-denying skepticism is incongruent with good science. While some scientists may indeed reject psi out of prejudice, they typically do not “discriminate” against psi; they show a similar “prejudice” against any claim that seemingly violates fundamental principles of current scientific theory. A healthy first reaction to any departure from existing frameworks is to look for defects in the supporting evidence. If such defects are not apparent, it is time to insist on obtaining independent replications. Until such evidence is forthcoming, it would be difficult for the scientific community to accept a claim for an anomaly.

Highly biased perceptions of reality may be at odds with the findings of science, and establishing the existence of paranormal phenomena might well comprise an intractable task. If compelling evidence were to materialize, however, scientists should be willing to change their minds. Members of the scientific community should be amenable, at least in some measure, to the possibility of novel phenomena. At the same time, proponents of new claims should provide compelling “proof,” and everyone should be sufficiently critical to dismiss claims that already have been found specious. While some of us may have concluded that the Tooth Fairy seems unlikely, others may keep on looking for her…. Still others may be undecided.

This entry was posted on Saturday, February 14th, 2009 at 8:10 pm and is filed under SKEPTICAL & ANALYTICAL VIEWS. You can follow any responses to this entry through the RSS 2.0 feed. You can skip to the end and leave a response. Pinging is currently not allowed.