 |
Cherry picking: pointing to individual cases or data that seem to confirm a particular position, while ignoring a significant portion of related cases or data that may contradict that position |
In part I of this post, I discussed the issue of inductive reasoning, and how it can be a source of quite valid scientific evidence. This brings up the question of what is a valid conclusion based on inductive reason versus an invalid conclusion based on one person’s personal experience.
The following comes from my book, How Dysfunctional Families Spur Mental Disorders.
Anecdotal evidence in medicine is often misleadingly defined as evidence based on only one clinician’s personal experience with a treatment or diagnosis in question. If that is the standard that is to be used, clearly many reasons exist to question the validity of inferences drawn from these experiences. Individuals are well known to have various biases that color their observations and the conclusions they draw from them.
They may have blind spots because of their own emotional conflicts. They may ignore evidence that is contradictory to their point of view. Their observations may be limited by their pet theories about the phenomena in question.
Perhaps even more importantly, they may be seeing unusual cases that are not representative of more “typical” cases in one way or another – a so-called selection bias.
Perhaps even more importantly, they may be seeing unusual cases that are not representative of more “typical” cases in one way or another – a so-called selection bias.
An obvious case of selection bias was illustrated by a statement I heard made at a conference by a family therapy pioneer, the late Jay Haley. I had always admired Mr. Haley for many of his fascinating and utilitarian ideas and observations. However, in this case he betrayed some ignorance. He stated that he did not believe antidepressant drugs were ever effective because none of the patients referred to him had ever responded to them.
Of course, his being a well-known family therapist who did not believe in medication had a tremendous effect on exactly who would be referred to him. Not everyone does respond to drug treatment. Anyone who had responded to an antidepressant would, in all probability, rarely if ever darken his door. Hence, with his sample, he would be misled into thinking that the medicines were not effective for anybody. This form of bias is very common and can be quite subtle. For example, it can affect one’s beliefs about such matters as racial stereotypes or a determination of how trustworthy members of a city’s police department are.
Do these types of biases invalidate all clinical experience? Hardly. First of all, we have to distinguish between the descriptions of the actual events contained within specific anecdotes, and the conclusions or inferences which are drawn from these events.
Let us first examine the descriptionsof what actually happened. A specific anecdote may be accurately observed and described, or not so accurately. If important details are altered or left out entirely, the anecdote may indeed be worthless. However, the exact same thing can be said about empirical studies.
Let us first examine the descriptionsof what actually happened. A specific anecdote may be accurately observed and described, or not so accurately. If important details are altered or left out entirely, the anecdote may indeed be worthless. However, the exact same thing can be said about empirical studies.
Important details may not even be known to an observer. Particularly with observations of family behavior within a practitioner’s or researcher’s office, important information is almost always hidden, either by design or unwittingly. In addition to the fact that one does not see the whole picture in any single context, there is also a basic problem inherent in the nature of interactions between intimates.
With verbal behavior, for instance, linguists refer to a quality called ellipsis. What this means is that in conversations among people who have known each other for a while, certain information is not spelled out verbally because the other person already knows it. Strangers such as therapists who are listening in and who have not been privy to these prior experiences may think they know what the family is talking about, but they may in reality be completely clueless.
Let us now consider the separate issue of conclusions that are drawn from anecdotes, as opposed to their description. The questions raised by an accurately-described clinical observation can be quite valid, but the answers inferred from it can be completely wrong. Conclusions based on clinical “anecdotes” exist on a continuum from relatively accurate ones to those that are extremely biased to those that are based on spectacular inferential or logical leaps of faith.
Relatively unbiased clinical conclusions based on anecdotes by mental health professionals have many things in common:
· They are based on a sample that one has a reasonable expectation is at least somewhat representative of a larger population.
· They make use, not just the practitioner’s observations, but of the observations of other professionals whom one knows to be reliable and open minded. These clinicians should also be ones known to take the time with their patients necessary to take a complete history. Widespread clinical experience by competent clinicians is something upon which someone can make a very valid inductive conclusion, and is not just “anecdotal evidence.” In fact, conclusions drawn from this source tend to be more accurate that those drawn from so called “empirical” studies. Many dangerous side effects from drugs that did not show up in the initial drug studies have been discovered in this manner.
· They make use of other informants besides the patient when possible.
· They take into consideration that people and their family members behave quite differently behind closed doors than they do in public, and therefore if at all possible include observations of patient behavior when patients are unaware that they are being observed.
· They are based on longitudinal observations. That is, the patients on whom conclusions are based have been seen on multiple occasions over an extended period of time.
· They are not contradicted by commonly observed examples of behavior in everyday life related to the behavior in question.
· The person proposing the conclusion acknowledges potential biases, such as a financial stake in a certain drug or allegiance to a specific school of therapy, and acknowledges his or her limitations. What former president of the Society of Clinical Psychology, Gerald C. Davison, calls “ex cathedra statements based upon flimsy and subjective evidence,” a hallmark of some psychotherapy gurus, are always highly suspicious. In fact, charlatans are relatively easy to spot. They have a tendency to sidestep challenges. I will give an example of this shortly.
· The conclusions reached should lead to predictions of patient behavior under certain circumstances that prove to be accurate in a significant number of cases. This is called predictive validity. Of course, human behavior being as unpredictable as it is, at times the predictions will not be completely accurate even if the conclusions are valid, and so this fact must also be taken into account.
· Conclusions based on anecdotes about treatment efficacy or the reasons for certain observed behavior should consider several alternate possible explanations for the observations. If several explanations are possible, one must make a judgment about which ones are more likely and which are less likely based not on the anecdote alone, but on all sources of data available. These sources include empirical studies, but also include observations from everyday life, as well as material seen in some relatively reliable media such as reputable newspapers.
Now of course stories in the media also do not tell the whole story or may be biased, so one needs to realize again that one can be fooled, and take this into account as well. I used to believe the common myth, for example, that in nature under certain conditions the animals called lemmings would follow each other off a cliff and commit mass suicide.
I was surprised when I learned that this was untrue because I had as a child in 1958 seen a film clip of said mass suicide that was part of a Disney “True Life Adventure” nature movie called White Wilderness. I later learned that, because the Disney crew could not find a real example, they had from behind the scenes driven the group of lemmings off the cliff for the cameras.
On the other hand, many people believe that men have never been to the moon and that films of the moon landings were made in a movie studio using special effects. I must say, I tend to believe that those film clips are real, but few know for certain.
· If other anecdotes about similar patients and treatments seem to contradict the conclusions based on a given anecdote, an attempt should be made to account for this difference.
As an illustration of the latter point and an example of a the “quick step side step” in scientific presentations. I once heard an expert present new evidence from neuroscience that certain capabilities of which human brains are capable seem to develop only at certain times during early childhood development. This brain development could be adversely affected by a baby’s early social environment. Of course, that is somewhat true.
Like psychoanalysts will, however, the expert went on to conclude that if the adverse early experiences had taken place, the child had no chance of growing up to be normal. I raised my hand and asked about those children who come from horribly adverse backgrounds, are adopted away at an age past the alleged crucial developmental time, and yet still turn out wonderfully. The expert then changed the subject without ever addressing my question.
If the data doesn't fit your pre-conceived conclusions, just change the subject!
Remember, there are NO empirical, placebo-controlled, double blind controlled studies on whether parachutes reduce the incidence of deaths or injuries after falling out of airplanes. Or that appendectomies are effective in preventing complications and deaths from appendicitis. And yet we all take those things for granted.
If the data doesn't fit your pre-conceived conclusions, just change the subject!
Remember, there are NO empirical, placebo-controlled, double blind controlled studies on whether parachutes reduce the incidence of deaths or injuries after falling out of airplanes. Or that appendectomies are effective in preventing complications and deaths from appendicitis. And yet we all take those things for granted.
