Doubting Observational Studies
Epidemiology, Medicine, Research Studies, Science
A stopped clock is right twice a day, and even the New York Times occasionally publishes an intelligent article.
In this week’s Sunday Magazine, Gary Taubes offers some much-needed skepticism about the omniscience of the kind of research whose results we continually hear trumpeted in the media.
Mr. Taubes’s reflections could readily be extended to other areas, particularly to climate studies.
Many explanations have been offered to make sense of the here-today-gone-tomorrow nature of medical wisdom — what we are advised with confidence one year is reversed the next — but the simplest one is that it is the natural rhythm of science. An observation leads to a hypothesis. The hypothesis (last year’s advice) is tested, and it fails this year’s test, which is always the most likely outcome in any scientific endeavor. There are, after all, an infinite number of wrong hypotheses for every right one, and so the odds are always against any particular hypothesis being true, no matter how obvious or vitally important it might seem.
… hypotheses begin their transformation into public-health recommendations only after they’ve received the requisite support from a field of research known as epidemiology. This science evolved over the last 250 years to make sense of epidemics — hence the name — and infectious diseases. Since the 1950s, it has been used to identify, or at least to try to identify, the causes of the common chronic diseases that befall us, particularly heart disease and cancer. In the process, the perception of what epidemiologic research can legitimately accomplish — by the public, the press and perhaps by many epidemiologists themselves — may have run far ahead of the reality. …
The goal of the endeavor is to tell those of us who are otherwise in fine health how to remain healthy longer. But this advice comes with the expectation that any prescription given — whether diet or drug or a change in lifestyle — will indeed prevent disease rather than be the agent of our disability or untimely death. With that presumption, how unambiguous does the evidence have to be before any advice is offered?
The catch with observational studies…, no matter how well designed and how many tens of thousands of subjects they might include, is that they have a fundamental limitation. They can distinguish associations between two events — that women who take H.R.T. have less heart disease, for instance, than women who don’t. But they cannot inherently determine causation — the conclusion that one event causes the other; that H.R.T. protects against heart disease. As a result, observational studies can only provide what researchers call hypothesis-generating evidence — what a defense attorney would call circumstantial evidence.