Meet Mark

Let me introduce myself. My name is Mark Sisson. I’m 63 years young. I live and work in Malibu, California. In a past life I was a professional marathoner and triathlete. Now my life goal is to help 100 million people get healthy. I started this blog in 2006 to empower people to take full responsibility for their own health and enjoyment of life by investigating, discussing, and critically rethinking everything we’ve assumed to be true about health and wellness...

Tell Me More
Stay Connected
October 29, 2010

Should You Trust Medical Studies?

By Mark Sisson
50 Comments

Last week I shared news from the Pharma front, specifically the latest on questionable business practices and undisclosed alliances that continue to dog the industry’s public image. Pharma, however, isn’t the only part of the health field cast in doubt these days. What would you say to a person who claimed that some 90% of medical studies were false – flawed in their parameters, riddled with basic error, skewed by presumption and bias? (No, it’s not me sayin’….) I’m talking about an insider: a highly esteemed, widely acclaimed, much admired, often cited medical researcher who has to turn down speaking engagements left and right. He’s a professional with a flair for statistics and penchant for credibility. His initial groundbreaking research (meta-research, actually) hit the medical scene years ago, but he’s still going strong and in more public headlines these days as the focus of an in-depth Atlantic Magazine feature.

His name is John Ioannidis, a name perhaps familiar to some who live, breathe or earn a living in the medical field. He’s made his mark as a keen analyst and critic of medical research, but he didn’t set out to rock the establishment boat. Early in his career at Harvard, he studied the intricate and thorny labyrinth of diagnosing and treating rare conditions. With little research to guide them, they navigate with “intuition and rules of thumb.”

As The Atlantic author explains, he discovered that practitioners were led by the same process when diagnosing and treating common medical conditions. As he analyzed the relative lack of hard science and credible studies, he took up the cause of examining research integrity, the foundation of the medical field and clinical treatment. He embarked on a massive analysis of then-current research and discovered a barrage of contradictory findings even in the “best,” peer-reviewed studies.

In 2005, Ioannidis published the results from his initial landmark analysis in PLoS Medicine, an open access online journal.

He presented a mathematical proof illustrating the breakdown of accuracy in various models of research: 80% of non-randomized trials, 20% of randomized trials, and 10% of large randomized trials can be expected to produce false results. In addition to these calculations, Ioannidis suggested that “researchers were frequently manipulating data analyses, chasing career-advancing findings rather than good science, and even using the peer-review process…to suppress opposing views.” (Ouch.)

He quickly followed up with a meta-analysis that left little room for argument or dismissal. To bring his point home his point about faulty research credibility, Ioannidis took apart 49 of the best known, most “highly cited original clinical research studies” in the field. Of the 49, 45 studies professed the targeted “intervention was effective.” Of this group, 14 (more than one-third) had been partially or wholly contradicted by later research. Ioannidis later found that subsequent studies often cited old findings without considering contrary results shown in later research – a pattern that caused refuted or questionable medical connections and recommendations to persist in both the research world and clinical practice.

As The Atlantic article brought Ioannidis’ research to public light, some have responded with frustration, wondering “What now?” It’s a fair reaction to some degree. Ioannidis’ conclusions are more than disillusioning; they’re jolting in their foundational, systemic emphasis. First, there’s the question of where his findings leave the medical field. Five years after his PLoS report, not much has changed on the research or clinical front. His message, which could and should have lit a fire under medical researchers, funding agencies, government oversight groups, and clinical practitioners, has been lauded but largely unheeded. Too many apparently feel too tethered to (or buried within) the existing system. The same brand of studies get funded and published and the media continues to spotlight the same spectacle. Nonetheless, the critical assessment is there, and perhaps some clinicians are the more wisely skeptical for it.

Ioannidis doesn’t suggest that the medical field is imploding before our eyes. He says researchers have been sloppy. He takes them to task for initiating their study with leading questions, with maneuvering data, and contriving results. The field as a whole is fraught with accidental and purposeful error, but it’s not going to hell in a hand basket. As Ioannidis suggests, there are still research practices that offer the best promise of accuracy, namely the large, randomized trials. These take money and time, however, which means there needs to be support, clout behind their undertaking. Few topics and treatments will be able to garner these resources. It’s a bit of a scientific cul-de-sac, if you will.

Ioannidis’ assessments don’t reflect the undoing of the research field, but they do remind us to take things with a grain of salt. His critiques represent a necessary check, a much needed perspective on the nature of science itself – science as it’s observed, examined and interpreted in the real world. As direct and concrete as scientific events and medical occurrences might be, we’re ultimately fumbling observers, experimental seekers even in the best of circumstances, even with the best of intentions. Nonetheless, we’re obliged to bring our highest standards and most open minds to the enterprise. Like the practitioner treating a patient with a rare disease, it pays to listen to the patient, to truly see rather than dismiss his/her experience.

Beyond the medical establishment’s inner workings, there’s the inevitable question the rest of us are asking ourselves: what do we, medical care and information consumers, do with these insights? Along that vein, Ioannidis’ analysis implies a broader scope, more holistic approach to health. In pursuit of a healthy and prevention-minded lifestyle, Ioannidis suggests we can give up the magic bullet mentality that has us chasing the latest “miracle” drug or nutrient to begin with. A single health measure in isolation, as Ioannidis suggests, isn’t a panacea. Popping a single nutrient, as those old, now refuted vitamin E studies proposed, isn’t going to make any kind of significant difference. (A rational person somehow always knew this on some level.) We need to understand, to acknowledge the nature of the beast – our own physiological (and dare I say, Primal) functioning. There’s truth in that underlying logic after all.

Once we give up the quest for the cure-all, we’re ready to live in the here and now, which is where real change happens after all. We contribute to our own health care and the relationships we have with our doctors. We can approach and question the research put before us when it directs our course of treatment or informs the choices we make. We can accept the complex responsibility that contributes to our overall wellness. Exchange the magic bullet metaphor for the mosaic. A healthy life is a compilation, an intermingling design formed by a thousand small acts, decisions, and questions. Each healthy thing we do, each life-giving choice we make adds animation and dimension to our overall wellness picture. There’s a curative integrity that we can cultivate for ourselves.

I hope today’s post has offered some food for thought. Let me know what you think, and have a great weekend.

Subscribe to the Newsletter

If you'd like to add an avatar to all of your comments click here!

Leave a Reply

50 Comments on "Should You Trust Medical Studies?"

avatar

Sort by:   newest | oldest | most voted
Simon Kemp
Simon Kemp
5 years 10 months ago

“Exchange the magic bullet metaphor for the mosaic”

Great line from a great article. Thanks for posting this.

Ahmed Serag
5 years 10 months ago

Really liked that last paragraph, nice job Mark!

Daniel Merk
5 years 10 months ago
“Each healthy thing we do, each life-giving choice we make…” The problem with this statement is that vegan perspectives will tell you what their paradigm is for “healthy”; CW and FDA tells you what their paradigm is for healthy; and now the paleo/primal community tells us what is “healthy.” As a consumer of health and fitness paradigms, this is where I claw at what is correct. Obviously I follow historical evidence and anthropology on what to eat, but every once and a while I ask myself, “I’ve been eating only meat for 7 days, some say this is ok to… Read more »
Jen
Jen
5 years 10 months ago

I think that’s where the Primal logic comes in, as Mark alludes to. Understanding the nature of the beast — ourselves as we evolved. In some way evolution was the ultimate clinical trial.

Patty
5 years 10 months ago

I read this last week and left a comment about how much common sense is in the article. And like you say, a rational person knows on some level, there are no magic bullets. I really enjoy your take on this and appreciate how you tackle potentially explosive topics (like Big Pharma) with such finesse!

JohnC
JohnC
5 years 10 months ago

Nice post Mark, thank you for bringing that to my attention. The article in the Atlantic is fascinating and should be required reading for everyone.

lale
lale
5 years 10 months ago
There is an old saying, which I believe is very true: what body craves, is good for the body.. I am not talking about craving twizzlers, or reeses pieces, but if you *want* to eat meat, go ahead eat it, if you *want* to eat yogurt eat it, eat chocolate when you want energy, and heck eat that rice or bread if you really crave it.. Listening to the body is the most important thing. We ain’t smarter than it is, and what we can all do is to achieve the balance (regulating our blood sugar etc) and live like… Read more »
Maxmilliana
Maxmilliana
5 years 6 months ago

Does that mean that alcoholics should have a drink if they really crave it? Some of us are carb addicts. Eating that piece of bread may set off our craving that only intensifies and we have a hard time getting back off of it.

Jenny
Jenny
5 years 10 months ago

This phenomenon also plays a key role in why CW doesn’t sway easily from its current path…

Guess the best thing we can do is to know from experience and experimentation what works for us as individuals. (insofar as that’s realistic and seems safe.)

For me I’m pretty confident from my experiences so far that the primal blueprint approach is it, even though I’m nowhere near 100% yet.

Derrick
Derrick
5 years 10 months ago

i read this last week and immediately thought about mark, robb wolf, taubes, eades, cordain and all the others who have been beating this drum for years. the mainstream is finally catching up with this, hopefully. i posted it on my fb after i read it but most ppl reacted with a yawn, unfortunately…

Aaron Curl
5 years 10 months ago

I did the same exact thing as you. I post links on facebook every morning. Out of 500 people I get about 2 responses every other week. People need to wake up.

Chris Johnosn
5 years 10 months ago

Same here…no reaction, no interest

Tribe of Five
5 years 10 months ago

Loved the last couple of paragraphs, Mark. How wise you are.

Jeff
Jeff
5 years 10 months ago

With regards to the problem of funding large randomized studies, perhaps it’s time for researchers to think outside the box and try thepoint.com. The basic info is here.

You can start campaigns to raise money, but for the people that pledge, no credit cards are charged until the goal is met, so no money is thrown away.

Elizabeth Hull
5 years 10 months ago

As a member of the medical community I just wanted to say that I agree with “buyer beware” when it comes to any type of research, not just in the medical field. It is never wise to implement a new practice based on one study unless that study just happens to confirm what you have already been seeing in practice. Medicine is an art as much as a science and sometimes even more so. We are each unique, created as such, and should be treated that way with respect.

Kevin Holbrook
5 years 10 months ago

This post is a masterpiece.

Kethry
Kethry
5 years 10 months ago
I’ve worked on and off for one of the largest medical device companies. I’m back working for them in their post approval study group because the FDA now wants studies showing the benefits throughout the lifetime of the product or the person, whichever ends first. Their studies are no better than pharma’s because the whole thing is money driven. Show a good result, get the FDA’s approval, make money before the problems show up, then pull the item off the market. There are no consequences for putting out a bad product in the first place. I do have to pick… Read more »
Derek Weiss
5 years 10 months ago

Everything in moderation, that’s probably the answer. Your mother told you that as a child, she was right.

Rory Taj
Rory Taj
5 years 10 months ago

“Once we give up the quest for the cure-all, we’re ready to live in the here and now, which is where real change happens after all.”

Absolutely brilliant!

trackback
5 years 10 months ago

[…] Should You Trust Medical Studies – Mark’s Daily Apple […]

Meg
Meg
5 years 10 months ago
trackback

[…] article: Should You Trust Medical Studies? […]

hiker
hiker
5 years 10 months ago

Mark – thanks for sharing this article with us.

I have heard that some of the ‘successful’ studies are stopped when the results begin to go downhill. Thus, they can say in all honesty that this drug is effective for x days. There will be a cautionary statement telling you not to take it if symptoms persist longer than x days.

Melinda
5 years 10 months ago
I recently had the opportunity to talk to an oncologist and asked him about the study showing that cancer thrives on sugar if you could theoretically “starve” cancer. He said that for every study I show him, he can show me one that negates it – in essence, people can “prove” anything they put their mind to in a research study. How are Drs supposed to treat patients in that kind of environment? I then told him that I eat a primal diet and explained what that was. He said that eating a higher protein diet is just a healthy… Read more »
Aaron Curl
5 years 10 months ago

“So if Drs know that, why don’t they prescribe that for their patients, over “eating whole grains, etc.” I don’t trust anything I read or even Drs anymore.”

That is exactly how I feel! It’s actually scary if you think about it. All doctors were taught from skewed medical research and texts, therefore the knowledge they gained from it is skewed. Only the doctors that don’t blindly follow money or CW “knowledge” are the doctors that I trust (which are few and far between)!

Arty
Arty
4 years 10 months ago
I’ve carried a bacterial sinus infection around with me for 6 years!!! My face was swollen (1 side), I had chronic ear pain from it and a foul breath, yet nobody gave me anti-biotics. I had to go to work, feed my family and do all the other life chores WITH that pain…just imagine trying to enjoy life for just a minute?! After 6 years I went to another doctor, being fed up with my foul breath and pain, and he prescribed me a STEROID. I was outraged. I never took it, I finally got smart, went to a specialist… Read more »
trackback
5 years 10 months ago

[…] Read more of this here: link […]

Michael
Michael
5 years 10 months ago
There’s a philosophical method of thinking, and then there’s an overzealous philosophist who doubts everything he hears. Everything. To him there are no truths, and there are no answers or guidelines he can express to dislike or like, so he’s just beating around the bush, waiting for someone smarter than him to come up with something and then doubting even that answer. I think we make our lives complex enough by doubting things that are researched, but I also know researchers in person, and these researchers have a passion of finding out what they want to at the time they… Read more »
KestrelSF
KestrelSF
5 years 10 months ago
I have a friend that worked in a HIV research lab until he couldn’t deal with the moral dilemma. The professor that ran the lab owned the patents to all the stuff they were researching and stands to make billions (with a B) if any turn into a product that gets wide distribution. He saw results falsified to make the drugs seem more effective than they were. As the guy that was actually running the tests, he could see that the drug was useless. The problem was that the way the medical journals work and with this guy in control… Read more »
aretae
5 years 10 months ago

Mark,

I think you substantially understate the error rate that the study found.

49 results, 14 failures on retest? That’s only a 28% error rate
No….
49 results 34 retests, 14 failures of those 34. That’s about a 40% error rate in the most prestigious studies found. Who knows about the rest.

David
David
5 years 10 months ago

I’m a nutrition major and I feel that to actually contribute (and not to mention stand out) in my field I am going to have to know how to know a good study from a bad one and learn how to view journal articles critically.

Can anyone point me towards some professional resources to help me critically view studies? Strangely, that isn’t in my curriculum on my road to becoming a health professional.

Martha
Martha
5 years 10 months ago

David, Try looking at an accredited school of library science. Most will have one or more courses that specifically deal with how to determine if a study is good or bad. Even if you can’t attend the courses, you should be able to see the recommended texts.

bokbadok
bokbadok
5 years 10 months ago
I would recommend a research statistics class. Understanding what factors roll together to make results statistically significant, including random samples, sample size, correlation vs cause, etc., helps in weeding out studies that are poorly designed and executed. The bigger problem is when data is intentionally excluded from results — as in the 7 Countries study by Ansel Keys. The researchers actually gathered data from more than 7 nations, but half the data didn’t support the hypothesis (that saturated fats are unhealthy), and so was excluded from the analysis. This major bit of research bias was not discovered for a decade… Read more »
Adam Ball
Adam Ball
5 years 10 months ago
Hey David, Reading through the crap to find decent Research is a pain in the a$$ these days. Having graduated from school recently, we had a few classes where all we did was review literature. Most of it is garbage, which makes me sad. Where you’ll find that out is if you just read the methods, materials and results. No reading the abstract, intro, discussion or conclusion. Read the methods and results and come to your own conclusions, then read what they have to say. It’s unbelievable how different they can make the conclusions sound compared to the real results.… Read more »
DThalman
DThalman
5 years 10 months ago

so well written and exactly in line with what i believe

The Primal Palette
5 years 10 months ago

“Medicine being a compendium of the successive and contradictory mistakes of medical practitioners, when we summon the wisest of them to our aid, the chances are that we may be relying on a scientific truth the error of which will be recognized in a few years’ time.” – Marcel Proust

bokbadok
bokbadok
5 years 10 months ago

He’s got a point. :-\

Mark
Mark
5 years 10 months ago

The take home message shouldn’t be to trust no one, but to question everyone. Be a skeptic, read, ask questions, dig deep, and think about the big picture, not magic bullets (even if they come in fish oil capsules).
The most confusing contradiction is that that skepticism is supposed to be the fuel of the scientific fire, not the result of it.

JD
JD
5 years 10 months ago

Sounds like Ioannidis contradicts himself in his approach: He uses 49 of the best know studies (which is not a random sample) and then criticizes the studies for not using large randomization trials. Why doesn’t Ioannidis select a large number of studies randomly? How can he ensure the studies were chosen without a bias towards his hypothesis? (whether that be intentionally or unintentionally)

Evan
5 years 10 months ago

Disagree. I think he chose some of the best known studies to show that even well-known studies under much greater scrutiny are not immune to error. I think it would be much less interesting if he had chosen obscure studies by some random grad student…

The 100 Best Health Sites
5 years 10 months ago

“Should You Trust Medical Studies?”

No, not me. I’m not a health professional, but I have read literally thousands of articles in peer-reviewed medical journals. and I have seen so much shoddy and self-serving nonsense that I don’t trust any of them.

And I don’t trust the doctors who rely on those articles, instead of listening to their own patients.

trackback

[…] http://www.marksdailyapple.com/should-you-trust-medical-studies/ Posted: November 17th, 2010 Tags: Comments: No Comments. Write a comment Name: […]

mikehell
mikehell
5 years 6 months ago
Lately I have been having trouble deciding if we are on the cusp of great scientific discovery or if we are stuck in a scientific Dark Age. If Ioannidis’s analysis is correct, then we’ve been in a Dark Age for some time now: a mountain of negative knowledge generated from a mountain of human effort. No one can even begin to calculate the human cost of the negative knowledge. It’s possible that we conclude one day that the hypothetico-deductive experimental method is not that robust after all (and never forget that there is no experiment that can show that experiment… Read more »
Maxmilliana
Maxmilliana
5 years 6 months ago

Part of the problem is that journalists cherry pick results from studies and over-simplify the outcomes to sensationalize them. It then sounds like a new cure for, or cause of, a disease has resulted from the research, when in actuality, it showed a slight trend that may have resulted from the design of the study and needs further confirmation (this is even frequently stated in the study itself).

trackback

[…] Should You Trust Medical Studies? – Between bias, cherry-picking, false results, and a research integrity deficit, it’s getting harder to know what to think about the studies you see. […]

trackback

[…] Should You Trust Medical Studies? – Between bias, cherry-picking, false results, and a research integrity deficit, it’s getting harder to know what to think about the studies you see. […]

trackback
4 years 10 months ago

[…] Should You Trust Medical Studies? – Between bias, cherry-picking, false results, and a research integrity deficit, it’s getting harder to know what to think about the studies you see. […]

trackback

[…] Should You Trust Medical Studies? – Between bias, cherry-picking, false results, and a research integrity deficit, it’s getting harder to know what to think about the studies you see. […]

trackback

[…] Small Steps Are the Key to Your Healthy Success in 2012By Debra@Blue Raven Wellness on December 21, 2011 in Diet, Health, Nutrition Share A healthy life is a compilation, an intermingling design formed by a thousand small acts, decisions, and questions. Each healthy thing we do, each life-giving choice we make adds animation and dimension to our overall wellness picture.” — Mark Sisson of Mark’s Daily Apple […]

weight loss hcg
4 years 23 days ago

Wow, this piece of writing is fastidious, my sister is analyzing these things, therefore I am going to tell
her.

wpDiscuz