Should You Trust Medical Studies?

Last week I shared news from the Pharma front, specifically the latest on questionable business practices and undisclosed alliances that continue to dog the industry’s public image. Pharma, however, isn’t the only part of the health field cast in doubt these days. What would you say to a person who claimed that some 90% of medical studies were false – flawed in their parameters, riddled with basic error, skewed by presumption and bias? (No, it’s not me sayin’….) I’m talking about an insider: a highly esteemed, widely acclaimed, much admired, often cited medical researcher who has to turn down speaking engagements left and right. He’s a professional with a flair for statistics and penchant for credibility. His initial groundbreaking research (meta-research, actually) hit the medical scene years ago, but he’s still going strong and in more public headlines these days as the focus of an in-depth Atlantic Magazine feature.

His name is John Ioannidis, a name perhaps familiar to some who live, breathe or earn a living in the medical field. He’s made his mark as a keen analyst and critic of medical research, but he didn’t set out to rock the establishment boat. Early in his career at Harvard, he studied the intricate and thorny labyrinth of diagnosing and treating rare conditions. With little research to guide them, they navigate with “intuition and rules of thumb.”

As The Atlantic author explains, he discovered that practitioners were led by the same process when diagnosing and treating common medical conditions. As he analyzed the relative lack of hard science and credible studies, he took up the cause of examining research integrity, the foundation of the medical field and clinical treatment. He embarked on a massive analysis of then-current research and discovered a barrage of contradictory findings even in the “best,” peer-reviewed studies.

In 2005, Ioannidis published the results from his initial landmark analysis in PLoS Medicine, an open access online journal.

He presented a mathematical proof illustrating the breakdown of accuracy in various models of research: 80% of non-randomized trials, 20% of randomized trials, and 10% of large randomized trials can be expected to produce false results. In addition to these calculations, Ioannidis suggested that “researchers were frequently manipulating data analyses, chasing career-advancing findings rather than good science, and even using the peer-review process…to suppress opposing views.” (Ouch.)

He quickly followed up with a meta-analysis that left little room for argument or dismissal. To bring his point home his point about faulty research credibility, Ioannidis took apart 49 of the best known, most “highly cited original clinical research studies” in the field. Of the 49, 45 studies professed the targeted “intervention was effective.” Of this group, 14 (more than one-third) had been partially or wholly contradicted by later research. Ioannidis later found that subsequent studies often cited old findings without considering contrary results shown in later research – a pattern that caused refuted or questionable medical connections and recommendations to persist in both the research world and clinical practice.

As The Atlantic article brought Ioannidis’ research to public light, some have responded with frustration, wondering “What now?” It’s a fair reaction to some degree. Ioannidis’ conclusions are more than disillusioning; they’re jolting in their foundational, systemic emphasis. First, there’s the question of where his findings leave the medical field. Five years after his PLoS report, not much has changed on the research or clinical front. His message, which could and should have lit a fire under medical researchers, funding agencies, government oversight groups, and clinical practitioners, has been lauded but largely unheeded. Too many apparently feel too tethered to (or buried within) the existing system. The same brand of studies get funded and published and the media continues to spotlight the same spectacle. Nonetheless, the critical assessment is there, and perhaps some clinicians are the more wisely skeptical for it.

Ioannidis doesn’t suggest that the medical field is imploding before our eyes. He says researchers have been sloppy. He takes them to task for initiating their study with leading questions, with maneuvering data, and contriving results. The field as a whole is fraught with accidental and purposeful error, but it’s not going to hell in a hand basket. As Ioannidis suggests, there are still research practices that offer the best promise of accuracy, namely the large, randomized trials. These take money and time, however, which means there needs to be support, clout behind their undertaking. Few topics and treatments will be able to garner these resources. It’s a bit of a scientific cul-de-sac, if you will.

Ioannidis’ assessments don’t reflect the undoing of the research field, but they do remind us to take things with a grain of salt. His critiques represent a necessary check, a much needed perspective on the nature of science itself – science as it’s observed, examined and interpreted in the real world. As direct and concrete as scientific events and medical occurrences might be, we’re ultimately fumbling observers, experimental seekers even in the best of circumstances, even with the best of intentions. Nonetheless, we’re obliged to bring our highest standards and most open minds to the enterprise. Like the practitioner treating a patient with a rare disease, it pays to listen to the patient, to truly see rather than dismiss his/her experience.

Beyond the medical establishment’s inner workings, there’s the inevitable question the rest of us are asking ourselves: what do we, medical care and information consumers, do with these insights? Along that vein, Ioannidis’ analysis implies a broader scope, more holistic approach to health. In pursuit of a healthy and prevention-minded lifestyle, Ioannidis suggests we can give up the magic bullet mentality that has us chasing the latest “miracle” drug or nutrient to begin with. A single health measure in isolation, as Ioannidis suggests, isn’t a panacea. Popping a single nutrient, as those old, now refuted vitamin E studies proposed, isn’t going to make any kind of significant difference. (A rational person somehow always knew this on some level.) We need to understand, to acknowledge the nature of the beast – our own physiological (and dare I say, Primal) functioning. There’s truth in that underlying logic after all.

Once we give up the quest for the cure-all, we’re ready to live in the here and now, which is where real change happens after all. We contribute to our own health care and the relationships we have with our doctors. We can approach and question the research put before us when it directs our course of treatment or informs the choices we make. We can accept the complex responsibility that contributes to our overall wellness. Exchange the magic bullet metaphor for the mosaic. A healthy life is a compilation, an intermingling design formed by a thousand small acts, decisions, and questions. Each healthy thing we do, each life-giving choice we make adds animation and dimension to our overall wellness picture. There’s a curative integrity that we can cultivate for ourselves.

I hope today’s post has offered some food for thought. Let me know what you think, and have a great weekend.

About the Author

Mark Sisson is the founder of Mark’s Daily Apple, godfather to the Primal food and lifestyle movement, and the New York Times bestselling author of The Keto Reset Diet. His latest book is Keto for Life, where he discusses how he combines the keto diet with a Primal lifestyle for optimal health and longevity. Mark is the author of numerous other books as well, including The Primal Blueprint, which was credited with turbocharging the growth of the primal/paleo movement back in 2009. After spending three decades researching and educating folks on why food is the key component to achieving and maintaining optimal wellness, Mark launched Primal Kitchen, a real-food company that creates Primal/paleo, keto, and Whole30-friendly kitchen staples.

If you'd like to add an avatar to all of your comments click here!

50 thoughts on “Should You Trust Medical Studies?”

Leave a Reply

Your email address will not be published. Required fields are marked *

  1. “Exchange the magic bullet metaphor for the mosaic”

    Great line from a great article. Thanks for posting this.

  2. “Each healthy thing we do, each life-giving choice we make…”

    The problem with this statement is that vegan perspectives will tell you what their paradigm is for “healthy”; CW and FDA tells you what their paradigm is for healthy; and now the paleo/primal community tells us what is “healthy.”

    As a consumer of health and fitness paradigms, this is where I claw at what is correct. Obviously I follow historical evidence and anthropology on what to eat, but every once and a while I ask myself, “I’ve been eating only meat for 7 days, some say this is ok to do while others say its toxic.”

    1. I think that’s where the Primal logic comes in, as Mark alludes to. Understanding the nature of the beast — ourselves as we evolved. In some way evolution was the ultimate clinical trial.

  3. I read this last week and left a comment about how much common sense is in the article. And like you say, a rational person knows on some level, there are no magic bullets. I really enjoy your take on this and appreciate how you tackle potentially explosive topics (like Big Pharma) with such finesse!

  4. Nice post Mark, thank you for bringing that to my attention. The article in the Atlantic is fascinating and should be required reading for everyone.

  5. There is an old saying, which I believe is very true: what body craves, is good for the body.. I am not talking about craving twizzlers, or reeses pieces, but if you *want* to eat meat, go ahead eat it, if you *want* to eat yogurt eat it, eat chocolate when you want energy, and heck eat that rice or bread if you really crave it.. Listening to the body is the most important thing. We ain’t smarter than it is, and what we can all do is to achieve the balance (regulating our blood sugar etc) and live like a normal human being.

    1. Does that mean that alcoholics should have a drink if they really crave it? Some of us are carb addicts. Eating that piece of bread may set off our craving that only intensifies and we have a hard time getting back off of it.

  6. This phenomenon also plays a key role in why CW doesn’t sway easily from its current path…

    Guess the best thing we can do is to know from experience and experimentation what works for us as individuals. (insofar as that’s realistic and seems safe.)

    For me I’m pretty confident from my experiences so far that the primal blueprint approach is it, even though I’m nowhere near 100% yet.

  7. i read this last week and immediately thought about mark, robb wolf, taubes, eades, cordain and all the others who have been beating this drum for years. the mainstream is finally catching up with this, hopefully. i posted it on my fb after i read it but most ppl reacted with a yawn, unfortunately…

    1. I did the same exact thing as you. I post links on facebook every morning. Out of 500 people I get about 2 responses every other week. People need to wake up.

  8. With regards to the problem of funding large randomized studies, perhaps it’s time for researchers to think outside the box and try The basic info is here.

    You can start campaigns to raise money, but for the people that pledge, no credit cards are charged until the goal is met, so no money is thrown away.

  9. As a member of the medical community I just wanted to say that I agree with “buyer beware” when it comes to any type of research, not just in the medical field. It is never wise to implement a new practice based on one study unless that study just happens to confirm what you have already been seeing in practice. Medicine is an art as much as a science and sometimes even more so. We are each unique, created as such, and should be treated that way with respect.

  10. I’ve worked on and off for one of the largest medical device companies. I’m back working for them in their post approval study group because the FDA now wants studies showing the benefits throughout the lifetime of the product or the person, whichever ends first.

    Their studies are no better than pharma’s because the whole thing is money driven. Show a good result, get the FDA’s approval, make money before the problems show up, then pull the item off the market. There are no consequences for putting out a bad product in the first place.

    I do have to pick one point – meta analysis. When the original studies are flawed, the meta analysis is just as flawed, if not moreso. Just for fun I’ve pulled apart the original study and the meta analysis and it’s been a train wreck.

    Buyer beware. Do your homework, then trust your gut. If it doesn’t feel right, don’t do it.

  11. “Once we give up the quest for the cure-all, we’re ready to live in the here and now, which is where real change happens after all.”

    Absolutely brilliant!

  12. Mark – thanks for sharing this article with us.

    I have heard that some of the ‘successful’ studies are stopped when the results begin to go downhill. Thus, they can say in all honesty that this drug is effective for x days. There will be a cautionary statement telling you not to take it if symptoms persist longer than x days.

  13. I recently had the opportunity to talk to an oncologist and asked him about the study showing that cancer thrives on sugar if you could theoretically “starve” cancer. He said that for every study I show him, he can show me one that negates it – in essence, people can “prove” anything they put their mind to in a research study. How are Drs supposed to treat patients in that kind of environment? I then told him that I eat a primal diet and explained what that was. He said that eating a higher protein diet is just a healthy way to live based on what it does for metabolism, etc. So if Drs know that, why don’t they prescribe that for their patients, over “eating whole grains, etc.” I don’t trust anything I read or even Drs anymore. I know that eating and living a primal life makes me feel and look good, and my lab tests back that up. That’s good enough for me.

    1. “So if Drs know that, why don’t they prescribe that for their patients, over “eating whole grains, etc.” I don’t trust anything I read or even Drs anymore.”

      That is exactly how I feel! It’s actually scary if you think about it. All doctors were taught from skewed medical research and texts, therefore the knowledge they gained from it is skewed. Only the doctors that don’t blindly follow money or CW “knowledge” are the doctors that I trust (which are few and far between)!

      1. I’ve carried a bacterial sinus infection around with me for 6 years!!! My face was swollen (1 side), I had chronic ear pain from it and a foul breath, yet nobody gave me anti-biotics.
        I had to go to work, feed my family and do all the other life chores WITH that pain…just imagine trying to enjoy life for just a minute?!
        After 6 years I went to another doctor, being fed up with my foul breath and pain, and he prescribed me a STEROID.
        I was outraged.
        I never took it, I finally got smart, went to a specialist for allergies and got anti-biotics and allergy shots. Around the same time changed to primal, stopped drinking pasteurized milk and haven’t had anything wrong with me yet. Not even a cold or the sniffles, nada.
        I’d say 90% of all medical professionals are brainwashed and have been turned into ‘robots’.

        I’ve lost a lot of respect for anyone working in the medical field…most don’t even want to see the light.
        We live next to a huge hospital. I have not seen a single slim, normal weight nurse come out of those doors, yet. They’re all obese.

  14. There’s a philosophical method of thinking, and then there’s an overzealous philosophist who doubts everything he hears. Everything. To him there are no truths, and there are no answers or guidelines he can express to dislike or like, so he’s just beating around the bush, waiting for someone smarter than him to come up with something and then doubting even that answer.

    I think we make our lives complex enough by doubting things that are researched, but I also know researchers in person, and these researchers have a passion of finding out what they want to at the time they want. Meaning basically if a group of people want to know how a body reacts to a low carb diet with different macros and different levels of activity, they conduct a research that suits their needs to find that answer. Whether they present it or not is up to them, but I, for one, am infinitely grateful for researchers who publish their theories and findings on the human body. It helps me focus on what I want to accomplish and do rather than drowning in an ocean of disbelief, doubt and cynicism which I term false wisdom. It’s an expression of wisdom that has the general public thinking a person is smarter than he actually is: It looks smart to doubt everything.

    Just some food for thought.

  15. I have a friend that worked in a HIV research lab until he couldn’t deal with the moral dilemma. The professor that ran the lab owned the patents to all the stuff they were researching and stands to make billions (with a B) if any turn into a product that gets wide distribution. He saw results falsified to make the drugs seem more effective than they were. As the guy that was actually running the tests, he could see that the drug was useless. The problem was that the way the medical journals work and with this guy in control of the most important journal on the subject, there was really no way to get word out that the results were falsified. Attempting to whistleblow would mean he’d never work in a research lab ever again and would not likely succeed anyway because it is almost impossible to go up against some one with an established reputation, research grant money, backing of an institution, etc.

    I believe it is yet another example of why having healthcare as a for profit industry is a bad idea.

  16. Mark,

    I think you substantially understate the error rate that the study found.

    49 results, 14 failures on retest? That’s only a 28% error rate
    49 results 34 retests, 14 failures of those 34. That’s about a 40% error rate in the most prestigious studies found. Who knows about the rest.

  17. I’m a nutrition major and I feel that to actually contribute (and not to mention stand out) in my field I am going to have to know how to know a good study from a bad one and learn how to view journal articles critically.

    Can anyone point me towards some professional resources to help me critically view studies? Strangely, that isn’t in my curriculum on my road to becoming a health professional.

    1. David, Try looking at an accredited school of library science. Most will have one or more courses that specifically deal with how to determine if a study is good or bad. Even if you can’t attend the courses, you should be able to see the recommended texts.

    2. I would recommend a research statistics class. Understanding what factors roll together to make results statistically significant, including random samples, sample size, correlation vs cause, etc., helps in weeding out studies that are poorly designed and executed.

      The bigger problem is when data is intentionally excluded from results — as in the 7 Countries study by Ansel Keys. The researchers actually gathered data from more than 7 nations, but half the data didn’t support the hypothesis (that saturated fats are unhealthy), and so was excluded from the analysis. This major bit of research bias was not discovered for a decade after Key’s study was adopted as the holy grail of the grain based diet. That was 30 years ago and we’re still reeling.

    3. Hey David,

      Reading through the crap to find decent Research is a pain in the a$$ these days. Having graduated from school recently, we had a few classes where all we did was review literature. Most of it is garbage, which makes me sad. Where you’ll find that out is if you just read the methods, materials and results. No reading the abstract, intro, discussion or conclusion. Read the methods and results and come to your own conclusions, then read what they have to say. It’s unbelievable how different they can make the conclusions sound compared to the real results.

      Checking out the statistical analysis is also a good way to find people trying to find “significant” differences in things. The stats stuff can get dicey and difficult to understand though.

      Then there’s common sense. If the research is telling you something you don’t think sounds right, try reading into it. Some people are ninja’s about it. I think it takes practice and a curious mind.

      this site is good, Robb Wolf, PaleoNu, Hyperlipid, Whole Health Source are all good guys to check out.

      Keep fighting the good fight!

  18. “Medicine being a compendium of the successive and contradictory mistakes of medical practitioners, when we summon the wisest of them to our aid, the chances are that we may be relying on a scientific truth the error of which will be recognized in a few years’ time.” – Marcel Proust

  19. The take home message shouldn’t be to trust no one, but to question everyone. Be a skeptic, read, ask questions, dig deep, and think about the big picture, not magic bullets (even if they come in fish oil capsules).
    The most confusing contradiction is that that skepticism is supposed to be the fuel of the scientific fire, not the result of it.

  20. Sounds like Ioannidis contradicts himself in his approach: He uses 49 of the best know studies (which is not a random sample) and then criticizes the studies for not using large randomization trials. Why doesn’t Ioannidis select a large number of studies randomly? How can he ensure the studies were chosen without a bias towards his hypothesis? (whether that be intentionally or unintentionally)

    1. Disagree. I think he chose some of the best known studies to show that even well-known studies under much greater scrutiny are not immune to error. I think it would be much less interesting if he had chosen obscure studies by some random grad student…

  21. “Should You Trust Medical Studies?”

    No, not me. I’m not a health professional, but I have read literally thousands of articles in peer-reviewed medical journals. and I have seen so much shoddy and self-serving nonsense that I don’t trust any of them.

    And I don’t trust the doctors who rely on those articles, instead of listening to their own patients.

  22. Lately I have been having trouble deciding if we are on the cusp of great scientific discovery or if we are stuck in a scientific Dark Age. If Ioannidis’s analysis is correct, then we’ve been in a Dark Age for some time now: a mountain of negative knowledge generated from a mountain of human effort. No one can even begin to calculate the human cost of the negative knowledge.

    It’s possible that we conclude one day that the hypothetico-deductive experimental method is not that robust after all (and never forget that there is no experiment that can show that experiment is the best way to gain knowledge of the world; intuition must still be used to choose among competing epistemologies). This is why I stick to first principles and try not to get hung up on the minutia because sometimes the minutia will lead us astray.

  23. Part of the problem is that journalists cherry pick results from studies and over-simplify the outcomes to sensationalize them. It then sounds like a new cure for, or cause of, a disease has resulted from the research, when in actuality, it showed a slight trend that may have resulted from the design of the study and needs further confirmation (this is even frequently stated in the study itself).