Meet Mark

Let me introduce myself. My name is Mark Sisson. I’m 63 years young. I live and work in Malibu, California. In a past life I was a professional marathoner and triathlete. Now my life goal is to help 100 million people get healthy. I started this blog in 2006 to empower people to take full responsibility for their own health and enjoyment of life by investigating, discussing, and critically rethinking everything we’ve assumed to be true about health and wellness...

Tell Me More
Stay Connected
April 06 2016

15 Reasons Not to Trust That Latest Nutritional Study

By Mark Sisson
88 Comments

Reasons Not to Trust That Latest Nutritional Study FinalNutritional studies are often the best we’ve got. Without them, we’d be plucking anecdotes from a swirling vortex of hearsay, old wives’ tales, and prejudices. Some actionable information would definitely emerge, but we wouldn’t have the broader vision and clarity of thinking offered by the scientific method. Most of them are deeply flawed, though. And to know which ones are worth incorporating into your vision of reality and which only obfuscate and further muddy the waters, you have to know what to watch out for.

Today, I’m going to discuss many of the reasons you shouldn’t trust the latest nutritional study without looking past the headlines.

1. Industry distorts the research.

Last year, Marion Nestle looked at 152 industry-funded nutrition studies. Out of 152, 140 had favorable results for the company who funded it. An earlier analysis of milk, soda, and fruit juice nutrition studies found that those sponsored by milk, soda, and juice companies were far more likely to report favorable results than independent studies. The same things happens in cardiovascular disease trials and orthopedics trials.

2. Ego distorts the research.

People become wedded to their theories. Imagine spending 30 years conducting research to support your idea that saturated fat causes heart disease. How hard will you hold on to that hypothesis? How devastating would opposing evidence be to your sense of self-worth? Your research is your identity. It’s what you do. It’s how you respond when chit chatting at cocktail parties. You’re the “saturated fat” guy. Everything’s riding on it being true.

Scientists are used to being the smartest person in their respective rooms. It’s not easy to relinquish that or admit mistakes. Heck, that goes for everyone in the world. Scientists are not immune.

3. Correlation masquerading as causation.

In day-to-day life, correlative events imply causation. You cut someone off, they honk at you. A man holds a door open, you thank him. You flip a light switch, the light turns on. We’re used to causation explaining correlations. So when two variables are presented together in a nutritional study, especially when it seems plausible (meat causes colon cancer) or reaffirms popular advice (saturated fat causes heart disease), we’re likely to assume the relationship is causal. Correlations provoke interesting hypotheses and tests of those hypotheses, but they’re very often spurious. Everything we eat is associated with cancer if we look hard enough. Does that actually tell us anything useful?

4. No control group.

If you want to know the effects of an experimental intervention, you need a group of people that don’t receive the intervention. That’s the control group. Without a control group to compare against the group that received the experimental intervention, a clinical trial doesn’t mean much. You can’t truly know that the experimental variable caused the change without it.

5. Fake controls.

The presence of a control group doesn’t make it a good study. The control group has to be a real control. Take this paper from last year claiming that oatmeal for breakfast promotes satiety. Sure, when you’re comparing oatmeal to a cornflake breakfast. It doesn’t take much to beat the satiating (non)effects of cornflakes. How would oatmeal compare to bacon and eggs, or a big ass salad, or sweet potato hash? This study doesn’t tell you that.

6. Small sample size.

The smaller the sample size, the less impressive the results. The larger the sample size, the more meaningful the results and the more likely they are to apply to the larger population. That’s why the results of n=1 self-experiments are mostly useful for the person running the experiment on themselves and less useful for others; the sample size of one isn’t enough to generalize the results.

Small sample size studies shouldn’t be ignored. They can lead to interesting questions and hypotheses that larger studies can tackle. But they shouldn’t sway public policy, scientific consensus, or your decision about what to eat and how to live.

7. Demographics.

Make sure you know who participated in the study. If you’re a Latino male of 20 years, the study on low-carb diets in post-menopausal black women may not apply to you.

8. Food frequency questionnaires (FFQ).

FFQs require people to recall their typical diet over the last year. That’s hard. Here’s a sample FFQ (PDF); see how you do trying to recall the foods you ate over the last 12 months. Suffice it to say, they aren’t very reliable. People lie. People forget. People tell you what they think you want to hear, downplaying the unhealthy stuff and overstating the healthy stuff. FFQs are probably the best option available for assessing, but they aren’t good enough.

9. The adherer effect.

Michael Eades calls it the “adherer effect.” I’ve called it the “healthy user effect.” Whatever phrase you prefer, this describes the fact that there’s “something intrinsic to people who religiously take their medicine that makes them live longer,” even if that medicine is a completely inert placebo. Perhaps they’re also more likely to heed other medical advice, like exercising regularly, getting checkups, eating healthy foods, and other behaviors that improve health which could explain some of the beneficial effects. But it’s a real thing, and it has a real effect on the results of nutritional studies.

10. Statistical significance versus clinical significance.

You see the phrase “significantly associated with” a lot in scientific papers. “Fat is significantly associated with type 2 diabetes” sounds like “dietary fat has a large effect on your risk of type 2 diabetes.” But what that phrase really means is “The association between fat and type 2 diabetes is unlikely to be a coincidence.” It says nothing about the size of the association. It doesn’t mean eating fat doubles your chance of getting type 2 diabetes. The clinical significance—the biological effect—is very likely trivial.

11. Relative risk versus absolute risk.

Papers will often talk about “the risk” of something. More often than not, that’s a relative risk. Take something like colon cancer. Though it’s the third most common cancer (and cause of cancer-related deaths), the absolute risk of developing colorectal cancer, even in old age when the risk is at its highest, isn’t exactly high. For the average 50 year old, his or her lifetime absolute risk of colorectal cancer is 1.8%. If that 50 year old has a relative with colon cancer, the absolute risk is 3.4%. Having two relatives with a history of colon cancer pushes it up to 6.9%. On the big scale of things that can kill you, colorectal cancer isn’t even in the top five.

So anything that increases the risk of colon cancer starts from that otherwise meager degree of absolute risk.

12. Nutrients versus foods.

Most nutrition studies attempt to measure the effect of specific nutrients on health outcomes. But people don’t eat palmitic acid. They eat dairy and meat. People don’t eat linoleic acid. They eat almonds, or soybean oil, or pumpkin seeds. People don’t eat glucose, fructose, resistant starch, and prebiotic fiber; they don’t even eat “carbs.” They eat cold potatoes, sweet potatoes, blueberries, wild rice. Studies that look at specific nutrients can’t tell you accurate information about the effects of foods, because foods contain far more than just single nutrients.

13. Most research is wrong.

In 2005, John Ioannidis published a paper called “Why most published research findings are false,” citing conflicts of interest, small sample sizes, insignificant clinical effects, and failures to replicate—in other words, most of the stuff mentioned in today’s post. It eventually became the most widely cited paper ever published in PLoS Medicine, and it’s still true today. Keep it in mind.

14. Journals prefer to publish and researchers prefer to submit exciting studies with strong results.

You’re more likely to have your paper published if it presents a new, exciting finding with a strong result. If two researchers run similar studies and only one gets a positive result, a journal will usually publish the “successful” study and ignore the other one. For their part, researchers are more likely to submit “successful” papers to journals. The end result is a lack of negative results, even though they’re informative and vital for accurate science to prevail.

15. We know very little.

“Blueberries improve memory.” In who? People with dementia, people who are at high risk for it? Can kids improve school performance by eating blueberries? What about college students? What if the college students are female—does that change anything?

“Nuts reduce mortality risk by 40%.” How long do you have to eat the nuts? Does the type of nut matter? Does your age affect the protective effects of nuts?

There’s too much we don’t know. There are too many variables we can’t control.

This isn’t to suggest that nutritional studies are useless. I cite and refer to them all the time. They’re often the best, most objective angle on the situation available. Like democracy, it’s the worst except for all the others. But we have to recognize and consider their limitations. Hopefully after today’s post, you’ll know what to look for.

That’s it for today, folks. Let’s hear from you. What do you think? How do you analyze a nutritional study? What do you look out for?

Prefer listening to reading? Get an audio recording of this blog post, and subscribe to the Primal Blueprint Podcast on iTunes for instant access to all past, present and future episodes here.

If you'd like to add an avatar to all of your comments click here!

88 thoughts on “15 Reasons Not to Trust That Latest Nutritional Study”

Leave a Reply

Your email address will not be published. Required fields are marked *

  1. Informative and thought-provoking post. These sorts of issues with studies are all too often forgotten when examining a wide range of topics that the media presents us with studies about on a daily basis. People are too often easily convinced with scarcely more evidence than a flashy primetime news report or a sassy internet headline.

  2. People lie like a rug, aren’t very bright and life’s complicated. That actually makes the Primal lifestyle that much stronger. No one can make a serious argument about the past.

    1. Actually the primal lifestyle is refutable. For the most part your meats, veggies and fruits are distinctly different than what people were eating 100,000 years ago.

      1. Actually, the meats, veggies, and fruits are distinctly different than what people ate just 10 years ago, I don’t recall much clamber about GMO’s and Grass Fed this and Range grazed that so there has been a lot of change in the recent past regarding what seems to be the “in” thing to talk about regarding our diets and what is “out” as “old wive’s tales” and recently proven “wrong”. I can think of many examples of both of these old time habits that have been refuted by newer theories and then re-refuted by people like Mark who know that the diets of our kin folk were supplemented by hard work, outdoors living, active lifestyles, and mostly water as a thirst quencher. (Oh yeah, and a few brews at the local pub when one went to town to buy fencing or to sell eggs or raw milk.) And we could go on with the obvious absence of chemical additives to prolong freshness and retard spoilage, the use of raw products right out of the garden only minutes before serving them on the dinner table, there were no massive industries that controlled our use of sweeteners, “refined” food products, or medicines.

        Many of the workable “cures” for common ailments, that actually worked back in the day, had been shoved off the retail turnip wagon by chemically created artificial remedies that carried more negative side effects than the good that they may or may not have offered for various sicknesses and mal-adjusted lifestyles of the time. And, in most cases, it has nothing to do with helping to alleviate some ailment as it does with some overpriced chemical concoction that worked on a few lab rats in a “clinical test”! Listen to the average commercial presentation for most any ailments on today’s boob tube, it will be comprised about one third content of smiling giddy young lasses and lads who have found the secret to their bloating, gas problems, headaches, and other maladies, while the rest is comprised of warnings about allergies to the drug ingredients, signs of impending side effects and/or possible death, and so many other negative points of concern that one wonders why anyone would abuse their body by taking such a concoction in the first place!!

        Yeah, I agree, things WERE different back in the history of our race, and in most cases it was much better than we have offered to us today and at a reasonable cost or for free for the foraging.

        1. Agreed, foods were much different. But besides the artificial ingredients, GMOs etc, 10,000 years of artificial selection and cultivation since the agricultural revolution has rendered many of our modern commercial fruits and vegetables nearly unrecognizable from their original form. It is nearly impossible to truly recreate a paleolithic lifestyle eating the plants that are available to us today.

      2. Hobbes – we are aware of that. Try looking around some here. You’ll find that topic discussed periodically.

        1. The inability to re-create the Paleolithic diet is exactly why I have created my own diet (book deal, TBD, if Mark’s on the comments section, maybe he can throw me a bone). Working title: Eat the REAL Paleo Diet: Go Paleozoic. Weight loss is guaranteed, as you’ll spend most of your time searching for trilobites.

  3. Nowadays, the rush to publish (anywhere–in a scientific journal or the NTY, doesn’t matter) is so great, and so many institutions rely on that “prestige” to garner more funding…so they can continue to publish studies that won’t pass scrutiny, rest on their prestige laurels, garner more funding, etc. Lather, rinse, repeat.

    Research, or what passes for research, has become a fund-raising machine. They think, “who cares about accuracy?” as they trade research crap for dollars to do more of the same.

    We’re not supposed to know the paper is (or parts of it are) bogus, and if it weren’t for Mark and people like him (Denise) showing us how to parse a research paper, we would never know. And that’s what they count on.

    1. That’s overly negative. Just TRY to come up with a perfect study, that is actually fundable and doable! Plenty of these researchers are doing their best. But the consumer should still approach the info with healthy skepticism.

      1. There’s no such thing as a perfect study, which, IMO, makes them all pretty worthless.

  4. My favorite example of “correlation does not equal causation” was a satirical article published many years ago. If I remember correctly, the article argued for the reduction of juvenile delinquency through the ban of all tomato products, The justification was a fictional but plausible study indicating that 98% of all juvenile delinquents were known to have eaten tomatoes.

  5. What recent study are you referring to mark? I did see a study referring to the paleo diet as being protective against heart disease and diabetes in elderly women. However the more I read into it the more it sounded like a mediterranean diet. It was only 19% saturated fat and a whopping 71% polyunsaturated fat. They didn’t specify if it was a omega 6 omega 3 dominant diet, which would tell us a lot. However the way they wrote about saturated fat it seemed they held a very conventional view on “artery clogging saturated fat”. Not exactly a view point most paleo/primal people agree with, myself included.

    1. I saw that study too, from Sweden I believe. Your general assessment is correct that the diet followed more of a mediterranean principle than paleo principle however I would add that is was more like a grain free mediterranean diet. You should also realize there’s a few differences in the paleo an primal diets. Loren Cordain who started the official paleo diet said saturated fat needs to be under 7%. He wasn’t and really even isn’t now a fan of saturated fat. He has changed his stance from 7% to 15%, he practically doubled it but still well below the primal diet suggestion. Cordain also believes dairy (the main source of saturated fat) causes heart disease and gives some pretty compelling research on why.

    2. The study participants ate lean meat, olive and canola oil. I haven’t noticed a lot of “paleo” diets recommending canola oil.

      1. Canola oil is frequently used in “Paleo” diets used by experimenters, frequently in rodent studies.

    3. barry – I don’t think that Mark was referring about any specific recent study. This post is about why you shouldn’t blindly trust every new nutritional study that comes along, which we already knew. He then goes into what to look for. This advice is good for the “latest nutritional study” whether it came out last week, today, next week, or a year from now.

  6. This is quite timely, as I just had a conversation about a recent study on the benefits of corn oil on heart health. My first question was ‘Who funded/sponsored the study?’ My next questions were about length of study, numbers of participants, control groups, and long-term evaluation of results. Needless to say, the conversation didn’t go very well.

  7. I look at the traditional diets of the cultures that are not too far from my geographical area and think long and hard. Is what they eat beneficial to me? I love the Paleo system of nutrition but Ikarians eat little meat and plenty of pulses. So, I eat some meat and some pulses (soaked, sprouted and well cooked) and observe my reaction. It works for me!

  8. Good article that everyone should read. I know quite a few people who change their eating habits like they change their socks, based entirely on what the latest study indicates. It’s a knee-jerk reaction, even with intelligent persons. They assume these studies are gospel and seldom bother to question them.

  9. My favorite study from the past stated that people who drank less than 2 glasses of water per day were more likely to have heart attacks. Do we think these people perhaps never exercised? And the study was funded by drinking water providers.

  10. I have to be honest here, this article depresses me. This reads to me like an attempt to make the marksdailyapple community more insular and inoculate it against research that might be contrary to the themes of this blog. I truly appreciate all the information that’s been made available in this community, but this particular article pretty much reads like self-serving bull.

    It could be a useful article, to someone who reads it very carefully, but having 15 points about why an upcoming research article is probably wrong and then mentioning just twice that research is still pretty much our best method for finding things out is probably unhelpful for most readers.

    1. My interpretation of the post is that we need to have a critical eye to all studies because that’s all we have. That’s not a self serving contradiction, that’s a honest assessment of the situation. I don’t honestly believe all people are kind and truthful and want to do good, but I do believe that me acting as if they do is the only way to create a better society. I don’t honestly believe that the other car coming towards me in the opposite lane will always stay in their lane. However, it would be impossible to drive if I behaved as if every car was about to hit me. Studies are like that. Many are false or flawed, but if you act like all of them are false or flawed, we can’t ever learn and improve.

    2. I tend to agree more with Clay’s interpretation. My personal “point 16” would be that any single study is highly likely to be wrong or flawed in some way, because that’s how science works.

      Someone constructs a hypothesis, develops an experiment to test said hypothesis, and publishes the results. “Hey, I tried this, and that’s what happened.” That’s the cue for other scientists studying the same subject to replicate the study & compare results. Some will succeed, some will find some flaws, and some completely demolish the original study. The cycle continues until something approaching a consensus emerges. It still isn’t complete, definitive information, but more that we had before.

      Problem is, this process can take years and the media – and pretty much every human on the planet – prefers quick sound bites that trumpet THE ONE ANSWER when there really isn’t any such thing. Except 42, of course. 🙂

      1. I really like both of these responses to my comments. If this is how most people are reading the article, then that’s be a good thing. It’s research and followup that separates a well-founded disagreement conventional wisdom from basically a conspiracy theory. I think it’s important to keep that in mind before deciding to reject new ideas that disagree with your current position. With that in mind, I think that questioning the merits of individual studies is also an important part of new discovery.

    3. Brian – The following comments are a direct copy and paste from my reply to barry, a few comments up. “I don’t think that Mark was referring about any specific recent study. This post is about why you shouldn’t blindly trust every new nutritional study that comes along, which we already knew. He then goes into what to look for. This advice is good for the “latest nutritional study” whether it came out last week, today, next week, or a year from now.”

  11. The short version to viewing nutritional studies and otherwise: Take everything with a grain of salt.

    The Long version: We can’t walk through life trusting no one and nothing; the better approach, would be to expect the unexpected and deal with it, one case at a time.

  12. I would also add one of my favorite #16s:

    Every biological process has a range of response. If 100 people follow the same diet protocol, some will lose 20 pounds and some will gain 3. If the average is 9 pounds lost, that says little about where i fall on that curve. The same goes for medication trials. The only way to know if something really works for me, and to what degree, is to try it for myself.

    1. This is a really important point. Carb tolerance, for example, is fat going to vary widely. Some will be responders in a bad way to more saturated fat.

    2. YES! When you are looking at nutrition, your own personal response is what you need to determine. Sure, that is not what you want for distilling general guidelines….

      1. As a biological sciences student I am in agreement with much of this article
        It’s important we do meta studies and have proper peer review to make studies as useful as possible. The Cochrane Foundation is a good example of this.
        Some comments here suggest that we can rely on our own self experiments. However, this is equally a problem and why we do science in the first place.
        Right now we all operate on our prejudices, limited information that we have that generally agrees with what we already thought. Few of us change significantly based on evidence – we justify why we are right rather than change. We do all the things that we criticise others for.
        We cannot know the effects of our diet choices on our futures just based on how we feel now or seeming improvements we have had – underneath we might be eroding our health subtly.
        So my suggestion is that we realise that just by being human predisposes us all to a whole raft of error making thoughts and that the scientific process is the most honest product of humanity we have to gather knowledge. It may take a long time for science to reach accurate conclusions due to all the reasons stated in the article, but it’s a worthwhile endeavour.
        We all know so very little………

  13. These studies are a dime a dozen! Every time I hear about one I yawn. Coffee’s good for you, coffee’s bad for you and on and on!

  14. I’ll add Reason #17: the time length of the study. I remember a study which “proved” that low-fat was better for weight loss than low-carb. Time of the study: 12-14 days. That’s not even long enough to get to carb flu!

  15. Having read the post – which I agree with wholeheartedly – I am curious to know this communities thoughts on the China Study. Real or fake? Or somewhere in between?

    1. Denise Minger debunked the China Study, google her, you’ll find her blog, it’s fascinating!

  16. Perhaps the problem in research is also a lack of what Ralph Waldo Emerson call self reliance, which can also mean truth above all else.

    ‘A foolish consistency is the hobgoblin of little minds, adored by little statesmen and philosophers and divines. With consistency a great soul has simply nothing to do. He may as well concern himself with his shadow on the wall. Speak what you think now in hard words, and to-morrow speak what to-morrow thinks in hard words again, though it contradict every thing you said to-day. — ‘Ah, so you shall be sure to be misunderstood.’ — Is it so bad, then, to be misunderstood?

  17. Mark, I’ve been a regular reader and Primal Blueprint follower for years and I love your work; thanks. However, if you believe your quote of “We know very little” is accurate, how can you profess to be such an expert on this important and rapidly growing field of exercise and nutrition.

    1. I love to read Mark’s blog for precisely the reason that he DOESNT “profess to be an expert”. He consistently presents all sides of every discussion, applies common sense, and tests every notion with a sensible approach of, “is it something mankind has eaten/done for hundreds of thousands of years? Or is this new to our genes? ” He also presents what works for him with appropriate humility and acknowledges that we are all different. I used to come here just to read about nutrition, now it’s my first stop for any subject..sleep issues, stress, plantar fasciitis, just yesterday was a post about organization.

    2. Bill, each point refers to a reason we should not trust ‘the latest nutritional study’. If we know very little (about that latest nutritional study), maybe we should distrust it.

  18. Great article and I couldn’t agree more. Now let’s replace “nutritional study” with “climate study” and guess what happens?

    Crickets…

    1. Yes, totally the same. Millions of years of verifiable climate history combined with decades of international global research in all aspects of how this interacts with biology, chemistry, geology and physics, is totally the same as a four week study on blueberries and memory on eight senior citizens. Glad you pointed that out.

      1. Keep believing it. Reason #1 will lead you to all the money behind the global cooling, errr global warming, ummm I mean the climate change scam. Now back to my 30 degree day in Michigan which used to be under a glacier before those darn prehistoric SUVs caused them to recede/advance/recede so many times. Wake up and follow the money.

        1. Yes, lots of money to be made living in the arctic and studying permafrost. Same with taking core samples in swamps. Big money. Really glamorous. Total chick magnet too.

        2. If you follow the money, you will immediately see that all the industries – and their lobbies – associated with fossil fuels are interested in debunking climate change. There is really no financial incentive in acknowledging the deeply depressing data.

        3. Actually, though I get Clay’s drift (and enjoy the sarcastic bite), I don’t think climate change studies are off the hook of the issues Mark brings up.

          There were decades of international research in biology and chemistry showing saturated fat is bad for us. Yet here we are, reading Mark’s Daily Apple because we now believe all that research was wrong.

          And yes, there is big money behind climate change research – it is called grant money, and it is not usually granted to those swimming against the crowd. Yea, I know, not as big as big oil, but it’s the same kind of big money that early anti-fat crusaders were trying for – to keep their grants and laboratories alive.

          The earth systems which govern the climate are every bit as complex, and perhaps more so, than human biology (when you get past, say, a few million degrees of freedom, who’s counting?). I actually do believe the climate is changing, and probably with anthropomorphic drivers, but to pretend we KNOW all the answers, and that climate science doesn’t have systematic flaws like those Mark enumerates is, well, a bit naive IMHO.

        4. We are chimps in the trees. Tell us we are causing environmental degradation and harming the earth’s human-carrying-capacity and you will get looked at quizzically and then all will leap around the trees and shout, thinking someone will take things from them. If you say the world is getting warmer, they will look at you jump around the trees in fear, thinking the sky is going to fall in, then look back at you looking for an answer to their problems and probably give you stuff to solve the problem (and take things from you whilst you are jumping around the trees shouting). Our affect on the planet is not in a name: but a name is important in causing an effect, which confuses everything… return to start of my comment.

        5. Look. The bottom line is whether you are on the left or on the right, whether we are talking about nutritional studies, climate studies, no child left behind standardized testing, gun control vs. gun freedom, whatever the study… you can be assured that whichever position is shown in a positive light, its advocates are benefiting or are financially connected to the study. Both sides – liberal/conservative. And both sides seem to accept the truth only when it’s convenient for them to do so. Mark – I’m sorry for hijacking the conversation. Your website is awesome and you are a great resource changing so many lives in a positive way. Keep up the great work.

      2. “Millions of years of verifiable climate history…”

        Really? Records have been kept for millions of years? Then why do we question the diet of our ancestors? Surely somebody kept food logs back then too.

        1. Yes. It’s called geology and biology. There are historical markers all over the planet. By examining tree rings you can tell when there we drought years. Core sample in ice a mile deep you can tell the co2 concentration in the atmosphere on any give time period. Same with ocean floor core samples. You can tell what pollutants were in the air as well. Our mountain ranges tell us what the sea level were, what was living in the ocean at the time, and how fast the continents are moving due to plate tectonics. Our coral reefs are rich with historical data. Do you know the central valley in California is sinking at about a foot a year due to the aquifers being drained. The lower water tables cause the earth to collapse. These can never be regained because it’s a layer of compressed clay. So every year, the total water carrying capacity of California’s aquifers is permanently reduced. The entire planet, and entire known universe, is one giant history book. All we need to do is open it and look at.

          And yes people did keep food logs in the form of isotopes in the bones, hair, and tissue. Our diet leaves distinct markers in our body. It’s a pretty accurate way to tell what our ancestors ate.

        2. “Surely somebody kept food logs back then too.”

          They did. The food logs all *said* they were eating roasted mammoth and wild kale like good little Groks, but in reality they were sneaking Toll House cookies and ice cream during Game of Thrones. :rolleyes:

  19. I submitted public comments when government nutritional guidelines were being revised. My public comments were conveniently deleted from the government website. Funny how factual, honest medical science can be blocked by the dogma of the “stakeholders” in the government-agriculture-industrial complex.

  20. Oh, thank you for this, Mark! So many times I have clients come to me with understandable concerns stemming from mass media hype over “the latest study.”

    This list offers a wonderful, clear resource–one that will help me help others understand why study findings often (or usually) aren’t what they seem.

  21. Actually, it seems that old wives’ tales have a longer time frame of reference and just might have a better basis of accuracy.

  22. This is so true. I always cringe when I read about a “scientific study” using 100 or less individuals being touted as dogma. Thanks for the post Mark!

  23. Yes…..thank you Dianne. The ‘old wives tales’ I grew up with turned out to be right-carrots were good for your eyes, fish was brain food, when baking, the ‘goodness’ of the cake was dependent on the number of eggs you incorporated in the batter (the more eggs the more ‘goodness’). And sweets were to be consumed as a treat, not as an every day event. Was this science lost to the annals of time?

  24. Thus the rise of n=1 self experimentation and finding what works for the individual. I like to experiment one thing at a time to see how it impacts me regardless of the promises from studies or experts.
    That plus a bit of understanding on how to assess an individual or study’s credibility when reading the publications. Being aware of the items you’ve listed in this post helps to start wading through the “latest thing” shoved in your face by a friend, co-worker, or client. And I think that’s extremely important. I was sent an article on how the “original paleo diet” included oat flour today. Being able to nicely explain how 32,000 years ago doesn’t exactly qualify as the “original paleo diet” and how eating oats impacts our body helped win over someone who was genuinely interested in trying the lifestyle.
    As much as I’d love to not have to explain how the latest research paper using knockout mice doesn’t wow me, I’d much rather help coach interested people into understanding why I live the way I do.

  25. These are all excellent points and should all be used as criteria to judge studies and “Breaking News!” reports. I remember how flabbergasted and stunned I was, many, many years ago, to find out that, at that time, the vast majority of health studies that were being done had only men in their studies. We’re not talking about prostate studies here. We’re talking heart, lung, stroke……all things that BOTH genders get or have. Know the rationale behind excluding women? Their monthly hormonal fluctuations would screw up the data. :\

  26. Asking the wrong research question is a problem as is misstating study elements. You read a negative study on low carb diets only to learn the rat study calls 67% of energy from carbs as “low carb” as compare to the usual 80%… and then they made up the difference with real crappy vegetable oil and blamed the results on the “low carb” diet.

  27. I totally agree with the info provided. We’re very similar at some point however we are all individual in many ways.We all react in different food groups & nutritional needs. Studies & experiment can perform in many different ways. Its up to each & every one of us to see what suits us best in the long term.
    Overall, its about taking responsible of our own health & well being.

  28. This is an excellent summary of issues with research literature. As a researcher, I constantly had to dig to find any of the details that would help to put the resulting data into context. Without knowing the conditions of a study, the data means next to nothing.
    In that study that seems to indicate that drinking coffee inhibits Alzheimer’s, were any of the non-coffee drinkers avoiding coffee.for reasons such as stomach sensitivity? Bladder issues? Any other tolerance issues that might indicate.minor health problems that could affect, ultimately, proneness to Alzheimer’s?
    In studies indicating the healthfullness of drinking red wine, are the non-red-wine drinkers avoiding red wine because of lower tolerance to alcohol, or other reasons that could connect to their overall health?
    One should ask these sorts of questions.

  29. Good post. Unfortunately, as the “research” can lead to lucrative book deals and exposure on TV, the desire to undergo more stringent scientific review is diminished. I also think you hit an important point near the end – that research that shows negative or even neutral results usually gets tanked. Its hard to sell a million copies of a book entitled “We Aren’t Sure What You Should Eat”.

    1. Not so fast J L Gentry. You may just have a million copy best seller here.

  30. One could argue that n=1 is in some ways the only valid study there is. To base a science on differences between a hypothetical average person in one group and a hypothetical average person in another group, neither of whom truly exists and whose data overlap so much that one can only make probability statements about whether the differences are “real” is in some ways ludicrous. Consider for example the recent finding that there can be wide variation in how different people respond to different diets, which was touted in the media as a real breakthrough. This has for years been common knowledge on blogs like this where n=1 self-experimenters interact all the time. A major problem in nutritional research is the over-reliance on control group studies which are designed to compensate for individual differences and statistically wash away variability to the point where researchers can essentially pretend that it doesn’t exist. Unless your concern is simply to obtain some data on relative effectiveness of different interventions for the “average person” — e.g., so as to reduce overall health care costs — group designs and statistical analyses are in some ways the scientific equivalent of burying your head in the sand.

    This also suggests a way to more effectively investigate the effectiveness of different diets. Establish research centres with live-in participants (perhaps hire people who are unemployed) whose diet is systematically manipulated over time, e.g., by repeatedly alternating phases of one diet versus another over a period of, say, a year. Each participant in the study would be their own control and would thereby constitute an n=1 experiment, with each other participant being a replication of that experiment. Thus, 20 participants would constitute 20 complete experiments, experiments in which individual differences will be front and centre and in which the major effects of each diet, if they truly are major, will likely need little if any statistics to determine. I suspect you could easily obtain enough funding for the study simply by cutting by half the amount of money presently spent on poorly controlled, largely useless, nutritional studies on groups of people/rats/mice. Unfortunately, the processed food industry would quickly realize what a disaster this could be for them and would do everything they could to prevent such studies. Real data on real people. God forbid.

  31. Good debunking analysis. The obvious question, though, is can you offer scientific support for paleo/primal that avoids your 15 reasons?

  32. Medical, biomedical and biological research as it is done today, it is about: the prestige of the principal investigator and the team, the funding body/company and what they are expecting from the PI and team, and the prestige of the University or organisation where said PI and team do the research. Then, it comes down to publishing and there is quite a debate about the non-reproducibility of many a study and the whole publishing arena is another can of worms. A study is supposedly to be more credible if published in a journal with a higher score. Many nutritional studies are published in journals founded by food/drug companies. It is not a question of conspiracy, it is the reality and business of science. Whoever works at a university knows. When it comes to nutrition, it is better to say, this works for me, and this is what I found, I tested it and works for me and ignore those who attempt to control us or manipulate us to feel guilty for our choices.

  33. The article and comments make some good points. My background is in toxicology and occupational hygiene. It was Philippus Aureolus Theophrastus Bombastus von Hohenheim who said “The dose alone makes a thing not a poison.” Why any parent would inflict a total name on their offspring that is that long is beyond me So, PATB Von Hohenheim had a pen name “Paracelsus”. At the end of the day it is all about dose and response. Most of these studies really measure either associations or a qualitative relationship. Medical science overall tends to emphasise the response above the dose. That is one of the reasons why we have for example a vitamin D pandemic world wide – although primal blueprint adherents get their dose of D one way or the other. Then there are routes of exposure. In the case of D the effects of oral D may not be the same as the effects of D made in the skin from UV penetrating into the skin. There are a lot of isomers of D….. Then were have the potential for additive effects – for example heat (reduces central blood flow) + carbon monoxide + methylene chloride (metabolised to CO) will have a greater effect at same dose when given together rather than separately. Then we have synergistic effects where 2 doses multiply. It was when a cohort of smokers who worked in asbestos textile companies were looked at that we discovered asbestos and smoking multiplied the risk of lung cancer. Then even macronutrient changes in diet can have a big effect…. if you want to shift the damage from the liver to the kidneys just reduce dietary protein. Liver tends to down regulate and kidney copes more of the load.
    Personally I emphasize the toxicology\biochemistry etc less as I have gotten older as getting right back to basics is more reliable. Meat fruit veg tree nuts and fish, a bit of exercise- including some postural prehab to stave off the effects of “civilised life”, a bit of sleep and an emphasis on stress management including the reduction of unnecessary nervous impulses (anxiety) by practicing relaxing meditative experience. Throw in a bit of variation to replicate what happened in pre-agricultural times etc. OB

  34. Oh fabulous! I need to summarise this in point form, print it out, and carry it with me at all times for all those occasions when people say, ‘Meat causes cancer, all the scientific evidence says so.’

  35. In my opinion there is a significant omission. By definition food is grown and therefore has characteristics particular to the place where it was grown (“terroir”)

    Even if all 15 shortcomings could be ironed any study that concluded eating “XYZ is good for you” would mean little to me if the study participants ate XYZ grown in Arizona and I eat XYZ in Dusseldorf.

  36. Really good article, interestingly I watched this interview about cholesterol last night https://www.youtube.com/watch?v=7gZt9DQqtZI

    What resonated with me was when Dr Krause basically says that the kind of funding structure to definitively do a proper wide ranging dietary study to put the matter of diet to bed once and for all has diminished so we may never know for some time.

    Lunacy really when you consider the amount of people dying from degenerative diseases year on year that they can’t put a few £/$ into a pot and work out why??

    The only conclusion I draw is that there are a lot of people with vested interests in maintaining the status quo – ultimately motivated by money despite the fact it could save 1000’s of lives.

    To me its the same logic as to why the UN wont sanction the Gulf States for their role in funding ISIS – oil and weapons revenue.

  37. One other element is there are occasional studies that actually demonstrate conventional wisdom and status quo as incorrect. However, the scientific journal peer review staff are often are influenced by special interests and/or professional jealousy. For this reason, good information and breakthrough findings are often denied publication. This supports the observation that there is still so much we do not know but somebody may have solid evidence that is just not available through conventional channels.

  38. Once again, corporatism infiltrate every walk of life.

    I’m pro-capitalism but it’s sad to see this state of things. Its sad that we don’t know what to believe anymore

  39. I’m 73. I’ve tried to live healthy and I am. My favorite over the years has been the discussion of eggs. News flash…eat eggs and die of a heart attack soon. The next day science has proven eggs are good for you, followed by OMG colesteral is certain death. In my opinion after all the news regarding eggs, the only way they will kill you is if you eat them without chewing first.

  40. “nuts reduce mortality risk by 40%”

    I love the studies that report increased or decreased risk of mortality. Last I heard, we all have a 100% risk of death.

  41. 16. You want the study to be wrong.

    Come on:
    6. Small sample size, 10. Statistical significance versus clinical significance, and 14. Journals prefer to publish and researchers prefer to submit exciting studies with strong results.
    Statistics take into account for small sample sizes. In (6) uncertain results are bad but suddenly in (14) they are good. This list is rubbish.

  42. Hello Mark! Wonderful points you have discussed . I completely agree with your research work. But the fact is that we have no nutritional food in real in the market . We only have option that either we should be educated so that we can easily judge what is real and what is fake in the market. or we should contact direct to the farmer to get real organic food with full of nutrition from there.

  43. Excellent points! the sad thing is that when a person has been wedded to their idea for a long time they know that if they waver from their position they may lose the respect they are accustomed to. It’s possible to recover from this position, but it is a long and hard road if not handled deftly.

  44. So glad that we’re thing in this manner. Gives me hope for the pursuit of truthfulness.