Meet Mark

Let me introduce myself. My name is Mark Sisson. I’m 63 years young. I live and work in Malibu, California. In a past life I was a professional marathoner and triathlete. Now my life goal is to help 100 million people get healthy. I started this blog in 2006 to empower people to take full responsibility for their own health and enjoyment of life by investigating, discussing, and critically rethinking everything we’ve assumed to be true about health and wellness...

Tell Me More
Stay Connected
June 23 2015

The Pitfalls and Limitations of Self-Experimentation

By Mark Sisson
23 Comments

I’m a huge proponent of self-experimentation. We can’t always rely on funding for research relevant to our needs, interests, and desires, and those studies that are relevant are still using participants that are not us. We like control, when it comes down to it. We want to be the arbiters of our own destinies, and running (formal or informal) self-experiments of 1 can help us get to that point. But as helpful as it can be, there are both inherent limits to self-experimentation and common pitfalls people fail to take into account when designing their experiments of one.

I’m not referring to the basics of experimentation, like the need to control for variables or the importance of limiting the number of interventions you test at once. You guys know that stuff. I’m talking about the limitations most people don’t foresee:

No baseline measurements.

The best studies establish a baseline against which any experimental effects are plotted; these are baked into the design. If you have no baseline, you can’t compare your “findings” to anything. In fact, you don’t even have any solid findings without a baseline. You just have feelings, or hunches, which are fine, but they don’t tell the complete story or establish causation. Since many self-experimenters tend to be less formal than clinical researchers, more “fly by the seat of their pants,” failing to establish a baseline is a common weakness.

Lack of placebo.

When applicable, a placebo — an inactive control against which the active experimental intervention is compared — strengthens an experiment’s impact. If a person takes the placebo (often a sugar pill) and it works just as well as the active intervention, researchers know the latter probably isn’t all that active after all.

Inserting a placebo into our self-experiments isn’t always viable or possible, of course (how would you placebo control an experiment testing the effect of cold water plunges?), and it adds a level of complexity to what we often expect to be a simple case of “let’s try this out.” Also, some people simply scoff at the notion that they’d be susceptible to placebo. (“Sure, those anonymous people in the other study might be “weak enough” to fall for it, but not me!”) Which is why it’s easy to forget (willfully or otherwise) to use a placebo.

Lack of blinding.

The best experiments are blinded, which prevents the participants from knowing whether they’re receiving the active intervention or the placebo. This is important because the simple knowledge of what you’re taking can change your response to the intervention, especially when it’s something you “want” to work — which is exactly what most self-experimenters are testing. Blinding allows researchers to measure effects independent of the subject’s expectations, desires, or biases.

Lack of double-blinding.

In a double-blinded study, neither the participants nor the doctors know who’s receiving the placebo and who’s receiving the treatment. This is important because if clinicians know they’re giving subjects a placebo, their interactions with subjects will change in subtle, subconscious ways, and this can affect the effect of the treatment. Ideally, double-blinding also blinds the data collectors, outcome adjudicators, and data analysts involved in the study to remove any trace of bias or expectation.

Lack of applicability to other people.

Most people who smoke cigarettes never get lung cancer. We all have an anecdote of a grandparent who smoked multiple packs a day and lived to the ripe old age of 90+ without ever developing a serious smoking-related disease. And if that cancer-resistant grandpa conducted a self-experiment to test the effects of smoking on cancer risk, he’d conclude that it was completely safe for him. And it would be — for him, in that instance — but because of research involving tens of thousands of subjects, we have a strong idea that smokers have a 15x greater chance of contracting lung cancer than non-smokers. The advantage of funded research and experiments of 20, 50, or a 1000 people is that the results are more applicable to more people.

This limitation isn’t a problem if you acknowledge that your self-experiment is about you and you alone and withhold application to the rest of humanity, but it is a limitation.

Regression to the mean.

Eventually, we recover from illness on our own. Left to its own devices, the body will heal. And while it’s attractive to ascribe credit to the vitamin C we took for stopping our flu in its tracks, the same thing could have happened had we taken nothing at all. This is even more likely when we realize that people generally turn to self-experimentation when symptoms are worst, where there’s nowhere to go but back to the baseline.

Using short-term effects to extrapolate long-term ones.

You might feel great in your month-long experiment foregoing sleep at night for micronaps throughout the day, but does that mean it’s safe? It might be. Maybe it is an effective replacement for traditional sleep patterns. But one month is not enough time to tell us that.
This is a problem with most studies, of course. But self-experiments aren’t exempt, and since we tend to trust ourselves and our experiences over the results from a PubMed abstract, self-experimenters are particularly prone to unwarranted extrapolation.

Most of these aren’t unique to self-experimentation; if it is to be powerful and valid, any type of trial must account for biases, blinding, placebo, and improper extrapolation. Most can be avoided, mitigated, or overcome with careful planning and rigorous design. But self-experimenters are all too susceptible to forgetting about or ignoring these limitations and skipping the rigorous design to get to the good stuff. Because, well, it’s kind of hard. (So I wrote a book to make it easier.) That’s the rub, though. We can cruise PubMed and read hundreds and thousands of studies whose authors accounted for everything for us. They did the work. But when we run a self-experiment, we must do the work because we are subject, researcher, analyst, and clinician.

We’re not just the participants in self-experiments; we’re also the authors. We handle everything. We can’t relinquish control to an external authority. Self-experimenters can certainly use placebo controls and blind themselves, but it requires a level of dedication and rigor that, for some people, is just unrealistic or beyond their means. Also, we often run self-experiments just to “try things out” or “see how it works,” and if it works, we’re happy. That our guess might work because of the placebo effect doesn’t bother most people who only want to make themselves healthier. That’s fair, but it also means our results can’t establish causality.

I don’t expect everyone to establish definitive cause and effect for every decision they make, bite of food they take, and lifestyle modification they enact. That’s just ridiculous. We’re not trying to pass peer review here. We’re trying to live better. And not every “study” needs to be rigorous. But by heeding some of these common self-experimentation pitfalls — not even necessarily all of them — or at least being aware of them, we can increase the power of our forays into personal science.

Are you a self-experimenter? If so, have you noticed yourself falling prey to any of the pitfalls mentioned in today’s post? Are there any that I’ve missed? How have you handled it in the past, and how will you do it in the future?

Thanks for reading, everyone. Take care!

Prefer listening to reading? Get an audio recording of this blog post, and subscribe to the Primal Blueprint Podcast on iTunes for instant access to all past, present and future episodes here.

If you'd like to add an avatar to all of your comments click here!

23 thoughts on “The Pitfalls and Limitations of Self-Experimentation”

Leave a Reply

Your email address will not be published. Required fields are marked *

  1. The thing is nutrition (and exercise) is especially hard to study objectively. We can’t really do placebo exercises, nor can we do many dietary interventions in placebo controlled formats.

    Chris Gardener’s A-Z study shows that the LFHC diets did not work for insulin
    resistant people and the government is pushing very low salt diets heavily, which benefit a very small percentage of the population, but if everyone adapted the guidelines the death rate would increase by 40%. Once the Anointed[1\ get started with a program, they stick with it come Hell or high water.

    [1] Oily to rise and oily to bed makes you anointed all day long.

    1. Left out conclusion, perhaps the only way to find out what works for you, is the N=1 method. Wurst case if it works for you as a placebo, it works for you.

      Lately I’ve been doing fasting periods of between 16 and 36 hours, and it seems to help. It’s a drag to stop fasting and have to divert energy to digestion and lose the ghrelin high. I seem to be much more clear headed while fasting, which might be some psychosis coming on. OTOH, I am getting posture corrections, I tend to slouch and lately I spontaneously straighten up without thinking about it.

  2. I don’t think it is necessary to replicate every aspect of RCT protocol for an effective N = 1 nutrition experiment.

    And most of us have baseline data if we were going to doctors. I looked up my pre-Primal numbers and they were very scary. Trigs/HDL were 7. Last check they were under 1.5 and I suspect the ratio is under 1 now.

    The placebo discussion is interesting because placebo is real. It is the mind-body connection. The funny thing to me is what do people think they are doing with meditation? Essentially harnessing the placebo effect, or practicing gratitude? Healing the body and mind, with the mind. Placebo always happens, we just have an odd way of thinking about it. Cancer survivors? Almost the biggest predictor is attitude. Yep placebo better than chemo in most cases. Of course I would go keto as well.

    Nutrition is very well suited to N=1. If you are good at observation and accounting for variables, recording (if you want), N =1 are extremely powerful for eating. When I give talks I tell people don’t believe me. Think I am making this all up. But you can see the results in me. Give up grains for a month, listen to your body, then if you think I am full of it, let me know. No one ever ends up saying I am full of it.

    1. I think that meditation is not about being a placebo but more that it helps to deal with stress.

  3. And there’s also the whole aspect of our individual genetics. I have a vague idea that I’m sure others have developed much more fully, which is that it takes a lot for our body to move away from the homeostasis underlaid by our genetics: humans must thrive under highly variable conditions, and the costs to the organism of constantly adapting to highly variable conditions would outweigh the benefits, and thus our bodies maintain a relatively constant condition. For example, large weight changes represent a large cost to the organism in the sense of the energy, nutrient, hormonal and etc changes involved in reshaping our bodies, and thus the body has evolved to be “resistant” to wide changes due to high variable conditions.

  4. > “Using short-term effects to extrapolate long-term ones”

    That’s the biggest pitfalls of self-eperimentation. How many people decided that vegan is they way to go after just 1 month of what, if you check closely, is just a detoxification from the SAD?

    1. Quite, even a few years can be ‘survived’ but the long term health effects of some ‘nutrition experiments’ can be very debilitating and it’s gradual, so the connection with the original nutrition experiment may be missed – Denise Minger is a great resource for all things that can go wrong!

      1. I totally agree. The process may last years and sometimes it is just accepted as “natural”.

    2. Great point Alex. And on the flip side someone like Mark doing Primal for 30ish years is much longer than any RCT. Very hard to get RCTs for two years.

      So actually N=1 can have the advantage here for time. Now 2 plus years for me. Easier to do than ever, no weight gain (after reaching goal), no calorie counting, no hunger, keep getting stronger.

      My wife has a good vegan friend from her childhood. She has even written some books. It is amazing how often this friend mentions feeling lousy, especially mentally, but often sick.

      However her response is thank goodness for vegan anything else would make it worse. So yeah N= 1 can rock, but not if you are misinterpreting the messages your body are sending. But yeah this person probably felt great for a month or 6 months or a year.

    3. What’s scary about the Vegan diet also, especially the ones that specify a high amount of fruit and an excessive amount of calories, is that your blood work can be great for 10-20 years, even in the face of excessive fructose, and if you do masses of cardio, you might get away with burning a lot of that fructose and reduce the metabolic damage, and excessive oxidative damage.

      Looking at straight out blood work – your readings can be fantastic, right up until metabolic damage is well under way, and it may take many years.

      All that is really happening though is your pancreas is creating more and more insulin to combat the excessive glucose, so its not until you get pancreatic burnout that the cracks really start to show, and then its too late (your telomeres in all your pancreatic cells are gone, meaning that new pancreatic cells can no longer be created – you just hit the end of your genetic lifespan) – you might be thin and healthy, but once you use up the pancreatic cells, you’ll self-destruct.

  5. The placebo effect was studied giving two groups medicine. One was given a drug costing “$1500” and the other it’s generic option. The group getting the more expensive responded better, but in both groups they were both getting a placebo. If that’s what we get with only 10% of the human brain being used, imagine the potential!

    1. The whole “we only use 10% of our brain” thing is a myth. Interesting story though.

  6. “If a person takes the placebo (often a sugar pill) and it works just as well as the active intervention, researchers know the latter probably isn’t all that active after all.”

    It depends on what you mean by ‘active’. Sometimes, placebos have a significant effect (mediated, obviously, by someone’s beliefs). This is a problem with trials that only have a placebo control.

    Better are trials that have a group using the intervention in question, a placebo group (‘sugar pill’), and a group doing nothing different from what they were doing before.

  7. Jumping into a Primal lifestyle at age 48 few years ago, I changed a host of variables: diet, exercise, sleep, going barefoot, standing at my desk, getting out in the woods more, meditating. No way to isolate the separate influences, but it’s had a great effect. The way I look at the health picture includes of necessity all of these elements–change one and it affects the others. I rely much more on intuition and don’t feel inclined to hack or tweak. I respect those who have the control and desire to go all N=1, but man, that’s not me.

    1. Yes, this. I just do what feels right. I’ve got better at recognising what is right and not so right for me over the years but for me self experimenting can get a little obsessive so I generally avoid it.

  8. Self experimentation should be a positive experience. If there are potential pitfalls and limitations, the experiment may be too precise to provide conclusive evidence. If I avoid grains, I lose body fat and feel better. This is positive and conclusive after one week. I’m not sure if yellow light or melatonin supplements help me sleep or not. I do know that avoiding alcohol is the key to a full nights rest for me, no placebo necessary.

  9. I’m really not sure how placebo would even work with self-experimentation. When I did my sleep experiment, I could definitely tell whether I was wearing orange glasses or not. How would one even do it?

  10. Something important to consider is that almost all researchers have agendas (someone has to pay them, and they also have their own biases and beliefs like anyone else), and that statistics can be manipulated – without outright fraud – to serve that agenda.

    Also, I’ve read that several human and animal trials to test a high-fat, low-carb diet either weren’t low-carb (150g or more, only low by USDA food pyramid standards) or used Crisco and similar no-go oils as the fat source.

    I know most of us are aware of this but I’m making the point that “proper” trials aren’t in themselves some kind of gold-standard of objective truth-finding.

  11. The first and last pitfalls also impact the perceived effects of the lifestyle changes that go along with getting primal. It’s hard to know how far you’ve come when you don’t know where you started.

  12. I have had some issues with vertigo and since it is not well understood by the medical community have done a fair amount of experimentaiotn with remedies [both diet, stretching and exercise]. Although certain things may work to mitigate symptoms it can be very challenging to determine cause and effect and especially the durability of the effect. I regularly discuss the self experimentaion I do with people in medicine to gain perspective on what I am attempting to do and on interpreting the results. Relying on yourself to be the sole source of analysis of experimental results is not a good way to find long term solutions.

    1. You have my utmost sympathies dealing with vertigo. I had it for nine weeks until they diagnosed me with the physical versus neurological version and were able to ‘fix’ me after three attempts with the Epley Maneuver. Took three days after that to feel like I was back on land. Vertigo. Sucks. I was told it can come back at any time.

  13. The major problem with group experiments is that statistically average results can easily have little or no relevance to us as individuals. In fact, one could argue that a major reason for the poor state of nutritional science, and perhaps even medical science in general, is the over-reliance on group research and the lack of appreciation for individual variability.

  14. I think the ONLY valid experimentation that you can do for your own diet and well-being is personal, individual self-experimentation. Sure, you cannot necessarily extrapolate from your own results for others unless you have the checks and balances you mention here in place. But for yourself? Self-experimentation is the way to go!
    I think that individuals vary so widely in their response to various diets, and individuals have such widely varying physical abilities and types, that group approaches are rarely very helpful.
    Actually, now that I look around, I absolutely agree 100% with Rap!