I recently did 3 weeks of a strict candida diet, which is similar to the Whole 30, except that I wasn't consuming any fruit, and actually am still not (no sugar allowed, even from natural sources). I loosened up recently and have had a few drinks and some chocolate, which has been nice, but I was feeling SOOO much better on the clean diet.
I think that we are less sensitive to things like dairy and wheat while we are continuing to eat them because our bodies are in a constant state of inflammation. I remember reading somewhere (I think it was in The Paleo Solution), that even a small amount of gluten here and there is enough to keep us inflamed. Once we've removed the causes of inflammation and our system gets cleaned up, I think it becomes much more sensitive when we reintroduce any of the offenders.
I'm thinking that the heavy cream may have been the culprit with my breakout. I stopped having it and didn't get any new pimples. This week I had a couple of cappuccinos with it and have got a brand new spot this morning. So now I know that it's definitely something that I can only do in moderation!