Ref the exercise, yes, if you're finding it is derailing you mentally then probably a good thing to reduce it. Also, bear in mind that primal fitness suggests lots of just moving about, so maybe some longer walks or gentle bike rides with just one gym session or prints or something per week.? That way you get to move your body and get sunshine (if you have any over there ...) and feel good about not just sitting around and you're still moving towards those abs and back muscles.
I've had to drop dairy completely. In my case I had acne for 35 years and while I didn't think it bothered me, living without it is so much better, I have no worries if I never have cheese or ice cream again and I've even adapted to drinking teas withut milk!