Generally, if it's lactose, you will fart a lot. If it's the casein, you will have allergy symptoms. This is because lactose intolerance means that the sugars aren't digested and they ferment in your intestines which causes gas and casein intolerance is an actual allergy.
The trouble with elimination diets is you can down-regulate your ability to digest lactose by going off it for a period. You'll become lactose intolerant when you weren't before. This has happened to me to some extent because I'm just not a regular eater of dairy even before going primal. It has only gotten worse over time. I have to eat a little bit just to keep my ability to eat it at all.
I would highly suspect that if dairy products cause weight gain it's probably the calories. Cheese, yogurt, ice cream and other such delicious foods are so easy to eat way too much of. Who can stop after a 1" cube of cheese? Not I!
Female, 5'3", 49, Starting weight: 163lbs. Current weight: 135 (more or less).
Starting squat: 45lbs. Current squat: 157.5 x 2