Yep, 'dairy' is a very broad-brush term so treating all dairy as equal is meaningless.
Development of lactose tolerance into adulthood in humans is the only widely acknowledged significant diet-related genetic change since the neolithic era. It has helped fuel population growth, for better or for worse. No other mammal has had the intelligence or mastery of tools to make this happen, so far.
F 5 ft 3. HW: 196 lbs. Primal SW (May 2011): 182 lbs (42% BF)... W June '12: 160 lbs (29% BF) (UK size 12, US size 8). GW: ~24% BF - have ditched the scales til I fit into a pair of UK size 10 bootcut jeans. Currently aligning towards 'The Perfect Health Diet' having swapped some fat for potatoes.