I love everything dairy, I may get 3-5 serves of it daily. 2-3 of yoghurt, in coffees, cheese if it's around, etc.
I know it's not Paleo, because we didn't eat it 10, 000 years ago? We aren't adapted to it and it upsets our stomach? They are the only 2 reasons I know people on Paleo don't eat dairy.
Before I gave up the grains, I was bloated most of the day, felt "heavy", thought it might be milk, or gluten, tried to work out which, couldn't really... though not drastically over-weight I was yo-yoing, over-eating, etc etc.
Then I cut out the grains, but upped the dairy, especially the yoghurt. My stomach has never felt better, my weight is perfect, I'm satiated and not over-eating and I'm building muscle much more effectively.
So does it really matter that we didn't eat it 10,000 years ago?
My opinion is, if it disagrees with you, stop eating it. If it's doing you good, eat it.