If you're not intolerant to dairy, I don't see any reason to avoid it entirely. Caloric density is really the biggest thing to fear there, but obviously that's relative to the rest of your diet.
Goat products are a good option. Goat's milk tends to be lower in lactose than cow's, and in some cases is safe for lactose-intolerant people. Chevre can be used to make a cream sauce, and feta & chevre have roughly the same protein/fat ratio as eggs.
If it's sold in your area, Daisy cottage cheese and sour cream don't contain all the nasty gums and emulsifiers that most others do. And the 2% cottage cheese is packed with protein and a decent amount of fat at only 90 cals.
“The whole concept of a macronutrient, like that of a calorie, is determining our language game in such a way that the conversation is not making sense." - Dr. Kurt Harris