I'm one of the (rare) people on the forum who eats quite alot of dairy (all pasture-fed, all raw) incl. cream, cheese, butter, yogurt and occasionally milk.
I think it's fine as long as you are not sensitive to casein....and it can be a great source of saturated fat. Goat's milk has only trace amount of casein (less than cow's) so it's definitely an option. For a few years I went cow-dairy-free (long story), and used goat's products as an alternative during that time. The only issue...most goat's milk (and cheese) is pasteurized or ultra-pasteurized. So, see if you can find raw options.