There's a whole range of opinions on this, and no easy answer. If dairy does not upset your digestion, then I'm inclined to say it's no big deal. Mark put out a post a few weeks ago about the spectrum of choices with meat, fish, veggies, etc., and IMO the best to worst would go: local, pastured, raw > local, pastured, pasteurized, un-homogenized > local, pastured, pasteurized, homogenized >> store-bought, organic >> store-bought, regular; and of course, whole > skim > NF.
Check out Dr. Cate Shanahan's discussion of milk in Deep Nutrition.
I tend not to use a lot of milk per se, but heavy cream, sour cream, yogurt, and cheese - yep. Fortunate to live in a state that allows (begrudgingly) raw milk sales and has several venues closeby where I can get some.
A way a lone a last a loved a long the ... riverrun, past Eve and Adam's ...