When I was a kid- about 4 years old- my parents both were without jobs. My dad was laid off and my mom was pregnant. They foraged, hunted, and the farm down the road had Scottish Highland cows (they still do.) A couple of the cows had calves and the farmer told my dad he could have some of the milk. I remember going to the farm with my dad and trying milk right from the cow. My mom would make butter- or more appropriately would make my older siblings make butter. It was so warm, rich and good.
I have access to raw goat milk now from an Amish farm. I don't really drink milk at all, but I make cheese from it sometimes. I generally limit my dairy to butter and cheese, and the occasional heavy cream. A taste of raw milk is wonderful from time to time though.
If you like it, and your body tolerates it, go ahead. That's my 2 cents.