When I was young and skinny and working on the farm I drank a from a pint to a quart of milk for lunch pretty much every day I could.
Quarts when I had late lunch in summer after a long morning of hard work.
It was almost always fresh raw goat milk... because that was one of my jobs. But still.
I stayed lean drinking it too... but I worked hard.
With your body type and metabolism I don't think it will hurt you one tiny bit.
I say go for it... as long as digesting dairy isn't a problem for you.
As always... if you can get at least organic it's REALLY a better option.