I don't believe that low-fat milk would be better for you Gadsie.
Low-fat milk is lower in fat-soluble vitamins and cholesterol - both of which are very supportive for building muscle.
As forgetmylastusername indicates, it's the hormones that come with higher fat dairy that may be of concern to you. If so in your case, then go for organic.
F 5 ft 3. HW: 196 lbs. Primal SW (May 2011): 182 lbs (42% BF)... W June '12: 160 lbs (29% BF) (UK size 12, US size 8). GW: ~24% BF - have ditched the scales til I fit into a pair of UK size 10 bootcut jeans. Currently aligning towards 'The Perfect Health Diet' having swapped some fat for potatoes.