I don't eat much dairy but I do add a little to my coffee. The image below shows the nutrient content of back coffee versus coffee with milk. Are you sure you want to stress yourself out for a minor improvement? You could just eat some almonds with your coffee. 5 almonds will provide you with 10% of your vitamin E requirement.

coffee.jpg