It really depends on how YOU feel drinking it. Personally, I usually stick to one 12oz. cup per day, but you might find that you do well with more.
The only way to discern its effects upon your body is to test it -- make note of how certain amounts effect your sleep, weight, skin, mood, energy, etc. If you experience negatives in any of these areas, try drinking a different amount. As a general rule with coffee, more is not necessarily better.
As a side note, the quality of your coffee can be just as important as the amount you drink. You can order unroasted coffee and roast/grind it at home -- unroasted single-origin coffee can be found online for about $5 - $7 per pound, which is an incredibly better deal than paying $14+ for a pound of stale coffee that was roasted or ground six months ago. If you use 3 - 4 tablespoons per 12oz. of coffee, one pound will give you 8 - 10 cups of coffee, which ends up being between $0.50 - $0.87 per cup. Awesome price for some of the best quality coffee you can have!
As far as your dairy intake goes, I wouldn't stress it. Heavy cream and yogurt (along with butter) are some of the more friendly types of dairy available. Again, if you're worried, test it. The most obvious signs that dairy isn't working for you are skin issues, digestive issues, fatigue, and weight gain.