How much calcium rich foods do you eat? That amount of magnesium seems fine, because excitatory calcium with magnesium, the calcium could be a problem. The calcium has to move somewhere, ideally in the bones. If something is wrong with calcium metabolism(magnesium, vitamins a, d, k) it can lead to problems.
I usually get a lot of magnesium through oxide, epsom baths, coffee, chocolate, and only drink 1-2 quarts of milk a day.