Mostly it appears to be a question of how long the human race has been consuming them. The longer we've been eating a food, the more time we've had to adapt to it. On the scale of history, we just started eating grains and dairy yesterday. I'm sure others will chime in with more specifics. People with different genetic backgrounds react differently, too -- just remember that a food may cause you problems which aren't "obvious" or don't seem related.