Well my first instinct would be to say grain, hands down.
Some people, myself included, don't give up dairy when going Primal. I don't have any problems with it though.
Since you do, I'd say aviod any dairy that bothers you. So if you can do paneer and ghee without problems, don't worry about it. If all dairy bothers you then ditch it all.