I just say that I've cut out grains and sugars. Doesn't seem to rile people up too much. If somebody presses me, I explain that grains in particular cause inflammation which is bad for my autoimmune disease. That shuts them up.
Everyone at work DOES bitch about my sardines I put on my BAS though. Out of kindness, I've promised not to eat sardines at work anymore. =)