For the most part, I try to avoid the subject, to be honest with you (although I definitely did have my annoying proselytizing phase).
When pressed, I'll say "a hunter-gatherer way of eating -- meat, vegetables, fruit -- and I've never felt better" (I specifically do not use the "D" word). I don't say anything about no grains, sugar, or dairy and let them make the connection for themselves if they wish to give it further thought.
I did tell my boss once, in the course of a discussion about how I no longer call sick into work all the time, "I stopped eating sugar and I'm cooking everything from scratch now." That seemed to go over well.
If someone point blank asks me why I'm not eating any bread or ______, I say, "I like bread (or ______), but it does not like me."
"Let food be thy medicine and medicine be thy food." -- Hippocrates