What's outrageous about this is that there's a stigma for taking good care of yourself. That's completely backwards.
Anyway, my default response is "I feel better when I eat well". It shouldn't take any more than that. If they can't handle your lifestyle, that's their problem and not yours.