Your weight seemed kind of low to me, so I looked at online BMI and weight calculators. You're pretty much right in the middle of the healthy BMI calculation and the healthy weight for a man with a medium frame. People who are giving you a hard time just don't know what they're looking at. Do they see how much you eat?
DH and I have had a similar experience right about the time our weights dropped from the overweight range into the healthy weight range. Telling people "I would have to lose X more pounds to be considered underweight" just elicits a bunch of blah, blah, blah about how you can't trust those charts. I think people are so accustomed to fatness that they have no idea what is healthy.
The only thing I ever say about how I eat is that I've pretty much eliminated processed foods from my regular diet. Who can argue with that?
CW-125, part calorie counting, part transition to primal
GW- Goals are no longer weight-related