Ugh...I run into this constantly with my Granny. 5'10, 165lbs, and pretty low body fat (probably low double digits), athletically built but not overly so. Unfortunately most people don't know what you look like underneath your clothing. If they want, I could show them my near 6 pack abs and my bat wing lats to prove to them that it's more muscle than anything!
If only my granny could see my twitter pic to prove it.