I've always reckoned its a combination of both diet and exercise and they interact differently for different people. I don't know how tall you are or what your frame size is, so it's difficult to say whether 155-160lb is heavy or not. I can't get below 140lb - I'm 5'5" - just seems to be the way my body is. I'm not built to be skinny. However, like you I build muscle easy if I work out a lot, so maybe it's just your body type?
Having said that, maybe you might want to get your thyroid checked if you're concerned?