Genetics play a large part. At what I call my 'stick' weight, I can assure you that even if I could squat/deadlift my apartment building, I'd never have huge thighs or a great bubble ass. We just don't have bubble ass genes in my family.
For the swimmers among us. Nothing scientific, just something I was told. A friend swam competitively in college. After college, she got a job as a director at a local Y. No time to swim, but coordinating classes meant that she taught both her classes and would sub teach for other teachers/leaders. She lost about 30 pounds not even trying - really, she seemed to always have a bag of something salty/greasy in her hands. She told me that swimmers develop a bit of fat to insulate them from the cold. Don't know if it's true, just putting it out there.
"Right is right, even if no one is doing it; wrong is wrong, even if everyone is doing it." - St. Augustine