I think you're right.
Just like some people naturally have more muscle than others, some get "ripped" if they do certain things, and others don't.
I notice when looking at photos of aboriginal people in excellent condition that their superb muscles are not defined like the bodybuiders work for. And they look just fine.
Carrying some fat is normal. Getting rid of it for cosmetic reasons is harder for some people than for others. One also wonders why they want to do this.