Because I see today, how much better I am treated as an "attractive person" in parts of life that are irrelevant to being attractive. Like when I was in design school, I used to go to studio to do my work, and other students mooed at me. I was not asking them for dates or to assess my weight. I was trying to get my assignments done. I was harassed constantly and hence had to work at home and was less productive. I'm not mad that there were pretty girls. Or that they liked the pretty girls. I'm mad that they harassed me because I was not "pretty" despite the fact that good product design has nothing to do with one's appearance.I don't understand being a sour grape over it, I guess.
I think above all, fat people want to be left alone. I don't think they mind when you look at a beautiful woman and call them beautiful. I think they get mad when they start being degraded for not being beautiful in irrelevant situations. It sucks the enjoyment out of things.
You are super lucky that it rolls off your back. For many women, it absolutely does not and it ends up being a limiting factor on what they can achieve because it erodes self confidence over time to be put down for your appearance in every aspect of your life.