I think the body has a "sweet spot" for weight loss that changes as we get trimmer (though not linearly). Too little food and body uses all it can to just run itself (I don't know about starvation, per se, but more along the lines of enough enough to get through the day). Too much, and well, we know what happens.
I also firmly believe sleep, stress, etc. play a big role. I could be wrong, but I've experienced it for myself (although maybe I eat too much when I'm tired, stressed, etc., and then we're back to calories).
As far as exercise, I don't believe it's necessary for weight loss. In fact, too much, and it messes with the appetite. I workout to strengthen my body for things I enjoy - like hiking. I also use it to maintain muscle mass so that hopefully when I do lose, I'm not losing my hard-earned muscle Actually, I've been working out like a fiend during this period I've gained 15 pounds (though I believe a little bit of it was actually muscle -- unfortunately, I gained some fat on top of it).