It is your perception that should you decline eating cake whenever it is offered to you, it will destroy the moment, and thus it becomes a self-fulfilling prophecy. Other people will rapidly get used to you making a legitimate-sounding excuse ("Sorry, my doctor told me to cut out all sweets") or even telling them the truth, that you weren't feeling well and you found you improved greatly after changing your diet. It's a matter of priorities to you whether you stick to your guns and train them. After a certain number of gracious refusals they will stop asking you. There are also plenty of times when you can bring snacks to share with others that won't kick you down the road into a binge.