I hear you too!
However, (and you can shoot me if you want!) I don't think there's anything wrong with the occasional non-primal off-piste indulgence, as long as it remains that, and more importantly, that it's a conscious choice, made with understanding and weighing up of the consequences, and not a reaction to stress or cravings.
I had a piece of a lemon polenta cake my mum made the other week - she had made it with love, she was really proud of it because she's just getting the hang of gluten free baking after my coeliac diagnosis last year, so I felt to not try some would have been rejecting of her efforts. I had a small piece, and that was fine. No ill effects, the rest of my day was paleo, and the days after that.
However, sometimes an occasional indulgence done for the wrong reasons turns into a few days of poor choices, and then I start feeling kak again. It depends very much on the mental state you are in when you make a choice to go off-piste I reckon.
I also feel that for some people (treading carefully here) Paleo/Primal can become mentally a bit less than healthy in terms of building a good positive relationship with food, or even a stick to beat ourselves with. Beating yourself up for an occasional non-primal food item - I don't think it's productive, personally, and actually could lead to unhealthy starve/binge cycles. Which makes it even less healthy than the SAD that we're all trying to get away from.
I think maybe it might be emotionally healthier to accept we live in the real/modern world, and occasionally, there might be cake. Just make a conscious choice to eat it or pass, and make it one non-primal choice and not several.