What's your most frustrating thing about being paleo?

Mine is definitely the fact I can't eat sugar. I love it to death. I don't think I can totally eliminate it from my life, though I've been trying.

What about you guys? And how do you manage to handle it?