I completely understand. I only eat fruit 1 day per week because I couldn't control myself if I was eating it daily. I cut out dark chocolate because if I bought a bar I would eat the whole thing in 1 sitting. I got rid of nut butters, because I would eat 1/2 the jar in 1 sitting.
So, I limit myself to some nuts on my daily salad & fruit on fridays (typically a small container of berries and a grapefruit).
However, I compensate by eating more protein and fat. I always eat until I feel fully satisfied and so far I haven't had any binging issues. I also find that if I eat something sour it curbs my sweet cravings. I just have a pickle or two.
"You have succeeded in life when all you really want is only what you really need." ~ Vernon Howard