I'm pretty much the same way. In part because at work they supply us with lunches. While I have tried, numerous times, to get just meat and veggies, I still end up with meat in gravy, etc. The bad stuff like breads and sugar filled deserts, I either trash or give away. I'm out in the middle of the desert so my shopping choices are a bit limited and I have to do the "meat is meat" mentality since either a) it is not grass fed or b) it is but I don't read Arabic so I just deal with it.
Same when I'm at home in the US. We have a large family and I'm the only PB one so I have to make allowances but like you said, the amount of "bad stuff" I eat is so small that I don't lose any sleep over it.