The straight dope?
Gluten is a lot worse than it is made out to be in terms of the general population.
Gluten is not nearly as bad as it's made out to be in Primal/Paleo circles.
Gluten is a potent irritant. It is made to hurt your gut. Even if you are not gluten intolerant - and you probably aren't, at least to a moderately severe degree - it will still take its toll on you when it is a regular part of your diet. However, it is not an evil monster that will kill your pets, beat your children and sleep with your wife. If you are out somewhere and it's worth it to you, I say go for it. The fact that you're posing this question means you clearly are not super intolerant and it won't come back to haunt you once in a blue moon.
What do I consider "rare"? Probably no more than 1-2 gluten heavy meals a month. I don't mean a beer - there's hardly any gluten in beer, and what's there is fermented. I mean a bowl of pasta, a 10" cheesesteak, 4 slices of a large pizza...you know, a typical American wheat-heavy lunch. That being said, I haven't had a slice of pizza in 2 years, a bowl of wheat pasta in about a year and a cheesesteak since Fat Tuesday 2012 (and it was awful, I only ate it because I was on the road with my boss and he bought it for me, what was I supposed to do?). But that's because I truly don't crave bread products, so they're almost never worth it to me.
Choose your battles wisely.
Don't put your trust in anyone on this forum, including me. You are the key to your own success.