Probably not going to kill you. My problem with gluten is not that it's a non optimal food, just that it's SO addictive. As a personal thing, the "every now and then" approach doesn't work. Like sugar and coffee, if you go cold turkey, you will lose the taste for it. Other things will start tasting better. That's because sugar, coffee and gluten have been crowding them out. That's why they are addictive. Remember, gluten in the the form of bread is just another form of sugar. So, when you want bread or cake, it's the sugar talking, not you. Eliminate it completely and you eliminate the desire. That's been my experience.
But if that's not a problem for you and you can eat these foods infrequently, I doubt it's going to have severe ramifications on your health.