i think you're the only one who can answer this question effectively, since it's your body and how it reacts to things. my suggestion is to just figure it out. try some non-primal food at a single meal and see how you do. a few days or a week later, try a SAD day...try a couple days in a row, and just be aware of how your body reacts.
is that going to kill you? nope. is it going to make you gain 40 pounds of fat? nope. there's nothing wrong with stepping away from primal once in a while...as long as it's not that often and you are sure to come back.