Cayenne is a little weak for my tastes, but I eat a hell of a lot of habanero and bhut jolokia. I carry powdered habanero everywhere I go and put it on anything that I eat with meat or eggs (which means I use a lot of it two or three times a day).
Some of the people in the Paleo community will not eat them because they fall under the nightshade family along with tomatoes. So, if you're concerned with that then they're not for you.
All the old wives tales about peppers eating out the stomach lining are wrong. If they were true, I'd have none left at all. Some people do get heart burn, but I suffer no ill effects. It's meant to help the metabolism a little, ward off certain types of cancers, and has lots of vitamin C.
The real reason that I eat peppers though is that I love the taste and the heat. Fact is I'm addicted and food doesn't taste right without them now. I even have a tattoo that has peppers as part of it.
So, in my opinion eat all the peppers that you want. Just don't try taking mine away from me.