In my case, rather than wear Vibrams, I wear sandals. I've made them myself. There are kits out there to make huarache sandals. I don't use the Invisible shoe kits as they are too thin for hiking (great for running in town though). I use Barefoot Ted's thickest Vibram sole material. I hate the whole toe-thong thing so I have my own method of fastening. I stitch together a leather sandal and then stitch the sandal to the sole. I end up with a thin-soled, uncushioned, zero-drop sandal that holds well on my foot without chafing. You can buy almost the same sandal I make from a company called Native Earth.
I have found with the thin, uncushioned, zero-drop sole I get all the barefoot benefits without the pain of gravel. Since my feet are not bound inside a sausage casing, they are cool, they can move freely, and my toes aren't hanging out there ready to have vines get stuck between them or be stubbed. Since they are sandals and my feet are bare, I take a little more care for foot placement than I would in any shoe. It's an unconscious thing. I don't want to cut up my toes or jam my foot between rocks and scrape up my skin. I have found for me this works great.
When I go backpacking I go a little less minimal and wear Chaco sandals with wool socks. So long as the trail isn't overgrown with stuff that'll get stuck in the socks. The Chacos have the freedom of foot of sandals but the contoured foot bed can sometimes feel annoying. They're heavy and it's a real brick of rubber under your foot. Strap on boots. You won't feel anything. I can hike like a maniac in those. I've done several hundred miles of the Pacific Crest in them.
Female, 5'3", 50, Max squat: 202.5lbs. Max deadlift: 225 x 3.