I have been barefoot 80% of the time for over 2 years. And I do wear the VFF when I have to wear shoes, which outside of work, is rare. City streets or stores, I'm usually barefoot.
Once you go barefoot, you never go back. While better than regular shoes, even the minimal shoes cut off the sensory input that you get from going barefoot. This sensory input is a safety feature built into your feet that helps prevent things like ankle sprains.
There is a ton of information, including medical sturdies, here: Society for Barefoot Living Home Page