Do more walking on the swim shoes, remove the insole for a closer to the ground feel and take your socks off, take it slow. I run on those, don't want to shell out the money for Vibrams and also I don't think I could wear them, two of my toes on both feet are kinda fused together halfway (it doesn't look weird, most people don't even notice) but since Vibrams separate the toes I could see that being an issue with me wearing them.
I've found jogging/sprinting in a more barefoot way gives you AWESOME calf muscles, soreness is very common at first but after a while they'll start looking real nice. My Muaythai buddies were interested in my "secret" since a lot of the kicking and kneeing techniques require you to get up on your toes and flex your calves.
I saw a guy at my college wearing all black Vibrams with jeans on, it looked rad! I bet not many noticed either ...
I used to seriously post here, now I prefer to troll.