Here are a couple of links that might help:
Primal Blueprint Shopping List | Mark's Daily Apple
Also, the "Start Here" link at the top of the page.
I think the learning which stores had what things at what prices once I switched to primal drove me batshit in the beginning. One thing that helps, is to primalize what you used to eat. So, if you love burgers, you can still have them, but no buns. Some folks like lettuce as a wrap, some people make paleo breads, some just eat burgers with a knife and fork. (The burger is just an example.)
Potatoes are fine, by the way. If for example you're trying to stay under 100 grams of carbs per day, a potato has about 4.5 grams of carb per ounce. So a 10 oz potato - which is a nice size amount - you'd get 45 grams there. Still 55 grams left from other sources. If you're not buying organic potatoes, you might want to peel them before cooking.
"Right is right, even if no one is doing it; wrong is wrong, even if everyone is doing it." - St. Augustine