Primitive man would have exploited roots and tubers. In fact, he would have exploited just about anything edible there is. Plains Indians ate "prairie turnip", collected by the women with digging sticks, with their buffalo.
The objection against potatoes would seem to be that they're pretty high in carbohydrate, and most modern people eat too much carbohydrate (and not enough fat) already.
If you're not wildly overweight and you don't want to eat fantastically large amounts is it a problem? I can't see that a few new potatoes or a small baked potato (eating either with plenty of butter) or a handful of potatoes fried in lard, as part of a meal would do anyone any harm. It's not the same as someone who eats five or six at a sitting and not enough of other foods: he would be asking for trouble.