From the skeletal remains that I've studied, it's pretty obvious that, as a general rule, our brains and bodies are smaller than they were during the Middle and Upper Paleolithic. The brain thing is complicated and has lots of factors. Smaller brain doesn't necessarily equal less intelligence. Our phones are smaller than they used to be, but they're much more complicated. It's a matter of efficiency. We also domesticated ourselves, which leads to smaller brains in just about every animal but doesn't necessarily make them dumber. Our mental capacity hasn't changed in about 200,000 years. But our bodies are definitely punier. Average height is only now starting to approach Paleolithic standards. Their bones were much thicker and denser than ours too, because they lifted heavy things all the time and ate foods conducive to building strong bones. Yes, we have access to more calories now, but they're mostly empty calories. Empty calories will keep you alive but they won't make you bigger or smarter or healthier. What's really important is nutients+calories, which is what was available to our ancestors when they had access to food.
Also, Neanderthals were pretty short. Their bones were even more robust than our ancestors who where their contemporaries, meaning they were short and compact, but very strong. Their brain situation is also very complicated and I won't go into it unless someone asks me to, just remember that brain size (after a certain point) has little to do with intelligence.
These are all things I've learned in studying anthropology and actually looking at the bones, so I don't have a link to give you.