AFAICT, it seems likely that a good supply of minerals (including but not limited to calcium) in the diet are important for proper growth — and not just height. I have a kind of sense — can put it no higher than that — that where there is both adequate protein (not massive amounts just enough) and plenty of calories there will be height. (C.f. modern Western countries).
BUT … Whether there's the width with it, too — and, more generally, ideal growth all round — is another matter.
Weston Price is particularly interesting on the New Zealand Maori. Observations at the time of early contact seem to indicate that these people, before contact with modern society and modern food, were in a near perfect state of physical health:
Nutrition and Physical Degeneration: Chapter 12The reputation of the Maori people for splendid physiques has placed them on a pedestal of perfection.
It seems to have been partly the high intake of seafood — a good source of minerals — that was responsible.
And when the bones grow right, it seems likely the soft-tissues (as with the eye) are not impacted on adversely. Again on the NZ Maori but stated in the chapter on the Aboriginal Australians:
Nutrition and Physical Degeneration: Chapter 10The marvelous vision of these primitive people is illustrated by the fact that they can see many stars that our race cannot see. In this connection it is authoritatively recorded regarding the Maori of New Zealand that they can see the satellites of Jupiter which are only visible to the white man's eye with the aid of telescopes. These people prove that they can see the satellites by telling the man at the telescope when the eclipse of one of the stars occurs. It is said of these primitive Aborigines of Australia that they can see animals moving at a distance of a mile which ordinary white people can not see at all.
As always with Price, one is impressed with his field observations, which seem so suggestive, and equally dismayed by the fact that all this stuff that he was bringing to light seems to have fallen on deaf ears and is still mostly unknown. Was he talking to a brick wall?
As for milk — well, yeah, difficult to tell what's going on there. Herders seem often to have been very tall. There are African peoples one could point to in this respect. Maybe this might have something to do with the blood-sugar spiking effects of milk??
Again, maybe it's not just how tall you are but how well-balanced your physique is overall that matters. Price himself notes of the Dinka that:
Nutrition and Physical Degeneration: Chapter 9This tribe lives on the Nile. Its members are not as tall as the Neurs [Nuer]. They are physically better proportioned and have greater strength. They use fish from the Nile ...