i don't think there would be any problem with it. it would certainly make you stronger...not necessarily bigger. you're right that size will come more with food.
20 lbs sounds like a good place to start, but you will probably want more weight not too far down the road. opt for a vest that lets you adjust the weight.