If your levels are that low, your doc is probably right. There is a problem with your HPTA (google if you need it) and you might not be able to fix it if it has been that low for that long.
Diet, exercise and de-stressing your life can raise it, but if something is broken, you might not be able to fix it with those remedies. I would suggest that if you are going to try and raise it through diet, eat a LOT of saturated fat. If you don't, nothing is going to happen.
There are a lot of guys on TRT and it really isn't that big of a deal.
People too weak to follow their own dreams will always try to discourage others.