Your doctor is not in charge of your health - you are. You don't have to follow his recommendations - and that's what they are. They are not orders.

It's your body, not his, and your risks to choose.