Yep swimming does this to me, as does surfing when I can get to cornwall in the summer (or falling in the water alot as it is more correctly described)
I would reccomend working out before a meal you would eat anyway, so you are due to eat anyway, that way you arn't eating an extra meal, and if you are really bothered then work out your calorie burn for your swim and don't eat more than you've burned off (pun warning) then at least you are treading water with the weight loss if that is your goal.
You know all those pictures of Adam and Eve where they have belly button? Think about it..................... take as long as you need........................