And don't feel bad that you can't be like that right now... you've been poked and prodded in completely the wrong way IMO. He may not KNOW how manipulative he's being, but it doesn't change the fact that you feel completely manipulated by what he is doing... and you've made him aware of that!
IMO don't ever do something just because you think that's what another person wants. If you would want to be with a woman on your own alone without husband within a million miles, just don't. It has to be your idea first and foremost, not something done to please... that would go sideways so fast you don't even want to know. I've seen it. It just doesn't work. Resentment on the grandest scale.