You have it definitely harder than me. Once I commandeer and clear a space, hubby does not re-inhabit. He complains that the only spaces left to him are the office and his Mazda. That is not much of a complaint, as the rest of the house is shared by everybody, and/or is special-use (bathrooms, schoolroom, dining room, all the usual suspects) and he still gets his own rumpus room (the office), which is not exactly a wee closet. As the mutually agreed upon division of labor here at Crabbcakes Manor is for me to manage the physical household, and he to pay the bills for it, I get to weed and clean at my discretion - outside of the office and Mazda and Creampuff Car, which I honor.
Yikes, badger - that is really hard. Hubby likes his stuff, but even hubby will clean out and reevaluate every blue moon.
I have a mantra that I have spouted for years... "If I eat right, I feel right. If I feel right, I exercise right. If I exercise right, I think right. If I think right, I eat right..." Phil-SC