The wine might be a piece of it, if you are indeed drinking a fair amount. The issue there, as I understand it, is that calories from alcohol are burned preferentially. This means that any calories you take in from wine will be used first, sidelining the burning of other ingested calories or stored calories (i.e., fat).
I don't remember what you posted regarding dairy consumption, but you might also reconsider your approach to that. Everyone seems to have a different response to it, but I think it's generally accepted that dairy creates a surprisingly strong insulin response and, on top of that, there's no real reason one would need dairy, so it might be another place to check.
The final thing I'll say is that, while it's generally talked about that someone who carries a lot of extra fat is unhealthy, it probably bears repeating that very *low* fat isn't really healthy either. In other words, part of your journey might be accepting where your body wants to fall in terms of composition once your metabolism is working properly.
“If you want to build a ship, don't drum up people together to collect wood and don't assign them tasks and work, but rather teach them to long for the endless immensity of the sea” -- Antoine de Saint-Exupery