The wine might be a piece of it, if you are indeed drinking a fair amount. The issue there, as I understand it, is that calories from alcohol are burned [I]preferentially[/I]. This means that any calories you take in from wine will be used first, sidelining the burning of other ingested calories or stored calories (i.e., fat).
I don't remember what you posted regarding dairy consumption, but you might also reconsider your approach to that. Everyone seems to have a different response to it, but I think it's generally accepted that dairy creates a surprisingly strong insulin response and, on top of that, there's no real reason one would [I]need[/I] dairy, so it might be another place to check.
The final thing I'll say is that, while it's generally talked about that someone who carries a lot of extra fat is unhealthy, it probably bears repeating that very *low* fat isn't really healthy either. In other words, part of your journey might be accepting where your body wants to fall in terms of composition once your metabolism is working properly.