For. I have dairy everyday in the form of raw milk and Kerrygold butter. I'm still leaning out without much effort. Would I be leaner without the dairy? Maybe. But the enjoyment I get from the milk and butter is more important to me.
The only way to tell how it's affecting you is to try it for a bit then stop ingesting it for a while and then compare.