There is no reason to avoid it unless you cannot tolerate it. If it does not make you gassy,full of mucus, upset your stomach, or make you feel bad then there is no need to avoid it.
There are better dairy choices than others. I put milk on the VERY bottom (I avoid this completely) but put a good aged cheese very high on the list.
Fermented dairy such as greek yogurt and kefir can be very nutritious.
Enjoy dairy if it works for you.
Oh yeah, I also listen to and read everything Robb Wolf puts out and while he personally doesn't do dairy he has no problem recommending it to people who can tolerate it. He puts it first on the things to eliminate when attempting to determine sensitivities but has no problem with adding it back to diet once its determined to be tolerated well by the body.