The breakdown of casein in the body produces a release of histamines. Releasing histamines produces an increase in mucus and inflammation. As an asthma/allergy sufferer, I can assure you, I'm not imagining my reactions. There is a biological response to consuming milk. I could entertain the possibility that it affects each person differently.
Lack of sleep gets my vote.