It really depends on the person. Some people think they can handle it, others don't. Whey is addictive and there is only so much meat I can eat (that's not true actually I've started eating beef for breakfast, heh) so I would really love it if it didn't pwn me but I have issues with dairy. Some people don't think that they have dairy issues and there still needs to be some conclusive evidence that dairy is necessarily bad for everyone in all cases so it's a gray area. On the one hand, cow's milk is a very recent addition to the human diet and a lot of people are intolerant of it. Just because you don't double over in pain after eating some doesn't mean that it's doing nothing harmful at all. On the other hand, there is little evidence other than anecdotes against something like a whey isolate, though there is plenty of rightful speculation.
Are you going to risk it? If so get something high quality.