I bought palm oil in Africa a few years ago (before I started eating properly) and the taste threw me off. It probably didn't help that I used it to make risotto and the rice ended up absorbing the red stuff and tasting like feet.
I'm curious how it'd fare if used in more constructive ways, though.
About environmental concerns: The refined stuff is usually an ingredient in soap and cosmetics, a relative mammoth when compared to the small quantity consumed as food. Some of the cosmetics companies are trying to address sustainability issues, albeit at a slow pace.
Spectrum makes a sustainably produced palm oil: