There probably is a wide variation in the pollution/sustainability factor. Perhaps someone else knows.
Apparently, our ancestors ate lots of little sea critters at some periods in evolution. E.g., see this interesting presentation.
If you can get good, sustainably caught or farmed critters, and you can afford them, go for it.