Every "body" is different. Some people just cannot loose weight if they're eating dairy, some find nuts to be like a gateway drug to snacking. I find that myself. A small bowl of almonds leads to chocolate, leads to wine...then scavenging for other sweet things.
If you have no weight to loose and are in great health, then you likely don't need to worry about any of it. Also, there is some concern with the levels of Omega 6's in some nuts, so some people avoid them for their own health reasons.