Of course it's the nuts! You write that you have a "couple of handfuls" 2-3 times a day? Try figuring out the total calories you're getting from those nuts, and I guarantee you'll be amazed. I've lost a significant amount of weight, but even now in maintenance, I count out my nuts into small portions in order to control. Nuts lead to 'mindless eating' which is death to weight loss.
If you're trying to lose, I'd drop the nuts completely. Have good meals with sufficient food, and you shouldn't need to snack at all, but if you do, try something like raw veggies.