I don't know what you consider "a lot" of nuts, but depending on how much you are eating you can easily be adding an additional 800+ calories a day to your diet. A 1/4 cup of pecans has 200 calories, so does about 20 macadamia nuts; the tiny 6.25 ounce jar of macadamia nuts in our house contains six servings, 1200 calories. Nuts are one of the easiest things to overeat. I also don't know how much almond butter you eat, but it's similar to peanut butter in calories, around 100 calories a tablespoon.
And yes, calories count. Too many calories, even if they come from "good" food sources, will make you gain weight. Try cutting out the nuts completely for a few weeks and see if your weight starts dropping again.