It's most likely due to what I call the ''lean threshold.'' You see, there's a point (a certain body fat percentage) where you start to look lean (around 10% for most men). There's also a point where you start to look fat (around 18-20% for most men). Then in between you have a certain zone where you basically look the same; you aren't lean enough to look defined so you don't really have any muscle separation.
At that point, even if you gain a few pounds of fat, you won't visually see the difference. This is compounded by the fact that you're seeing yourself every day, so you might not notice the small changes in appearance. Most men won't be able to see a visual difference in muscularity between 13 and 16%.
But if you're 200 pounds, going from 13 to 16% body fat can mean a six pound gain in fat!