Yes the amount of carbs matter. The larger the insulin spike, the more likely that excess carbs or even 'innocent bystanders' like fat get stored as fat.
If you look at what is widely known to be fattening foods, there are two common themes: foods with lots of simple carbs and fat (pizza, ice cream), and foods with lots and lots of simple carbs (pasta). Try to avoid eating too many carbs with fat, or lots and lots of carbs by themselves.
EDIT: Just to expand a bit. Insulin isn't like an on/off switch, it's always there after a meal to a degree. After a fat only meal there is a very small amount of insulin, after a simple carb only meal there's a ton of it. The more carbs, the more insulin.