As the author of two monster thread about calorie counting on MDA (for which I got all kinds of flack), I couldn't agree with you more that calories do matter. But it's just not so linear. One day eat x amount less, next day see x amount less on scale. This is an interesting article on "whooshes and stalls" in weight loss that you might enjoy.
Of Whooshes and Squishy Fat | BodyRecomposition - The Home of Lyle McDonald