Since I have the same problem, I'd say it's a little bit of both. Physiological if you had recently fed that craving - like if you gave in and had a candy bar the day before. The body thinks it's lacking a nutrient when it's missing sugar after being used to it being provided.
However, it could also be psychological, especially if you've had a habit of eating sweets or other things in the afternoons. In which case you'll need to break the cycle. Take a walk or have a cup of hot tea. Create a new afternoon ritual to replace the old ritual of sugary snacks.
I have found that if I prevent myself from caving, I can go a long time without sugar/snacks/"garbage gut foods". But once I give in, I'm in for the fight of my life all over again. I say break the cycle and create a new afternoon ritual if you can to break the mental ties to foods you are currently craving in the afternoons.