If I do a pretty hard session in the gym, whether it's cardio or heavy lifting, what happens if I don't eat anything for hours after I work out?

Because I've read countless times you need to replenish your glycogen stores.... so I'm curious what happens when I don't...