I've noticed this too.
My best guess is that it has something to do with stress. Even with my GPS up, I'm still ticking down the miles till I have to turn, trying to read the road signs to make sure I'm turning at the right place, and dealing with unfamiliar traffic patterns (is this a turn only lane!?!).
On the way back I can go, "oh, this road takes me straight to the highway" and take the GPS down and cruise with half my brain.