You'll get good results just sticking with the plan.
That said, some people can handle lots of workout volume well. I have no problem doing a lot of "extra" stuff--hiking, sports, metcon-type workouts, all sorts of playing around, etc.--and it doesn't affect my progress on my lifts. I think a lot of it is because I was always "high volume" with sports growing up, and lifted 6 days per week for years and years. I just hit my lifts 3x per week now, and fill in the other days with whatever I feel like. I take days completely off if I feel tired or don't have time or whatever, but if I want to do something I just do it.
Find what works for you; if you're not making progress dial things down a bit.