Don't fear the carbs. I have found that neglecting them really was affecting my performance and even fat loss. Yes, you can lose fat eating high amounts of carbs.
My workout days I eat 200-300g carbs, in all forms - Fruit, sweet potatoes, molasses, honey, potatoes, etc. I try to mix it up so that I get various types of carbs based on what I am eating, whether it's with a meal, snack or a smoothie.