I haven't tried crossfit, so I can't speak to it personally but it sounds like a bad fit for you (or as mentioned, the wrong gym.) The best exercises and workout plans are the ones that challenge you, make you sweat, but you still enjoy. Long-term you'll simply burn out and hate it. At your fitness level, you should be more concerned about your diet than anything. At the end of the day, if you're eating right (primal) and maintain a calorie deficit, you'll lose weight. If you only walk a bit as your exercise, you need to factor that level of activity into your target for calories per day.
Personally what worked for me is to try different workout plans in the gym until I found something I liked and that challenged me, but I could stick with for 10-12 weeks at a time. While science is science, each person is different and responds best to a different program. When I found what worked for me, the fat starting puddling around my feet!
Keep your diet in check and consider speaking with a trainer at your local gym about beginner programs. The fastest way to lose weight is to add muscle, and in my personal experience, compound movements combined with sprinting and you can supplemented with walks.
GW: 215 w/ <10% BF
Deadlift - 405lb - Goal: 500+
Squat - 375lb - Goal: 400+
Bench - 360lb - Goal: 225x30