- Try cutting out dairy. Milk is very anabolic and insulinogenic.
- Are you getting enough Vitamin D?
- How are your sleep patterns/quality?
- Try to move around more during your day (once every hour do some air squats, a lap around the office, a couple burpees)
- Have you tried skipping breakfast/ intermittent fasting?

4 days a week of Crossfit maybe a little overkill for your level. I'd focus on less glycogen crushing work outs (ie crossfit), and more on pure strength. I'd suggest 2 days a week of strength oriented training (google starting strength) and then 1 day of short, high intensity work (sprints, complexes, crossfit wod, time <15min)