Its all mental unless you fast for 3 days straight. I have been IF'ing for at least 16 hours/day and have thrown in a few 40 hour fasts over the last 8 months and all I have done is actually build muscle. You won't go catabolic if you don't eat for 6 hours (or whatever). I know a few bodybuilders that live by the Warrior Diet (which is eating once per day essentially) and they don't fret over losing mass. Actually, a LOT of bigger guys that I have worked with usually only eat once or twice per day just because it feels "right" to them.
People too weak to follow their own dreams will always try to discourage others.