I don't think it's a bad idea and it may actually help your body recover more quickly. Digestion does take up some of your body's resources. The hours that your body doesn't have food in your stomach should be hours that healing is happening faster.
As long as you're getting enough protein you shouldn't lose too much mass. Not working out for long enough will result in some muscle loss in any case though. (muscle is kind of a use it or lose it kind of thing)