I agree with Rob's SO.
I would say simply do not take the test. You can refuse any and all medical tests during pregnancy, if you wish. Your doctor can also 'fire' you, but ultimately, you are the person making medical decisions, and if you feel the test is unnecessary, then don't take it.
Many midwives (though not necessarily doctors) will allow you to do glucose readings throughout the day over several days (which you record), as well as doing a reading while you are in for your appointment, to follow how your glucose levels are doing and to help diagnose GD (which I do believe exists, but I do not believe that this test is a good indicator at all -- in fact, I think it's a ridiculous test).
I refused the test. IN fact, I refused all medical care during my pregnancy and birth (my choice, btw, and I'm not wholly advocating it. I'm just saying that it can be done). You are free to choose to not take this test and ask your doctor or midwife for alternatives.
Good luck! And congratulations on your pregnancy!