Choosing Private Doctor over VA healthcare

I prefer going to my private doctor over using VA healthcare doctors. I have heard negative things about VA Doctors. I was wondering if this would cause issues in the future with lowering my rating? If so is there anyway I can give the VA my medical records & should I let the VA know I am using a private Doctor? TIA