I do pretty much the same thing. See the VA once a year (or so), and use Tricare the rest of the time. However, I will turn 65 in February, so I will be transitioning to Medicare A and B, and Tricare for Life (TFL). Other vets I have talked with locally have their healthcare set up this way and have no issues. I am 60 percent VA disabled (sleep apnea and tinnitus), so keeping the VA in the loop is good. They also have free hearing aids, which is fantastic.
So, we'll see how the transition to Medicare goes soon. I'm sure it will be fine, but I just hate having to change something that is working just fine.