Asking your RE is definitely the way to go. As I understand it, they give valium on transfer day not so much for anxiety as to relax the uterus. (In my case, my mock transfer had been extremely painful, and even with the valium on transfer day there was some discomfort--not to scare you--other people seem to feel nothing.)
Back in December, when we were on a break trying to decide about DE, I was having anxiety and insomnia and my GP prescribed a low dose of klonopin (another benzo), which he said was safe even during pg. I no longer believe him about that. As we started this cycle, I mentioned it to the IVF nurse and asked when I should stop taking it, she said, "umm, now!" I was only using it occasionally, so it wasn't a problem for me. It's just unsettling that there is disagreement among doctors about what is safe. Also, it seems that there are things that are safe after the first trimester but not earlier.