For a while I thought I had a higher antral count when off of caffeine, but then around Christmas after drinking a TON of eggnog lattes I had one of my highest. I say don't beat yourself up. I have seen plenty of women get a surprise BFP after failed IVF's and long periods of doing "everything right". Sounds like you are being sensible. I do think having vitamin D and thyroid levels checked is important and medicating if out of range is a good thing. It didn't help me get pg., but I'm glad I have those levels in the normal range though.
xoxo, Tracy