What I did, is I never told any family or friends about doing IVF. They don't know. I wasn't interested in anyone's opinions about it! Only our doctors know. We have one child.
We had quite a few friends who don't want, or don't like children, and couldn't understand why we wanted to have them. They were unsupportive. I wouldn't want their opinions on IVF, because I suspect that would give them more ammunition to judge us. And once you tell, you can't "untell".
Some people will tell you to just ignore them, but I don't agree with ignoring people who are being abusive. I think they should be confronted about their negative remarks, and told to knock it off.