As long as you are the only one using the doll, you clean it well before you use it the first time and after you use it every other time AND dry it - it is perfectly safe - unless you have allergies to latex, plastic, rubber - depending on the doll.
If you share the doll with someone else and you are sure that both of you are only having sex with dolls it is just as safe. But if either of you also have sex with people, to be safe, use condoms.
The inside parts of the doll if not washed and dried well will possibily grow bacteria from the moisture.. That bacteria does not belong anywhere near your body or you could get an infection. If someone else uses the doll that has any kind of infection and leaves some of their body fluids inside the doll - the moisture of the doll could keep the infection alive for a long time and then pass iton to you.
So yes, having sex with a sex doll is safe, unless you don't keep it clean.