I can understand why you feel that way--I did too for a long time. The psychologist said something that really made sense to us, and that was that it is about the story of your child's life, and no one needs to know except the child. And, it is the child's decision if and when family members and other people should know, not yours. That helped me when I felt I wasn't being honest with my sister, who I am very close to--and she doesn't know.
In my case, we plan to tell our DD for the reason above, and also because she has a different blood type that can't be gotten from the two of us and that could be a way for her to find out without us telling her, which I think would be very bad to have happen. Better to hear it from us and so we have been telling her, but she's only 1 at the moment and doesn't understand. I think it is more for me to practice the story.
Bottom line is that you have to do what is comfortable and right for you and your family.