you would get more helpful answers over on the pink board. Many here are still struggling to have children.
Call estranged neighbor lady and in the sweetest possible way ask her why you are no longer friends or why she doesn't stop by and see the kids? Tell her you misss seeing her and you have no problem with her visiting with you and the baby. She may have her own issues that you are not even aware of. If she isn't straight with you/won't answer/accept your offer of renewed friendship then when her daughter wants to take the baby to see her tell the girl to have her mom come over instead (again, as politely and nonchalantly as possible).
You also might start looking for new sitters now because you may want to back away from the crazy here too.