There's been that entire issue about Dumbledore "invoking an ancient magic" to protect Harry, with the end result that while he lives with the Dursleys, Voldemort can't touch him. That's why they don't kick the Dursleys out and spirit him off to a loving family.