I know you are committed to the money once you pay (same way around here), but if you think of it as sunk costs, then you could still pull him out if it just isn't working. A shame, but no more money than you were going to spend if it were working, if that makes sense.
I"d think ds might benefit, as you said, from time around more kids than his older and younger sibs.
In a fabulous place, I don't think that 3 is too young. I don't think it's mandatory, but I don't think it's bad either. (full disclosure, dd has been in full-time care since 15 months. I don't think she is any worse for the wear, maybe even a bit better, but who knows. She still doesn't like saying goodbye (at age 5), but then she has great days--way more fun than she'd be having at home with me!).
me:smc (single mom by choice)
FSH: 16
Dd: Conceived when I was 42 after 2 years ttc. Conceived on 6th IVF cycle after 2 bfn's and 3 m/cs.