Wow. Just wow.
Don't have time to write much but if you really want to not ruffle feathers and to protect your boys (duh!) and also not turn your back on what is really just a little boy in need, you could make sure that you supervise their play when he is over. A pia for you, but it was not so long ago that you (or dh) supervised all play to keep your little ones safe due to their ages...
I wouldn't let my children be alone with him but I also don't know that I'd have it in me to either shut out a little kid or deal with the potential neighborhood fall out from doing so. So, maybe grab your laundry folding or your lap top and sit yourself down in the playroom while they play?
me:smc (single mom by choice)
Dd: Conceived when I was 42 after 2 years ttc. Conceived on 6th IVF cycle after 2 bfn's and 3 m/cs.