It would be one thing if these were his reasons and you had no children. IMHO, those particular reasons are BS when you have a family commitment to a wife and child. They are selfish, and very immature,even for a 32 year old. I know very, very, VERY few couples with children who have that level of passion and romance and excitement and "things in common" that it seems like your DH is looking for. Having things in common is very overrated as well. What does matter is that your VALUES are in sinc, not if you both like basketball, or dancing, or gourmet cooking.
I would be crushed if my DH had said he wanted more kids but not with me because of IF. How truly selfish, given he would get to contribute his own DNA and you would be making the greater sacrifice. I may sound paranoid, but if he cheated before I think there is a decent chance that all this has to do with another woman and not the "reasons" he is giving you. The reasons are just very flimsy.
I left my ex husband (we didn't have any kids) for similar "reasons", and I now very much regret having done that. I was very selfish and immature, but I was also suffering from major depression and so was he. I think our mental states nailed the coffin, and wish we would have waited until that was under control before making any decisions. So...I think in your case (unless he is cheating yet again) holding off on any permanent decisions seems warranted. Will he attend some counseling sessions with you? And if in the end he leaves, you will truly be ok. You will find the strength and the support and maybe even a great guy who would love to have children with you via DE.
Sending a huge hug,