You're argument does not hold water
....."and giving the assignors a wider variety of officials to pick from."
The assignors DO get a wider variety each year. They get a flock of new referees each and every season. Yet, you and I both agree that they lose more than just a few of the experienced referees.....some very good refs do not return.
The assignor DOES get a wider variety of referees to choose from. Yet year after year, the same guys get the bulk of the games, and many others quit solely because they know they are not part of that little clique.
It's not the rowdy fans that entice the bulk of your referees to quit. It's your own politics and favoritism, and some very good refs decide they want no longer want to be a part of your cute little club.