I, too have died to charm deaths in my early days of playing. I, too have learned how to not die to charm.
Why take charm away from illusionists, yet leave it with DKs and vamps if the problem is with charm and trash characters? Don't plenty of DKs trash PK? If anything, perhaps you should merely raise the level that illusionists need to attain before learning charm. Personally, if charm is taken away, I would never play an illusionist.
What good offenses shall illusionists have left if they cannot use charm? Colour spray? Ha, Ha!