Generator gtk/SDL was written after the svgalib version and tcl/tk versions which both used the unix OSS sound routines.
Maybe someone would like to have a go at doing what you suggest - there is one complexity - it doesn't look as if you can read the state of the SDL output buffer which means Generator would not be able to do dynamic frame skipping based on sound output - perhaps calling gettimeofday() to get sub-second accuracy would do it...