I'm not surprised that the software when "boom" on you. The fact that Vdub works with your card proves that its not hardware failure.
Try uninstalling the software again and reboot. Now re-load your ATI drivers, reboot. Reload MMC7.7 and you guessed it....reboot.
For a detailed way to uninstall, try this link:
http://www.dvdrhelp.com/ATI_AIW_Guide.htm
Its a little dated, as it talks about MMC 7.1, but the uninstall instructions will help.
cheers
gammer