The cheap, extruded stuff can be painful to work on because it's too soft!
6061 or 6080 is good stuff.
6063 is a not so much fun.
And without coolant...
What is the runout on your spindle?
And if you push on the head, how much can you deflect it 'easily'?
you'll have to run it with lower spindle RPMs so that each piece of swarf is larger.
Because each cut will heat the material, and getting rid of large chunks in each cut will remove most of the heated material. Very small cuts that just 'shaves' a very thin slice of the material will leave too much heat.
Is the 1/8 bit that thickness all the way, or does it have a thicker shank?
Going for a thicker shank, with a short as possible cutting length will keep it stiffer and slightly less prone to breaking.
You think 20"/Minute is bad...
I'm down to 250mm/minute(about 10") with a cut no deeper than 0.1mm on each pass, and that's on the 6080 and with coolant. but then again, my ShapeOko(first gen) isn't really all that stiff.
(Need to cut some 12mm thick aluminium parts to make it stiff enough to cut aluminium at a decent speed... )