Carbide is brittle as hell and in my experience will break most any time you try and drill by hand (meaning that either the work or the drill is free). I always fixture something in my mill or DP if I'm going to use carbide.
Nothing here looks like it would need carbide, anyway. Carbide is used in a couple of situations - where the material being drilled is high hardness (like 50+ Rockwell C) or when it's abrasive (high silicon cast aluminum). In the latter case you can make do with HSS drills if you're willing to accept lower life. (I'm ignoring the few cases where you simply need the additional rigidity of a carbide drill and hoping you don't point out carbide point drills).
If you have a high hardness material and you want a drill by hand you could go with a cobalt drill, they're pretty tough but not brittle as a carbide drill is.
Draw up what you're doing and post it here in this thread, maybe some fresh ideas can come out of it.