Were you using a hand drill? I could see that would be tough with keeping it all straight. Needle nose vise grips and a drill press make a world of difference. I use an old mirror nut to clamp the thing into the drill presses vise. Dremel tool to grind off any exposed knob, then the needle nose vise grips to hold the center. A nice sharp drill bit helps too. 5 minutes to set it all up and 5 seconds to drill .........
Yeah, using a cordless drill. I did have the body of the controller in my bench vise using soft jaws, and that helped a great deal. I have been looking for long deal on a nice drill press for the last year or so. My garage has hit the point where I have to decide what I am willing to give up floor space for, so it will more likely be a benchtop model. I do woodworking and would use it for that and metal fab work.
If there is enough of the stud left it is easier to cut a groove in the top of it and use a flat head screw driver to unscrew it.
The originals seem to be press-fit in place. No amount of turning does much. Fortunately the repops are threaded and will tap into the housing.