I gave up on the driver's side, and moved on over to the passenger side instead (which is the right side of the car for me). Lots more space to work with there!
This is where the wires goes from inside the car through the side of the car and out into the door
I peeled off the electrical tape and then I managed to pry away the rubber boot using a plastic trim removal tool and a lot of force. I accidentally broke off one of the plastic tabs holding it in place. But I don't think I would have been able to remove it without doing that. There's just no way I would have been able to get to the tabs around the back of that thing
Here it is removed
This allowed me to get my phone in there and snap a photo into the hole where that thing should sit. That's when I saw there's a yellow plug in there, to be able to separate the wires in the car from the wires in the door.
There's another rubber boot on the outside of the car body. It's similar to the one on the inside, but oval in shape. It's held in place by four plastic tabs. Two of them I could easily reach, but the two on the back of the thing (towards the front of the car) were impossible to reach with my fingers. But, I could get to them through the hole from the inside. So I managed to depress all four tabs and loosen the boot on the outside as well.
Next step was to loosen all wires inside the door of the car to give me some more wire to play with. Doing that I could get the door-side of the rubber boot that goes between the body and the door out of the way to give more room for the side with the yellow connector. And with that done I could get the connector to the outside of the car so I could unplug it.
And so I can finally route some speaker wires!
So now I just have to figure out how to do this same thing on the driver's side where I can't really get to the rubber boot on the inside of the car.