One of the biggest frustrations I have with driving right now is the divide between those who drive dangerously cautious to those who drive recklessly. How they drive is dependent not on the rules of the road, but on their own personality/style.
To a large degree driving is a social activity. It's full of being aware of others, and negotiations between people. It's having instincts about what someone is going to do before they do it.
How can you teach a machine to drive safely in a world where it's blind to both of those thing?
The answer to that is you likely don't. That's not useful information to it long term.
Lots of drivers are on the road that wouldn't be there if we had autonomous cars. As we have more ways of getting from Point A to Point B means we can toughen the requirements for earning a licence, and increasing the requirements for keeping the license. I don't see any other way of combating the level of disregard for other people's lives that drivers on the road currently have. It's so bad that fatalities are up in recent years even though we have safer cars than ever before.
We also have to fix an infrastructure problem where we have roads that are absolutely dangerous whether you're a human driver or autonomous driver.
We won't see full level 5 self-driving cars suddenly appear. Instead we're going to see autonomous driving services like waymo has over more areas where certain roads are blacklisted. As time goes on more, and more roads will be allowed.
Accidents will happen and we'll adapt as we go. The promise of autonomous cars is you only have to teach it once. Where with a human you have to teach all the humans.
At some point it's going to come down to how tolerant people are to deaths as a result of autonomous driving. Some people won't accept even not at fault fatalities (as if the car could do magic), and others are of the "as long as it's just better than a human mindset. I'm somewhere in the middle where I expect a >5 times safer on average from an autonomous car. I do because I think there is a lot of low hanging fruit that won't be grabbed if we make it too lenient.