Self-driving is hardest to achieve in towns and cities - I think that's a given.
Yet to achieve a robo-taxi fleet you need to do towns and cities, not just highways! That means coping with lots of things it can't cope with:
roundabouts - but maybe less common in the US.
lanes merging and disapearing
people
roadworks
vandalism and theft
giving way to ambulance and police cars
pedestrians waving you out
oh, and legislation, and being so perfect that you can defeat all the $100m lawsuits where a Model 3 runs over someone's dog or worse. I imagine the oil industry will do a lot to block automation of electric self-driving cars with lots of lobbyists.
Also the public are not ready to believe in self-driving cars. They barely trust Elon. In the UK he's just that bloke who accused that hero caver of being a pedo! He dug himself a hell of a hole with those comments. The caver has been knighted!
Elon's been seeing the behind-the-scenes stuff on Tesla automation and not really thought about legislation.
Roundabouts aren't common in the US, but the local area has become enamored with them. They put in 4 when rebuilding the access to the highway and now they want to put in a couple of roundabouts on the highway further down.
Tesla has combined the idea of electric cars and self driving cars, but the two are separate things. A lot of the legacy car makers are working on self driving ICE, though the economics of self driving ICE for robo-taxis and such get eaten alive by electric cars. I don't see the oil industry fighting automated cars all that much, but anybody who drives for a living will be fighting and in the US the Teamsters (truck drivers) are one of the last strong unions.
I feel the most for ride sharing drivers. They are trying to make ends meet by driving part time, but the company they work for is actively trying to make them unemployed. It's part of their business plan. It would be like YouTube trying to put all their video makers out of work.
Another thing automated cars will have to deal with more is human mischief. People being people, they will do things with automated cars they wouldn't do with human driven cars. Passengers in robo-taxis will be bigger slobs. And on the street people will do things to bully the cars or try to mess with their sensors. Messing with them will range from just trying to do things the car isn't programmed to expect to actual counter measures to mess up the cars.
Some hacker is going to figure out how to jam the lidar/radar on automated cars causing them to get confused. Other tricks would be to generate rogue ultrasonic signals to confuse the car into thinking there are objects near that aren't there. Less sophisticated attacks will be doing things like throwing balloons full of something opaque like paint on the windshields. Those are just the things I could think of off the top of my head. In the real world I'm sure the mischief makers will think of far more than I did.
These navigation systems are going to rely almost exclusively on GPS. What happens if that network is hacked or goes down for some other reason? Some people have raised the warning that a solar flare that hits Earth could disable a significant number of satellites. A severe solar flare in the 1800s affected the telegraph network. That's how we know what a solar flare can do to communication. We haven't had one as big since, but it happened before and will probably happen again.
A moderate solar flare with a direct hit on Earth may not affect many systems on the ground, but could blow out quite a few satellites. Satellites' electronics are hardened to deal with the harsh environments of space, but there are limits to what they can manage.
I remember when a single satellite that processed credit card transactions went down in the 1990s. That was annoying, but for a few weeks using some credit cards was hit or miss. If GPS was disabled or working at reduced capacity, it would be difficult to navigate for those who use GPS systems every day, but with humans driving vehicles, we'd manage to get from point A to point B. With automated vehicles, there is no plan B other than put drivers back in them. Cities would be paralyzed and essentials like food would get to be a problem after a few days.
A large satellite outage would last months if not years.
What becomes a problem is either the thing you never considered or the thing your unprepared for. With the planned, sudden change to the entire transportation network, there are lots of things we need to think through.
