Only about 81% of the U.S. population are in urban areas. Only 71% are in urban areas with a population over 50,000 people.
So somewhere between 19% and 29% of the U.S. population are special, and have legitimate needs that can't plausibly be met with robotaxis, because they live in areas that aren't dense enough or populous enough to have taxi or ride share services, much less robotaxi services.
Remember that for every New York City, there are a hundred Bucksnort, TNs.
Also, you're ignoring commuters entirely. The longer the commute, the less likely it is that a robotaxi will be able to do something useful while waiting for you to return, and the more your trip will cost. In dense urban areas with people commuting single-digit miles, that probably isn't an issue, but even in semi-suburban areas like most of the Bay Area, it can be a problem.
If they have a car, why will they bother paying more money for the service that's less convenient than having their own car there? Just to not have to drive? I mean for some people, maybe, but I think a lot of people will say, "Yeah, or I could spend $45k on a Model 3 and have the best of both worlds."
If you can actually make it that cheap, it might be the majority of people who live in areas that can practically be served by such a service. That still leaves out almost a third of the country, though.