diplomat33
Average guy who loves autonomous vehicles
Just quickly downloading and analyzing the California DMV disengagement reports for 2021 here: Disengagement Reports - California DMV
I don't think we can draw any conclusions in either direction. But I put some effort into this so I might as well share the graphs. Ultimately the data just isn't detailed enough to really dig into it very far.
First, miles per disengagement are indeed increasing as total miles are driven.
View attachment 786748
And the image is somewhat more interesting when you break it down by VIN. On average, Waymo reported 47,964 miles per disengagement for November 2021 on average, but the worst vehicle in the fleet drove 440 miles before its first disengagement that month, while the best vehicle in the fleet drove 3,344 miles without a single disengagement that month. (Points at the top of the chart represent Inf miles per disengagement).
View attachment 786747
It's a pretty extreme variance in disengagements per vehicle. So they could have a steady fleet of consumer-facing vehicles which they keep to a limited number of streets to keep public disengagements down, and a more disengagement-heavy testing fleet. Here's how that non-disengagement fleet has grown over the year:
View attachment 786752
And for the most part, miles driven is positively correlated with disengagements. Although if you plot all the VIN-months out in boxplots, it recognizes all of those 0 disengagement/high-mileage months as outliers:
View attachment 786753
And then I thought it would be interesting to break disengagements per month down into whether the disengagement was initiated by a safety-driver or by the Waymo-driver. And note, in this data, Waymo did not report a single disengagement from a driverless vehicle. So in all of these cases a safety-driver was behind the wheel:
View attachment 786745
That spike in system-initiated disengagements from April to June is a little odd. Would be interesting to see if that correlates with any expansion in the geofence, but for now it seems unexplained why it rose so dramatically and then dropped back, when driver-based disengagements stayed relatively steady.
And then lastly I'll just point out how lacking the data is on road type. They don't break miles down into Street/Freeway/Highway, but they do break disengagements down by that, so this is the best view we have of that slice:
View attachment 786744
Might be tempting to say the April-June disengagement spike was caused by Waymo testing more freeways and highways, but we can't really tell because we don't have miles per road type, just disengagements.
WOW!!! Thanks for putting in all that work to create such informative and detailed graphs and really breaking down the data!!
A few observations:
The high system initiated disengagements might be caused by Waymo testing the 5th Gen. If you look at the disengagements by cause, Waymo did experience a lot of disengagements caused by "system diagnostic" type warnings. Remember that 2021 was the year, Waymo really focused heavily on validating the 5th Gen hardware & software.
Waymo has to report miles and disengagements from all their autonomous vehicles. So the data will include all their autonomous driving, both the internal testing and the trusted tester program. It is possible the trusted tester program may have skewed the disengagement rate since Waymo is more cautious with public rides.
It is not surprising that most of Waymo's miles are city miles, with very few highway miles, since Waymo is focused primarily on their ride-hailing service in SF which is primarily on city streets.
Waymo says that they are ready to remove the safety driver in SF. So this would suggest that Waymo has achieved a disengagement rate good enough in that geofenced area in SF. Waymo would not remove the safety driver unless the Waymo Driver met a very strict standard based on their safety methodology. It would also make sense if we look at your first graph, where Waymo had a disengagement rate of nearly 50,000 miles per disengagement in Nov 21 when they were doing almost 500,000 miles per month. These miles are probably the trusted tester program in the geofenced area. So in that geofenced area, Waymo was doing nearly 50,000 miles per disengagement in Nov, perhaps even better in early 2022. They got the Waymo Driver to be good enough for driverless in that limited ODD.
Last edited: