Here is a very revealing article on where Cruise is at, in terms of FSD. Industry analysts rank Cruise only behind Waymo when it comes to FSD.
Cruise’s Secret ‘Apollo’ Robotaxi Milestones (Paywall see the comments)
Here is the most damning piece. Cruise is expected to be just 10% of where Human drivers are on average. So, they have one more 9 to go before reaching human level.
They also highlight that even after years of progress, Cruise still doesn’t expect its automated driving system to match human performance. Instead, based on the company’s 2017 calculations, the system it has at the end of this year is expected to be 5% to 11% as safe as human-level driving, in terms of the frequency of crashes.
So, what happened ?
What is clear from the documents is that Cruise misjudged how quickly its software would reach the final milestone, known as Apollo. Engineers led by then-CEO Kyle Vogt believed that partly by hiring more people Cruise would be able to improve its software exponentially. Instead, Cruise and other major programs in the industry improved quickly at first, but then saw diminishing returns from software development.
They also drive very few miles.
With more employees and better simulations to test early versions of road software, Cruise leaders said they believed the company would see “continued exponential increases in miles between” incidents. They added that “as we near the final milestone, we will be driving nearly one million miles per month across our validation fleet” and that doing so would help identify and fix bugs frequently. That hasn’t turned out to be the case, either. Cruise said it drove less than half a million miles during the 12 months that ended in November of last year, and the software still has bugs.
They face difficulty in proving their FSD is better than humans too, whenever it gets there. This is one place where definitely Tesla is in a better position. Tesla fleet can drive 40 million miles every month
Any reduction in the number of miles Cruise drives will make it harder for the company to compare its technology to regular cars. As it is, Cruise vehicles would have to drive 43 million miles for Cruise to believe with 80% confidence that its vehicles are marginally safer than human-driven cars—defined as causing 15% fewer crashes (of a Level 1, 2 and 3 variety), the documents show. Driving that many miles “in a reasonable amount of time” would be “infeasible,” the documents say.
There is also this very useful metric in terms of # of crashes for avg human drivers that Cruise is using as the base. So, as I suspected when we take all accidents, including scraping your tire on the curb, in to account, the number of accidents go up to about 1 every 10,000 miles or 99.99 % (4 nines). When only major accidents are taken it is 1 every 282k miles or 99.9996% (6 nines).
Miles between incidents for regular cars (according to Cruise’s modified version of Virginia Tech study, which removes crashes on highways and rural roads):
Miles between Level 1 incidents: 282,485
Miles between Level 1 and 2 incidents: 99,404
Miles between Level 1, 2 and 3 incidents: 20,661
Miles between Level 1, 2, 3 and 4 incidents: 9,218
ps :
Currently Cruise has passed milestone #2. So, they currently have a major crash every 3 months, instead of human average of once every 25 years ! They do not even have targets for smaller crashes.
Titan
Safety critical events: 3,000 miles per L1 event
Mission failures: 30 miles per event
Total miles required: 9,000 miles
Ride quality: 20% better than Sputnik
In early 2017 Cruise was doing this. So, we can compare the progress and see what they have done in 2 years.
Over 600 miles of testing in early 2017, Cruise vehicles had 27 “safety critical events”—including nine severe, or “L1,” events—ranging from an inability to recognize pedestrians, navigation issues, challenges in predicting the movement of objects and two “pedestrian politeness” incidents in which vehicles did not give pedestrians enough space while moving around them.
So, from 9 L1 events in 600 miles (or one every 67 miles) they have come to 1 every 3,000 miles. So, they gained nearly 2 Nines in 2 years (98.50% to 99.97%). This is quite good - but the problem is they have hit a plateau and not improving much now.
So, the question to Tesla is - how are they different ? Is it just access to training data or is there something else ?
