2016 California Autonomous vehicle disengagement reports

[LosAltosChuck]

Just catching up to this thread. It should be noted that the DMV report only covers disengagements on public CA roads. Tesla tests in other places, including non-public tracks where such reporting is not required. For example, substantial testing is done by them and others at Moffett Field in Mountain View on the runway and taxiways. (I've shared the runway with them during testing, so can state this with certainty.)

To use that report as a measure of Tesla's (or any other manufacturer's) testing program would be highly misleading.

 

[zmarty]

Just catching up to this thread. It should be noted that the DMV report only covers disengagements on public CA roads. Tesla tests in other places, including non-public tracks where such reporting is not required. For example, substantial testing is done by them and others at Moffett Field in Mountain View on the runway and taxiways. (I've shared the runway with them during testing, so can state this with certainty.)

To use that report as a measure of Tesla's (or any other manufacturer's) testing program would be highly misleading.

I get what you are saying, but I disagree a little. In the end real life testing is what makes or breaks a system. I am looking forward to the 2017 report coming up in 2018. I am very curious how much testing on public roads they have done this year. I currently do not feel so good about the entire Autopilot program at Tesla, I think they are struggling and nowhere near the competition. Hopefully throwing money at the problem will help, but I doubt it.

 

[LosAltosChuck]

I get what you are saying, but I disagree a little. In the end real life testing is what makes or breaks a system. I am looking forward to the 2017 report coming up in 2018. I am very curious how much testing on public roads they have done this year. I currently do not feel so good about the entire Autopilot program at Tesla, I think they are struggling and nowhere near the competition. Hopefully throwing money at the problem will help, but I doubt it.

My point isn't that track testing is a substitute. It is that the DMV report doesn't come close to being a measure of the amount of testing any of the companies have done. Testing is done on local tracks (eg, Moffett), on non-CA roads (eg, Nevada), and a lot is done by simulation. Bay Area roads are convenient, but are also not the most appropriate when testing new systems. A previous commenter suggested that Tesla was in the dark ages compared to Google, providing a link to simulation work being done at Google. Nothing at the link made any suggestion that Tesla was *not* doing its own simulation work. I'm certain they are, they just haven't been public about what they are doing.

All I'm suggesting is that there is not nearly enough public information available from any of the companies to compare testing and validation efforts, or to make a judgement about Tesla. There are hints and suggestions, but they don't establish fact.

I drive my AP2 equipped MS in Autopilot mode as much as possible, but I do so knowing it is "beta" software, and I am thus a tester. I treat it with the same respect I treat the autonomous technologies I build. AP2 fails on me frequently enough that I've learned many of its weaknesses, and thus am training myself as I'm sure the data I'm generating is contributing to training the system.

The one odd thing that occurs to me is this: since this is beta software, and we are all, thus, testers, why doesn't Tesla have to report the disengagements *we* experience on California roads?

 

[zmarty]

The one odd thing that occurs to me is this: since this is beta software, and we are all, thus, testers, why doesn't Tesla have to report the disengagements *we* experience on California roads?

Probably because AP2 does not fall under the definition of "self driving" as per the DMV. It's just a driver aid.

 

[zmarty]

Also, I made the stone ages comment. I find it extremely disingenuous that the only public testing they have done last year was for that video, and they had to try the same route for days to get decent footage and they disengaged hundreds of times. Google is at 5,000+ miles between disengagements. Tesla is at 3+ miles.

I want FSD and I want it now

 

[LosAltosChuck]

Also, I made the stone ages comment. I find it extremely disingenuous that the only public testing they have done last year was for that video, and they had to try the same route for days to get decent footage and they disengaged hundreds of times. Google is at 5,000+ miles between disengagements. Tesla is at 3+ miles.

I want FSD and I want it now

But you have no evidence that it was the only public testing. The DMV report reflects only the public testing done on open CA roads. They wouldn't report disengagements at, say, Gomentum Station in Concord, where there are rumored to be testing, or in Nevada, where they have permits to test.

I'd like to see more transparency, of course. I'd really like to know when I'm going to see substantial improvement, even in AP2. Mysterious, random updates with little information on content bothers me.

 

[zmarty]

But you have no evidence that it was the only public testing. The DMV report reflects only the public testing done on open CA roads. They wouldn't report disengagements at, say, Gomentum Station in Concord, where there are rumored to be testing, or in Nevada, where they have permits to test.

I'd like to see more transparency, of course. I'd really like to know when I'm going to see substantial improvement, even in AP2. Mysterious, random updates with little information on content bothers me.

Sure, but if they tested 1 million miles on private roads, and then they came on public roads and had hundreds of disengagements in 3 days like they did, I would still say they are way behind. Based on the disengagement report, this is a fail either way (if they tested or not on private roads beforehand). In the end real life is what matters, so let's wait for the 2017 report.

 

[AnxietyRanger]

All I'm suggesting is that there is not nearly enough public information available from any of the companies to compare testing and validation efforts, or to make a judgement about Tesla. There are hints and suggestions, but they don't establish fact.

I think there is ample evidence to know Tesla has at least been at this FSD game a much shorter time than major competition. While this in itself is no proof of being behind, combined with the rest of what we can see, it does raise legitimate concerns and questions.

Of course we do know Tesla is trying to do this the easy way, i.e. with minimal sensor fusion (while competition looks to mesh 360 radar and 360 lidar with vision), which might allow them moving quicker in some regards. With less sure results, though...