Autonomous Car Progress

[electronblue]

Reporters like Amir Efrati have revealed that the disengagements data is almost meaningless as a metric of autonomous driving capability or progress. Each company reports their disengagements differently but the numbers get compared side by side anyway. Companies can choose not to report the vast majority of disengagements.

Yet you continue to quite Elon Musk timelines on Tesla’s autonomous driving without much critique — and indeed without even any disengagement data or other proof they are getting there. How do you explain that?

I sense a discrepancy here. This is what I mean when I say I would welcome more balanced analysis and not something so optimistic for Tesla and pessmistic for the rest.

According to what Elon said in the ARK Invest podcast, the roadmap for Tesla’s Full Self-Driving product looks like this:

Step 1. FSD is a Level 2 system. It covers all driving, so you can drive from your garage to your workplace parking lot and back without ever steering, signalling, braking, or pushing the accelerator. However, it still requires drivers to monitor it and intervene if necessary, like Autopilot. Elon expects development to be done by the end of 2019.

Step 2. FSD becomes a Level 4 system (or possibly Level 5, but by the strict definition of Level 5 that would require activation in every country on Earth, including Vatican City). Human monitoring and intervention is no longer required. Elon guesses this will be ready by the end of 2020.

 

[lunitiks]

OT / probably unrelated, but I just noticed that there's been a change to Model S "refresh" option codes very recently.
Model S refresh "MI04"

 

[cwerdna]

Reporters like Amir Efrati have revealed that the disengagements data is almost meaningless as a metric of autonomous driving capability or progress. Each company reports their disengagements differently but the numbers get compared side by side anyway. Companies can choose not to report the vast majority of disengagements.

Meaningless? Give us a better metrics that being reported at least annually on well-known autonomous driving systems being operated on public roads.

Per https://www.dmv.ca.gov/portal/wcm/c...essAV_Adopted_Regulatory_Text.pdf?MOD=AJPERES

§227.50. Reporting Disengagement of Autonomous Mode.
(a) Upon receipt of a Manufacturer’s Testing Permit or a Manufacturer’s Testing Permit – Driverless Vehicles, a manufacturer shall commence retaining data related to the disengagement of the autonomous mode. For the purposes of this section, “disengagement” means a deactivation of the autonomous mode when a failure of the autonomous technology is detected or
when the safe operation of the vehicle requires that the autonomous vehicle test driver disengage the autonomous mode and take immediate manual control of the vehicle, or in the case of driverless vehicles, when the safety of the vehicle, the occupants of the vehicle, or the public requires that the autonomous technology be deactivated.

There's more stuff afterward about what the must be included in the report.

The # of miles driven, # of disengagements and disengagement rates are obviously not the only metrics but it's a starting point. Where it's being driven can make a huge difference (e.g. Cruise Automation tackling city of SF which is pretty tough) vs. highways w/no traffic, which unfortunately these metrics don't help with.

 

[electronblue]

Give us a better metrics that being reported at least annually on well-known autonomous driving systems being operated on public roads

Elon Musk’s tweets?

 

[strangecosmos]

Say a Waymo or Cruise or Zoox safety driver is driving around, doing testing. Whoops, the vehicle has mistaken a parked car for a car stopped ahead of it. The vehicle is stuck indefinitely and the safety driver takes over. Does the company have to report this disengagement? Nope!

Some companies might decide to report these disengagements, other companies might decide not to. For example, I believe Apple used to report planned disengagements, meaning at the end of a test run when the system would be disengaged, that would get reported. So, you can’t necessarily compare two companies against each other in an apples-to-apples way.

Companies also sometimes change the way they report disengagements from year to year. So, you can’t necessary compare a company’s disengagement rate from one year to the previous year. You could look at Apple’s disengagement rate and think you see marked technological improvement, but it would really be an artificial boost from a change in the reporting methodology.

Amir Efrati has done good reporting on this.

This Jalopnik article gives an example of a Cruise disengagement that wasn’t reported:

https://jalopnik.com/californias-autonomous-car-reports-are-the-best-in-the-1822606953

 

[electronblue]

The thing to remember about @strangecosmos is that he is overly pessmistic about everyone else and overly optimistic about Tesla.

In this case he is implying that disengagement data (which includes autonomous mileage data) is meaningless, while Tesla does not even offer any disengagement or mileage data at all.

The thing is: disengagement data may well be prone to misinterpretatioin but it is still data. It is something we can do a bit of science on. With Tesla it is blind belief at this stage when it comes to their Level 4-5 progress (or even Level 3 progress). We have absolutely no insight into Tesla’s obstacle avoidance other than there is nowhere near a driverless level of it, not in the features department even — so it is not just a question of reliability but about we not even seeing any such features or having any data on them...

I take science.

 

[mongo]

I take science

Science relies on repeatable controlled experiments. Reading the disengagement report multiple times is not science. Nor is total reliance on one non-peer reviewed data set science. Basing things on the lack of data is also not science. Repeatable tests of Teslas showing problems would be science, but you still do not have the current development branch to test.

 

[electronblue]

Science relies on repeatable controlled experiments. Reading the disengagement report multiple times is not science. Nor is total reliance on one non-peer reviewed data set science. Basing things on the lack of data is also not science. Repeatable tests of Teslas showing problems would be science, but you still do not have the current development branch to test.

Of course I use the word science here casually. When we have some data, we can do some observations based on that data. When we have no data, we have nothing. We have much more data about the autonomous features of other manufacturers. For example we have mileage. From Tesla all we have is data from their ADAS and nothing from autonomy since 2016.

For all we know, Tesla has not tested a single mile of Level 3+ autonomy since 2016. We have no data.

 

[lunitiks]

For all we know, Tesla has not tested a single mile of Level 3+ autonomy since 2016.

For all we know, they've got a fully working Level 5 system but push L2 to their customers due to LOS REGULATORS!!!

 

[electronblue]

For all we know, they've got a fully working Level 5 system but push L2 to their customers due to LOS REGULATORS!!!

Now this here would be religion, not science.

 

[mongo]

Of course I use the word science here casually. When we have some data, we can do some observations based on that data. When we have no data, we have nothing. We have much more data about the autonomous features of other manufacturers. For example we have mileage. From Tesla all we have is data from their ADAS and nothing from autonomy since 2016.

For all we know, Tesla has not tested a single mile of Level 3+ autonomy since 2016. We have no data.

Now that I can agree with.

 

[zmarty]

A chaotic market for one sensor stalls self-driving cars | Reuters

With the notable exception of Elon Musk’s Tesla Inc, most automakers have said their self-driving cars will rely on a detection system known as lidar. The state of the art sensors use laser light pulses to render precise images of the environment around the car.

...

Lidar remains a relatively young technology that is still in flux, with bulky electromechanical devices such as Velodyne’s popular rooftop unit rapidly transitioning to newer, more compact and more capable solid-state devices designed to sell for less than $10,000 in limited quantities, and eventually as little as $200 in mass production.

“This requires quantum leaps in innovation in lidar technology,” Thomas Sedran, in charge of evaluating Volkswagen’s autonomous strategy in commercial vehicles, told Reuters Tuesday at the Geneva motor show of the need to cut costs.

 

[cwerdna]

Navigant: GM Cruise, Waymo Lead Tesla And Others On Full Autonomy
Just Three Companies at Top of Navigant’s Autonomous-Vehicle Ranking

 

[cwerdna]

https://techcrunch.com/2019/03/19/w...put-a-lot-more-self-driving-cars-on-the-road/

It has a link to A slashed tire, a pointed gun, bullies on the road: Why do Waymo self-driving vans get so much hate?, which I'm not sure has been posted before.

 

[scottf200]

I think this example is relevant as I believe they are using a single camera lens. comma.ai referenced these guys and displayed their usage in some of their testing. They are using open source software called ORB-SLAM2. I would guess this would be more for HD mapping so you would know all permanent reference data points (signs, trees, fire hydrant, buildings, guard rails, etc, etc) for localization.

Below is one way that cameras are being used for 3D mapping. There are several ways already. Imagine multiple passes. Imagine multiple methods and just syncing the ECEF points to create the 3D world and on-going variations (construction) when the ECEF points deviate.

ORB-SLAM Project Webpage
webdiis.unizar.es/~raulmur/orbslam/
ORB-SLAM is a versatile and accurate SLAM solution for Monocular, Stereo and RGB-D cameras. It is able to compute in real-time the camera trajectory and a sparse 3D reconstruction of the scene in a wide variety of environments, ranging from small hand-held sequences of a desk to a car driven around several city ...

Check out a couple of ORB-SLAM2 example videos. Amazing (!!!) with basic cameras.

Driving and mapping

Walking through hallways and staircases to get a two-floor view

This type of Earth-Centered Earth-Fixed (ECEF) Cartesian coordinate points mapping was in the Tesla FSD presentation today. It was when there was a still image and the presenter said they thought it should be playing a video ... then he hit something on the remote and it played a video similar to above. ie. using cameras only to build 3D maps.

 

[heltok]

Waymo has a new Lidar:
https://thelastdriverlicenseholder.com/2019/04/21/new-waymo-lidar-spotted/

 

[scottf200]

This type of Earth-Centered Earth-Fixed (ECEF) Cartesian coordinate points mapping was in the Tesla FSD presentation today. It was when there was a still image and the presenter said they thought it should be playing a video ... then he hit something on the remote and it played a video similar to above. ie. using cameras only to build 3D maps.

The video: Tesla on Twitter

 

[mspohr]

Waymo has a new Lidar:
New Waymo LiDAR Spotted

Better buggy whip.

 

[cwerdna]

Hand Gestures And Horses: Waymo’s Self-Driving Service Learns To Woo The Public
US Postal Service will use autonomous big rigs to ship mail in new test

 

[cwerdna]

https://techcrunch.com/2019/05/23/c...-1400-successful-unprotected-left-hand-turns/

Cruise, the self-driving division within General Motors, today released a video reporting it successfully completed 1,400 such turns within a 24-hour period. The test took place on the busy and hilly streets of San Francisco.