Autonomous Car Progress

[spacecoin]

No I'm going by the safety report used by Tesla and all Tesla fans. The one regurgitated every single day on here, reddit, twitter and youtube. The one with thousands of news articles written about. The one that has been read over hundreds of millions of times.

The Tesla "safety report" was debunked as skewed marketing by researchers years ago, are you guys still going on about that?

Is Tesla want to produce a credible report, have external and independent researchers look at the raw data and write the report. Now it's super creative definitions and stats with the sole purpose of making the company look great.

 

[scottf200]

Some camera only solutions have cameras in the a-pillar for triangulation to improve safety. Tesla does not.

FYI, this is a bit of a fallacy. At longer distances, they are still too close together. Just like humans where 2 eyes are most helpful out an arms length or two. Humans have so many ways to judge distance and our brain does that automatically. Look at thing near and far with both eyes then close one eye at a time. The two angles don't matter at longer distances. I've studied this for a few reasons.

 

[spacecoin]

FYI, this is a bit of a fallacy. At longer distances, they are still too close together. Just like humans where 2 eyes are most helpful out an arms length or two. Humans have so many ways to judge distance and our brain does that automatically. Look at thing near and far with both eyes then close one eye at a time. The two angles don't matter at longer distances. I've studied this for a few reasons.

Yeah I know. Good clarification. I'm not claiming that it's the solution, I'm saying thet it might help with reliability in same situations. I'm firmly in the active sensing "physical measurement as ground truth" camp.

In HW3 there is no forward facing stereo vision. I am not sure if the two cameras in HW4 are overlapping with the same focal point / optics or if they are different.

Compound Eye has a nice "marketing" overview of it btw: Technology | Compound Eye

 

[willow_hiller]

The Tesla "safety report" was debunked as skewed marketing by researchers years ago, are you guys still going on about that?

Darn that tricky Tesla. Skewing the data by presenting all of it in its entirety without sampling.

Methodology:
We collect the amount of miles traveled by each vehicle with Autopilot active or in manual driving, based on available data we receive from the fleet, and do so without identifying specific vehicles to protect privacy. We also receive a crash alert anytime a crash is reported to us from the fleet, which may include data about whether Autopilot was active at the time of impact. To ensure our statistics are conservative, we count any crash in which Autopilot was deactivated within 5 seconds before impact, and we count all crashes in which the incident alert indicated an airbag or other active restraint deployed. (Our crash statistics are not based on sample data sets or estimates.)

Also another reason you cannot compare Waymo/Cruise crash reporting to Tesla ADAS: privacy. Waymo and cruise passengers are driving in a company-owned car, and have waived their rights to data privacy when it comes to reporting disengagement locations. Tesla cannot report individual driver location data without anonymizing it. Their vehicles are privately owned.

 

[spacecoin]

Darn that tricky Tesla. Skewing the data by presenting all of it in its entirety without sampling.

So they provide the full dataset to external researchers now?

Also another reason you cannot compare Waymo/Cruise crash reporting to Tesla ADAS: privacy.

Seems impossible to replace the vin with another random identifier?

The only thing that is sure is that Tesla with AP active crashes less ON THE HIGHWAY than the average 11-12 year old car mostly without any driver assist or active safety on ALL ROADS. Given that most accidents are outside the highway, I rest my case.

I would love to see AP safety stats at day vs night for example, and different road types and speeds. And disengagement stats for the FSDb.

 

[EVNow]

What would be a good comparison that I've not seen is between Waymo/Cruise vs Chinese AVs. When people talk about Waymo being a leader, are they just ignoring Baidu etc ?

 

[stopcrazypp]

No I'm going by the safety report used by Tesla and all Tesla fans. The one regurgitated every single day on here, reddit, twitter and youtube. The one with thousands of news articles written about. The one that has been read over hundreds of millions of times.

When you actually use the Tesla methodology, none of the Waymo/Cruise so called accident (which people here are losing their minds over) would count.

According to Tesla's own methodology, Waymo/Cruise HAS NO ACCIDENTS.

Which part of that don't you understand?

Waymo/Cruise would count this as an accident, Tesla doesn't. This also includes hitting the curb, cones, objects, also rear-ending cars, being rear ended, etc. Unless its a VERY SERIOUS accident, Tesla wouldn't count it as an accident. Waymo/Cruise on the other hand considers any contact even at 0mph as an accident.

Don't you see how flawed and utterly ridiculous Tesla's "safety report" is?

First admit that Tesla's two sentence is not a "safety" report but a PR report or explain how it is a safety report when less than 1% of all accidents count?

Here is another accident. This one actually got reported to insurance and police so automatically gets counted toward's NHTSA human driver statistics. But guess who doesn't count it? That's right TESLA! So the accident goes to NHTSA ~500k statistics. But Tesla doesn't count it towards theirs. Then Tesla compares themselves to the NHTSA stats and say... AHA see we are 5x better than human drivers!

When actually its just them not counting 99% of the accidents that NHTSA is counting.

https://twitter.com/x/status/1590165478634586112

Think about it, lets say you have 10 accidents. Rear-ends, side swipes, etc.

Police gets called, insurance are exchanged. It gets automatically counted towards NHTSA 500k miles stats. But it doesn't meet Tesla's airbag criteria. Now Tesla's stats is automatically 10x better than NHTSA. By virtue of just not including those accidents.

How is it that you can't see this?

Waymo/Cruise counts 100% of all collision as accidents. Tesla counts less than 1% of all their collisions as accidents. That's all that matters when comparing to Tesla.

FYI the stat being compared is not Tesla vs NHTSA, it's Tesla FSD vs Tesla disengaged. It's proven Tesla FSD has less crashes under the same methodology. Previously people dismissed AP stats due to skewing of road types (AP being disproportionately highway usage). This is no longer a factor in FSD.

As you link yourself, the stat is an accident every 3.2M miles for FSD. For AP off, that stat ranged from 0.94M miles in Q1 2021 (worse) to 1.92M miles Q3 2018 (best also first report), with most recent best after the worst being 1.71M miles (Q3 2022, Q4 was worse).

You can of course argue the details and if it matters, but that is the fact being pointed out.

On the question of airbag deployment, note as of 2022.20 update, pretensioners can activate in anticipation of a crash and it triggers the data collection (airbag activation is not required)

Tesla Vision’s vast potential highlighted by seat belt safety feature

Elon Musk briefly highlighted how Tesla Vision enhanced a seat belt safety feature introduced in Update 2022.20.

www.google.com

Also Tesla in prior reports have captured accidents where neither activated, as of the note as of Jan 2023.

Tesla Vehicle Safety Report | Tesla

At Tesla, we believe that technology can help improve safety. That’s why Tesla vehicles are engineered to be the safest cars in the world. We believe the unique combination of passive safety, active safety, and automated driver assistance is crucial for keeping not just Tesla drivers and...

www.tesla.com

Also, as others pointed out, the point being made about tracking, no automaker has as comprehensive tracking as Tesla that makes such comparisons possible among the same population. When people mentioned that, I knew they were talking about the NHTSA standing order. It's very obvious police reports and manufacturing reporting were not capturing anywhere near what Tesla is capturing (something pointed out as a limitation in most reports).

So while Tesla may miss some reports that would have been reported to the police, they also capture accidents that would not be, as police reporting requirements generally are not tied to airbag deployment. While state dollar amount requirements generally capture most airbag deployed accidents, there are states with no dollar requirements for police reporting of accidents, plenty of states even with dollar requirement allow after the fact reporting to the DMV that does not involve the police (and thus would not be in police databases), plus not all people follow the strict requirements of police reporting.

 

[EVNow]

Robotaxis Are Everywhere in San Francisco Now. But Something’s Wrong.

Blocking transit. Driving on the sidewalk. Speeding away from cops.

slate.com

They’ve blocked traffic, driven on the sidewalk, sped away from cops—and the city is powerless to stop them.​

Strange as it may seem, California stops requiring that AV companies share disengagement data and collision locations as soon as they begin collecting passenger fares, as Waymo and Cruise now do. From that point forward, if an AV vehicle jeopardizes safety on the street—for instance, by causing a crash or blocking a transit line—the public won’t know unless the AV company chooses to publicize it (unlikely) or if a passerby reports the incident to 911 or posts about it on social media (unreliable).

Urban leaders anticipated these problems and tried to forestall them. In 2020 officials from Los Angeles, San Francisco, and San Diego asked the CPUC “not [to] create a deployment program that would give participants blanket authority to operate a fared service anywhere in the State.” The CPUC rejected that request, which has hobbled cities’ ability to manage their streets.

Unfortunately, that now seems to be where autonomous-vehicle regulation is: stuck in an intersection, like that Waymo car in Phoenix, with no one in the driver’s seat.

Its high time the citizens of a city have a say in what goes on in the city.

 

[Dewg]

You can crash into a building and the airbags won't deploy.

I'm not going to get into the various arguments going on here, but I will correct you here.

Teslas have been crash tested repeatedly by US and European agencies and consistently received top marks on safety and crash survivability. The idea that the airbags don't deploy for significant crashes is absurd, and can give new users here on TMC the wrong impression and scare them.

 

[EVNow]

And then you have Tesla fans using these two sentence "Safety Report" to say that Tesla is safer than Waymo/Cruise. As you see @EVNow Omar and others do.

Show me where I've said that or apologize for lying.

 

[EVNow]

You and others also claim that Waymo/Cruise are UNSAFE compared to Tesla FSD and that the hundreds of pages that Waymo has released is nothing but PR.

Robotaxis Are Everywhere in San Francisco Now. But Something’s Wrong.

Blocking transit. Driving on the sidewalk. Speeding away from cops.

slate.com

Cruise and other AV companies maintain a ”Critical Response Line” dedicated to handling emergencies, but no public data measure their responsiveness. Cruise does not include information about right-of-way blockages or response times in its self-written, 175-page “safety report.”

 

[powertoold]

Alright guys you heard it from The Man. Cruise is done for:

https://twitter.com/x/status/1665516168805773312

 

[EVNow]

Alright guys you heard it from The Man. Cruise is done for:

https://twitter.com/x/status/1665516168805773312

How is this stupid thing allowed to operate ?

If this was a Tesla robotaxi they'd be calling for Elon's head.

 

[stopcrazypp]

How is this stupid thing allowed to operate ?

If this was a Tesla robotaxi they'd be calling for Elon's head.

To be fair, it didn't crash. Even if it did (like into the bus last time), this is happening on SF local roads and not affecting CA state roads, so the state (which is the party that regulates this) hardly has a reason to care. When they start venturing out on some state highways and clogging up those, then maybe CPUC will see a reason to care when state agencies like CHP/Caltrans starts having to deal with it.

 

[Bladerskb]

FYI the stat being compared is not Tesla vs NHTSA, it's Tesla FSD vs Tesla disengaged.

This is simply false. Come on...

https://twitter.com/x/status/1631063252875505669

On the question of airbag deployment, note as of 2022.20 update, pretensioners can activate in anticipation of a crash and it triggers the data collection (airbag activation is not required)

Tesla Vision’s vast potential highlighted by seat belt safety feature

Elon Musk briefly highlighted how Tesla Vision enhanced a seat belt safety feature introduced in Update 2022.20.

www.google.com

Also Tesla in prior reports have captured accidents where neither activated, as of the note as of Jan 2023.

Tesla Vehicle Safety Report | Tesla

At Tesla, we believe that technology can help improve safety. That’s why Tesla vehicles are engineered to be the safest cars in the world. We believe the unique combination of passive safety, active safety, and automated driver assistance is crucial for keeping not just Tesla drivers and...

www.tesla.com

This is also false.

https://twitter.com/x/status/1631063252875505669

Its all there black and white, clear as crystal.
Everything else you posted is 100% irrelevant.

 

[stopcrazypp]

FYI the stat being compared is not Tesla vs NHTSA, it's Tesla FSD vs Tesla disengaged.

This is simply false. Come on...

https://twitter.com/x/status/1631063252875505669

I'm talking about the "fact" being discussed:

FSD + Human has better accident rate than just human. Looks like a good perf goal that is being met.

The person was not comparing FSD vs NHTSA. You jumped in later to support @spacecoin 's rebuttal, but your data in fact supports @EVNow.

As mentioned, you can dispute if it matters given the limitations to the data, but it's comparing Tesla cars running FSD+Human vs Tesla cars with just human with the same criteria.

This is also false.

https://twitter.com/x/status/1631063252875505669

What's false about it? Tesla previous accident reports prior to January 2023 included some accidents that had neither airbags nor other active restraints deployed:

Specifically, we discovered reports of certain events where no airbag or other active restraint deployed, single events that were counted more than once, and reports of invalid or duplicated mileage records. Including these events is inconsistent with our methodology for the Vehicle Safety Report and they will be excluded going forward.

Its all there black and white, clear as crystal.
Everything else you posted is 100% irrelevant.

As above, it's 100% relevant, given the context of the discussion.

 

[diplomat33]

What deficiencies in vision-only do you see vs LIDAR, please give specific categories / examples of failures or deficiencies.

Nvidia has achieved a new state of the art transformer NN that creates a bird's eye view by unifying multiple camera views. In their paper, they state that it is far better than other vision-only NN for doing velocity estimation and it is approaching lidar level (implying not yet as good as lidar for velocity estimation):

"Previous camera-based methods [47, 31, 45] were almost unable to estimate the velocity, and our method demonstrates that temporal information plays a crucial role in velocity estimation for multicamera detection. The mean Average Velocity Error (mAVE) of BEVFormer is 0.378 m/s on the test set, outperforming other camera-based methods by a vast margin and approaching the performance of LiDAR-based methods [43]." (p 8)

In their conclusion, the paper also states that vision-only lags behind lidar in effect and efficiency and that 3D location from 2D remains a challenge for vision-only systems:

"Limitations. At present, the camera-based methods still have a particular gap with the LiDAR-based methods in effect and efficiency. Accurate inference of 3D location from 2D information remains along-stand challenge for camera-based methods." (p 11)

Source: https://arxiv.org/pdf/2203.17270.pdf

So we see that velocity estimation and 3D location from 2D information are two areas where vision-only lags behind lidar.

 

[JB47394]

our method demonstrates that temporal information plays a crucial role in velocity estimation for multicamera detection

Time is crucial when estimating velocity? Am I missing something or is this an absurd statement?

So we see that velocity estimation and 3D location from 2D information are two areas where vision-only lags behind lidar.

How significant is 0.378 m/s (0.85 mph) to autonomy software? I drive around with an mAVE considerably higher than that number.

 

[Bladerskb]

I'm talking about the "fact" being discussed:

The person was not comparing FSD vs NHTSA. You jumped in later to support @spacecoin 's rebuttal, but your data in fact supports @EVNow.

As mentioned, you can dispute if it matters given the limitations to the data, but it's comparing Tesla cars running FSD+Human vs Tesla cars with just human with the same criteria.

STOP. No one compares FSD vs Non FSD.
The number that is always pushed is the FSD vs. NHTSA number.
The number that is pushed on articles, here, twitter, reddit is the FSD vs. NHTSA number.
When that number is exposed. Then people jump to the FSD vs. Non FSD number.

Guess what number that Tesla pushes? Exactly :
"In the last 12 months, a Tesla with FSD Beta engaged experienced an airbag-deployed crash about every 3.2 M miles, which is ~5x safer than the most recently available US average of 0.6M miles/police-reported crash"

You Tesla fans are ridiculous. You will twist literally anything..

What's false about it? Tesla previous accident reports prior to January 2023 included some accidents that had neither airbags nor other active restraints deployed:

Bro its there in Black and White.
Tesla is very pointed that for FSD data, they looked at the last 12 months and they counted just airbag deployment.

"In the last 12 months, a Tesla with FSD Beta engaged experienced an airbag-deployed crash about every 3.2 M miles, which is ~5x safer than the most recently available US average of 0.6M miles/police-reported crash"

Even for the regular AP data, including "some accidents" without airbag is still only accounting for less than 1% of all accidents.
But in the FSD Beta data, none of that is being done.
Its strictly airbag deployed crash vs. police reported crash.

You Tesla fans, oh my goodness.

As above, it's 100% relevant, given the context of the discussion.

Tesla's own methodology when applied to Waymo/Cruise would mean they have no accidents.
Tesla's own methodology proves they account for less than 1% of all accidents reported compared to the NHTSA.
Waymo/Cruise counts 100% of all collisions as accidents, Tesla own methodology says they count less than 1%.

These are the facts. Let it Sink In. I'm not gonna go back and forth with you on something that is balant.

 

[willow_hiller]

Tesla's own methodology proves they account for less than 1% of all accidents reported compared to the NHTSA.

That's completely wrong. Look at how NHTSA collects crash statistics: Crash Report Sampling System | NHTSA

"CRSS obtains its data from a nationally representative probability sample selected from the estimated 5 to 6 million police-reported crashes that occur annually.

These crashes include those that result in a fatality or injury and those involving property damage. By focusing attention on police-reported crashes, CRSS concentrates on those crashes of greatest concern to the highway safety community and the general public."

By relying on police reports, only a small fraction of actual crashes are every collected by the NHTSA. Tesla's telemetry is far more comprehensive.