What is SAE Level 5 and can Tesla actually achieve it?

[Knightshade]

Question for you, are you saying that if *any* human driver is able to successfully navigate a particular situation, then the level 5 must also be able to navigate it?

Typically the law uses a reasonable person standard for such judgements.

So even if you're SUPER AWESOME at driving through white-out snow conditions, the SAE standard doesn't require L5 to do so (it even cites that as an example of something the typical driver is not expected to handle).


But "fairgrounds parking lot where hundreds, or thousands, of human drivers drive just fine every week" would certainly fall under a reasonable person standard as something a driver is expected to be capable of handling.

And so would an L5 system.

If not, do you think the sae definitions defines in which conditions specifically? Also if not, then who will ever define these conditions specifically or will be be struck forever debating what situation is actually "appropriate" for level 5 vs not?

In the sae definitions, it says *a* driver and not *any* driver. I think this is on purpose. If for any reason, *a* driver decides it's not manageable for them, then this driver isn't required to navigate it. In the same way, the level 5 can decide not to navigate a particular situation because for example, it can't see any lines or sees too much water in the road.

Yeah, no.

Again they SPECIFICALLY say temporary conditions that would prevent a human driver are ok to also prevent an L5 system--- in both cases the human and the L5 could pull over and wait for conditions to get better.


Waiting lets the snow storm pass, or the water recede.

it won't ever fix "Car can't figure out the parking lot- but humans can" though.

 

[powertoold]

Typically the law uses a reasonable person standard for such judgements.

So even if you're SUPER AWESOME at driving through white-out snow conditions, the SAE standard doesn't require L5 to do so (it even cites that as an example of something the typical driver is not expected to handle).

Not sure why you're taking "the law" into this. It's a simple yes or no question.

Also, we aren't talking specifically about white-out snow conditions. We're talking about conditions in general.

 

[Knightshade]

If the system is not allowed to be used in a certain ODD then it is L4. ie, you can't turn the system on or the system shuts off when it encounters that ODD, then it is L4. But the car just being uncertain about how to handle something, does not make it L4.

Again that's not what the SAE doc says.

It defines L5 as:

The sustained and unconditional (i.e., not ODD-specific) performance by an ADS of the entire DDT and DDT fallback without
any expectation that a user will respond to a request to intervene.

Then it further defines what unconditional/not ODD specific means:

NOTE 1: “Unconditional/not ODD-specific” means that the ADS can operate the vehicle under all driver-manageable onroad conditions.

If it's so uncertain about any on-road driver manageable situation that it can't do the task- then it's not L5.


There's no "must operate in all driver management on road conditions UNLESS IT IS UNCERTAIN" exception.

Not sure why you're taking "the law" into this. It's a simple yes or no question.

Because when someone sues the guy who claims his car offers L5 when it clearly does not- that's how they'll decide the case.

Also, we aren't talking specifically about white-out snow conditions. We're talking about conditions in general.

Right, if a human can drive in those conditions in general, an L5 system must also be able to

Otherwise it's L4.

 

[powertoold]

Right, if a human can drive in those conditions in general, an L5 system must also be able to

Yup, so that's where you're wrong.

 

[Dan D.]

Here's a line from Page 30 that is open to abuse:

The level assignment rather expresses the design intention for the feature...
As such, the manifestation of one or more performance deficiencies in either the driving automation system or in the user’s use of it does not automatically change the level assignment. For example:
• An ADS feature designed by its manufacturer to be level 5 would not automatically be demoted to level 4 simply by virtue of encountering a particular road on which it is unable to operate the vehicle.


So company X sells you a Level 5 car. It has no external microphone nor can it interpret hand signals. You get to the fairground and the car is unable to follow the hand gestures and voice of the attendant, and it stops in a safe manner. You complain to company X who says "It's still Level 5 because the manifestation of one or more performance deficiencies...does not automatically change the level assignment."

 

[diplomat33]

Again that's not what the SAE doc says.

It defines L5 as:



Then it further defines what unconditional/not ODD specific means:




If it's so uncertain about any on-road driver manageable situation that it can't do the task- then it's not L5.


There's no "must operate in all driver management on road conditions UNLESS IT IS UNCERTAIN" exception.






Because when someone sues the guy who claims his car offers L5 when it clearly does not- that's how they'll decide the case.




Right, if a human can drive in those conditions in general, an L5 system must also be able to

Otherwise it's L4.

The SAE is pretty clear that being uncertain and not being able to do a task, does not demote the car to L4, see page 30:

The level assignment rather expresses the design intention for the feature... As such, the manifestation of one or more performance deficiencies in either the driving automation system or in the user’s use of it does not automatically change the level assignment. For example: An ADS feature designed by its manufacturer to be level 5 would not automatically be demoted to level 4 simply by virtue of encountering a particular road on which it is unable to operate the vehicle.

Again, the SAE levels are based on how the manufacturer has designed the system. They are not based on performance. Now, performance matters in terms of safety but it does not change the level.

 

[Dan D.]

The SAE is pretty clear that being uncertain and not being able to do a task, does not demote the car to L4, see page 30:



Again, the SAE levels are based on how the manufacturer has designed the system. They are not based on performance. Now, performance matters in terms of safety but it does not change the level.

And that's how it will be abused by companies who do not honor the spirit of the SAE intent.
I'm selling you a Full Level 5 car (just that it may not do the following things .....lists 100 things)

 

[Knightshade]

The SAE is pretty clear that being uncertain and not being able to do a task, does not demote the car to L4, see page 30:

Failure on one, specific, road as they describe- sure.

Failure on "any parking lot that requires direction from a human to use" is a much wider capability gap, and right in the wheel house of a limited ODD.

otherwise you can just handwave any lack of capability away and call everything L5 that can drive automatically at all.

Imagine a company that says "We are selling L5 cars and they've been totes L5 in ALL OUR DESIGN AND TESTING"

But they designed and tested someplace it's always sunny and never rains.

You buy it- it doesn't self drive in any precipitation.

They insist it's still L5 though.

Sound legit to you?

 

[diplomat33]

Here's a line from Page 30 that is open to abuse:

The level assignment rather expresses the design intention for the feature...
As such, the manifestation of one or more performance deficiencies in either the driving automation system or in the user’s use of it does not automatically change the level assignment. For example:
• An ADS feature designed by its manufacturer to be level 5 would not automatically be demoted to level 4 simply by virtue of encountering a particular road on which it is unable to operate the vehicle.


So company X sells you a Level 5 car. It has no external microphone nor can it interpret hand signals. You get to the fairground and the car is unable to follow the hand gestures and voice of the attendant, and it stops in a safe manner. You complain to company X who says "It's still Level 5 because the manifestation of one or more performance deficiencies...does not automatically change the level assignment."

And that's how it will be abused by companies who do not honor the spirit of the SAE intent.
I'm selling you a Full Level 5 car (just that it may not do the following things .....lists 100 things)

Yeah, it could lead to abuse. I do think a big reason why the SAE does not make levels dependent on performance is because performance can be too subjective. Plus, there are so many different driving situations that defining levels based on what situations the car can handle is almost impossible.

But performance does relate directly to safety. if your L5 car is not safe enough, then customers will be less likely to use it or worse, regulators will stop it. Also, if the L5 car does not provide a satisfactory experience to the customer, that will hurt sales. So if the manufacturer tries to sell a L5 car but it cannot do those 100 things, and customers really want a car that can do those things, then they won't buy or use the L5 car. I think that is two ways that abuse will be stopped. So I think the free market and/or regulators will take care of cars that are "bad L5".

Basically, the SAE just worries about defining what makes a car autonomous. But there are plenty of other factors like safety, customer satisfaction, performance, that will affect how "good" the autonomous car will be.

 

[Dan D.]

otherwise you can just handwave any lack of capability away and call everything L5 that can drive automatically at all.

Sound legit to you?

No it doesn't sound legit, but as I posted on Levels 2 page: FSD - Level 2

This line from page 30
it is incorrect to classify a level 4 design-intended ADS feature equipped on a test vehicle as level 2 simply because on-road testing requires a test driver to supervise the feature while engaged, and to intervene if necessary to maintain safe operation.

To me, Tesla has designed and is intending FSD to be Level 4. They say that it is Level 2 only to avoid the responsibilities that are required to follow a Level 4 testing protocol.

Tesla is already going against the spirit of SAE rules. If they can do it, others can too for Level 5.

 

[diplomat33]

Failure on one, specific, road as they describe- sure.

Failure on "any parking lot that requires direction from a human to use" is a much wider capability gap, and right in the wheel house of a limited ODD.

otherwise you can just handwave any lack of capability away and call everything L5 that can drive automatically at all.

Imagine a company that says "We are selling L5 cars and they've been totes L5 in ALL OUR DESIGN AND TESTING"

But they designed and tested someplace it's always sunny and never rains.

You buy it- it doesn't self drive in any precipitation.

They insist it's still L5 though.

Sound legit to you?

That's where regulators and the free market come in. A company can claim a car is L5 but if it has a serious weaknesses, like it can't handle any rain at all, or it can't handle most parking lots, then regulators might step in to investigate. But also, customers may see the L5 as really crappy L5 and not buy or use the car. If that happens, the company could lose a lot of money. I think that is one reason why companies are focused on so much on testing and development. Companies understand that achieving FSD is one thing but the real goal is achieving "good" FSD because your FSD should be loved by the public, safe, reliable, enjoyable and profitable. Putting out crappy FSD will just hurt your company.

 

[Nuclear Fusion]

There should be a Level 6 created - anything a human can do (in traversing an environment)
Level 5 used to mean this but it’s been qualified & walked back
If a human can do a U turn of a bus on a single space cliff edge mountain road, then an autonomous car should be able to also.

And no, Level 6 doesn’t mean jumping over 50 cars off a ramp Evil Knievel style. It’s about normal transport

And when, after watching the YouTube video, you respond with “how do you expect an autonomous car to do that”, what do you expect in future when humans are no longer allowed to drive?

 

[Dan D.]

If a human can do a U turn of a bus on a single space cliff edge mountain road, then an autonomous car should be able to also.

Yeesh.
Well, that human did it, and survived. Shall we post videos of drivers that didn't survive next?
No reason that an autonomous Level 5 car couldn't do it too. In fact, better the autonomous car tries and succeeds or fails than a human. No Level 6 needed.

 

[powertoold]

I think there are widespread misconceptions about the level definitions. I didn't understand them myself, until I read the sae document a couple times.

A lot of people on TMC and in the news media use the levels as a "progress" or performance parameter. They'll say things like, Tesla is only level 2 so they're far away from level 4 / 5. That's not appropriate use of the definitions. A level 2 feature doesn't necessarily mean it's far away from being a level 5 feature. The manufacturer can just decide to modify its design to conform to the level 5 specs. Although it'd be stupid to do so if it's not reliable and safe enough.

I actually think this is the sae's fault for not making it absolutely clear that the definitions aren't mean for subjective assessments of performance or progress.

 

[diplomat33]

As I read the SAE document, I think the SAE was thinking of major restrictions of the ODD when they defined L4. I don't think they were thinking of small restrictions. They were thinking of significant restrictions like Waymo that geofences their robotaxi service to a specific 100 sq mi are or GM's Supercruise that is restricted to certain mapped highways.

 

[powertoold]

As I read the SAE document, I think the SAE was thinking of major restrictions of the ODD when they defined L4. I don't think they were thinking of small restrictions. They were thinking of significant restrictions like Waymo that geofences their robotaxi service to a specific 100 sq mi are or GM's Supercruise that is restricted to certain mapped highways.

It seems they were mostly focused on weather, driving speed, and location based ODDs, but I think level 5 really shouldn't have any ODD, however minor. Level 5 driving should be based on the cars certainty in navigating after the user has already specified the location.

Although I think Tesla will achieve a level 5 feature, I don't think they'll roll it out to the entire fleet. People can easily abuse an ADS without ODD. People will choose complicated / stupid dropoff locations like mountain tops or difficult to navigate roads in effort to get the cars stuck, lol.

 

[Nuclear Fusion]

Yeesh.
Well, that human did it, and survived. Shall we post videos of drivers that didn't survive next?
No reason that an autonomous Level 5 car couldn't do it too. In fact, better the autonomous car tries and succeeds or fails than a human. No Level 6 needed.

I always defined Level 5 as anything a human can do. I visit various forums. Haven’t you noticed how people are saying “well technically L5 isn’t anything a human can do”?
An autonomous car should also be able to travel off road ‘into the woods’ to circumvent a fallen tree across a map-marked dirt track/road - humans on camping adventures often have to do it.
Hence, why I’ve suggested L6

 

[diplomat33]

I think there are widespread misconceptions about the level definitions. I didn't understand them myself, until I read the sae document a couple times.

A lot of people on TMC and in the news media use the levels as a "progress" or performance parameter. They'll say things like, Tesla is only level 2 so they're far away from level 4 / 5. That's not appropriate use of the definitions. A level 2 feature doesn't necessarily mean it's far away from being a level 5 feature. The manufacturer can just decide to modify its design to conform to the level 5 specs. Although it'd be stupid to do so if it's not reliable and safe enough.

I actually think this is the sae's fault for not making it absolutely clear that the definitions aren't mean for subjective assessments of performance or progress.

I completely agree about people misunderstanding the levels. But I don't really blame the SAE. If you read the document, they tried to be very clear that they are not subjective assessments of performance or progress. I think the real reason for the widespread confusion is simply that people don't read the SAE document and just repeat what they hear somewhere else say or misunderstand the document and spread their flawed interpretation anyway. We all know how quickly misinformation can spread on the internet. Sadly, there are a "journalists" who don't do their due diligence and spread misinformation as well about the levels. I've seen articles purportedly written by an autonomous driving journalist that are full of misinformation.

I would nitpick a bit when you say that a manufacturer can just modify their L2 to conform to L5. I don't think it is that easy. After all, L2 means it is missing some OEDR. Maybe the L2 cannot detect road debris. Maybe the L2 cannot respond to construction zones. So making it L5 means adding those capabilities. That undoubtedly mean a lot of time and money in developing the software, testing etc... So it is a significant change.

 

[qdeathstar]

And no, Level 6 doesn’t mean jumping over 50 cars off a ramp Evil Knievel style. It’s about normal transport

Elon says the Roadster should be able to do that by 2023....

I agree with your larger point about level 6.

 

[powertoold]

I would nitpick a bit when you say that a manufacturer can just modify their L2 to conform to L5. I don't think it is that easy. After all, L2 means it is missing some OEDR. Maybe the L2 cannot detect road debris. Maybe the L2 cannot respond to construction zones. So making it L5 means adding those capabilities. That undoubtedly mean a lot of time and money in developing the software, testing etc... So it is a significant change.

I was mostly using that point to say that just because a feature is level 2, doesn't necessarily mean it's "far away" from level 4 / 5, in terms of progress.