What is SAE Level 5 and can Tesla actually achieve it?

[mikes_fsd]

Similarly, Waymo say that they are working to make Driver L5 some day. But the current design intent of the Waymo Driver is L4 because the Waymo Driver is currently designed to operate in a geofenced area.

And those L4 were with real drivers for a very long time - yet it was still L4 system being tested and Google/Waymo was expecting the safety driver to intervene at the right moments.

a la Tesla FSD, the stated intent of the system is L5, and Tesla demands that anyone in the beta agrees to supervise the car at all times, until the software matches the desired performance of the intended L5 target.

This is all pointless, as discussing intent does not get us closer to L5.
Executing and seeing forward/positive progress does.

 

[S4WRXTTCS]

We would interpret it that way if the SAE didn't say the definitions don't include any performance or safety criteria. They even say that level 5 is about design intent, not actual performance. So it's very clear we can't compare level 5 to a human.

If your design intent is to operate on roads, and conditions that can be driven by human drivers then you've set goal post for it.

With L4 the ODD is allowed to be limited, and there is no need to set a goal post.

With L5 the ODD needs to have some kind of goal post. It's meaningless without a goal post. The unrestricted ODD is what separates it from L4. But, they can't just say unrestricted as that would be really unrealistic. So they use driver-manageable as the goal post.

Tesla won't be able to drive on driver-manageable roads with the HW3 Sensor suite. Therefor it does not meet the design intent criteria. It will fall far short of being able to drive on driver-manageable roads, and conditions.

I agree that the SAE Levels are not intentionally setting performance criteria nor do they set safety criteria. That they avoid that as much as possible, but for L5 they had to have some reference point for design intent.

If Tesla had a design that could even get close to driving on driver-manageable roads, weather, and situations then this discussion wouldn't be happening.

What people don't seem to grasp is HW3/FSD is meant to fail. The entire purpose of it is to fail over, and over again as it gets better each time. That Elon knew before they ever released it that they were not going to achieve L5 with the sensor suite.

Elon accepts embraces failure, and achieves success through it.

 

[powertoold]

L5 they had to have some reference point for design intent.

The SAE definition doesn't have any performance reference point. They make that very clear. All other interpretation here is speculation, so we should agree on the clearest SAE statement on performance or safety. In this context, it actually makes the sae levels more meaningful, since the levels simply refer to a level of automation, not performance. See my dumb "box" analogy.

 

[S4WRXTTCS]

The SAE definition doesn't have any performance reference point. They make that very clear. All other interpretation here is speculation, so we should agree on the clearest SAE statement on performance or safety. In this context, it actually makes the sae levels more meaningful, since the levels simply refer to a level of automation, not performance. See my dumb "box" analogy.

In your original post you said "Here, we discuss whether or not Tesla can achieve level 5 in the near future, or is it impossible (given the sensor suite and/or feature limitations)?"

I've explained why they can't achieve it.

You, and I both know the SAE Levels are about design intent. We also agree that the design intent of FSD is Level 5 as Elon has been very clear about that.

So we know the goal is L5.

But, the disagreement comes down to what achieving L5 really is.

To me the SAE Level 5 clearly uses driver-manageable roads, and weather conditions. The Tesla HW3 Sensor suite isn't intended to handle driver-manageable weather conditions, and anyone with FSD that lives in areas with challenging weather knows this.

So I say HW3 isn't even an intent at L5 driving, and it's just marketing by Tesla/Elon in trying to claim it somehow is. So it's already failed as its not a serious attempt at L5.

And, actually achieving Level 5 is more than just design intent as the design has to actually work to get approval. I would be absolutely astonished if Tesla achieved any kind of approval from regulatory/insurance/customers to do L4 driving let alone L5 with HW3.

To get approval it has to meet certain Safety thresholds like Elon himself has used the 10x better than a human for the safety goal.

So even if we ignore regulatory it's unlikely to ever get Elons approval.

 

[powertoold]

The Tesla HW3 Sensor suite isn't intended

Huh? Are you saying Tesla doesn't intend the sensor suite achieve level 5? That makes no sense, as Elon said so himself.

Again, if Tesla intends it to be level 5 and it doesn't have any ODD and has no intervention expectation, it's level 5. That's the sae definition.

To get approval it has to meet certain Safety thresholds like Elon himself has used the 10x better than a human for the safety goal.

Safety is totally separate from the sae definition.

Anyway, it seems to me that you're doing the mental gymnastics on this topic, lol.

 

[mikes_fsd]

The Tesla HW3 Sensor suite isn't intended to handle driver-manageable weather conditions, and anyone with FSD that lives in areas with challenging weather knows this.

Also, have you watched any of the FSD Beta videos, especially guy from Michigan and Kim from Rhode Island. They seem to deal with snow and weather just fine.
Unlike Waymo, Tesla is now in ~23 states plus DC. There is NO restriction to when to use FSD beta or where to use it.

 

[S4WRXTTCS]

Safety is totally separate from the sae definition.

But, its critical towards achieving any kind of autonomous driving whether its L3, L4, or L5.

And, in your original post you said achieve.

You know HW3 doesn't have sensor redundancy

You know HW3 has issues with sensors being obstructed in adverse, but driver manageable conditions.

You know Elon is very much a believer of fail until you succeed. Sometimes you still try when you absolutely know you don't have what it will take, but there is enough things you can learn along the way that it doesn't matter.

The mental gymnastics is about what driver-manageable means. I don't know what kind of mental gymnastics you're doing to think the SAE is not using human drivers as a benchmark for the ODD for L5.

 

[S4WRXTTCS]

Also, have you watched any of the FSD Beta videos, especially guy from Michigan and Kim from Rhode Island. They seem to deal with snow and weather just fine.
Unlike Waymo, Tesla is now in ~25 states plus DC. There is NO restriction to when to use FSD beta or where to use it.

What's interesting about some of the videos is the car will even say that a sensor is obstructed or "bad weather detected" and yet it will still allow them to continue on.

So I'm curious what the actual delivered product will have as a threshold for weather. I don't think its going to be so lenient to us non-EAP people if/when released.

On twitter I'm trying to find the WA State FSD beta people to follow, but I haven't found them. There is at least 1 WA state person, and at least 1 Oregon one according to the map of known states with FSD beta (where you got the 25 number from).

 

[powertoold]

I don't know what kind of mental gymnastics you're doing to think the SAE is not using human drivers as a benchmark for the ODD for L5.

It literally says no benchmarks in the definition. The SAE definition removes any subjectivity from the levels. That's their goal.

Here you are, saying the sae definition uses humans as a benchmark for level 5 performance when the definition specifically says it doesn't.

Driver-manageable simply means the road *can* be driven on by the public. There's no implied performance criteria in that.

 

[S4WRXTTCS]

It literally says no benchmarks in the definition. The SAE definition removes any subjectivity from the levels. That's their goal.

Driver-manageable sounds a lot like a benchmark to me.

Don't you have better things to do like following a car so you can use AP?

 

[powertoold]

Driver-manageable sounds a lot like a benchmark to me.

Don't you have better things to do like following a car so you can use AP?

If there were any benchmarks or performance comparisons, the sae definition would make that clear in a 35-page description. Instead, you are here making your own interpretation of it, which is the opposite of what that document is trying to do. It seems they were aware people would make performance comparisons, so they specifically added a line to clear that up.

After all, your idea of driver manageable is different than Lewis Hamilton's, is different than mine, different with a drunk person, etc. That's exactly what the SAE wanted to avoid.

 

[mikes_fsd]

What's interesting about some of the videos is the car will even say that a sensor is obstructed or "bad weather detected" and yet it will still allow them to continue on.

So I'm curious what the actual delivered product will have as a threshold for weather. I don't think its going to be so lenient to us non-EAP people if/when released.

On twitter I'm trying to find the WA State FSD beta people to follow, but I haven't found them. There is at least 1 WA state person, and at least 1 Oregon one according to the map of known states with FSD beta (where you got the 25 number from).

This guy makes pretty maps... https://twitter.com/jeadly/status/1349512344888602627
he has 23 states plus DC (not the 25 I mentioned)

Hawaii is not on that map but has FSD Beta!

 

[Redbrick]

So say if tomorrow tesla removes the need for driver to keep your hands on the wheel, and autopilot nags...then those with fsd immediately get L4?

 

[mikes_fsd]

(where you got the 25 number from).

I knew I had a real reason for 25 states.
That same thread is a response to a guy getting it in New York state https://twitter.com/stephenpallotta/status/1349483235231805440
And further down, in the same thread a guy says he got it in Pennsylvania!

 

[powertoold]

So say if tomorrow tesla removes the need for driver to keep your hands on the wheel, and autopilot nags...then those with fsd immediately get L4?

No, because level 4 requires some fallback feature, like pulling over.

 

[Bladerskb]

That's my understanding as well.

If the feature is currently designed to have no ODD and doesn't expect the user to intervene, then it's level 5, regardless of its performance.

It's a bit of a gray area with FSD beta. I don't think the fsd beta is designed (in the software) to expect intervention. However, Tesla expects testers to intervene and disengage (in agreements or coaching). But yes, I still think this is level 2.

There's a prompt that shows up that says something similar to "FSD Beta can't complete this maneuver please take over." Ofcourse im paraphrasing.

This is wrong. If we get stuck on this, then there's no point debating any further, since our reading comprehension is disjointed.

Source: “On-road” refers to publicly accessible roadways (including parking areas and private campuses that permit public access)

This clearly means that many private campuses (like Target / Walmart / businesses / Universities / etc.) have roads / parking lots / garages that are PRIVATELY owned but are publicly accessible for purposes of business / etc. Anyone is allowed to drive on these roads, despite them being privately owned.

Publicly accessible roadways:
1) Parking areas (parking lots)
2) Private campuses with publicly accessible roads / parking lots

Parking areas are not parking lots and is not the intent of the SAE language. Parking lot is defined as a big place with durable or semi durable surface.

This is why they said "areas" not "lots". They knew what

There are different types of parking areas, parking spaces on side streets, regular unmarked side street parking, grassfields, driveways, gated community, private condos, etc. There are also different types of driveways. Them using the word "parking lot" would eliminate all of that.

To think that L5 doesn't cover the biggest usecase of driving which is helping the elderly and disabled. Usecases that L4 parking systems would cover.Usecases that L4 system in certain communities would cover.

Waht you are saying that L4 would actually be more functional than a L5 system. When in a nutshell, the definition of L5 is L4 without limits or L4 without restrictions. You can't have a L4 system doing things and covering use-cases that L5 doesn't cover.

For example: Showing up to a concert or an event and the parking area is not some lot (durable or semi-durable surface)...

That's my understanding as well.

If the feature is currently designed to have no ODD and doesn't expect the user to intervene, then it's level 5, regardless of its performance.

It's a bit of a gray area with FSD beta. I don't think the fsd beta is designed (in the software) to expect intervention. However, Tesla expects testers to intervene and disengage (in agreements or coaching). But yes, I still think this is level 2.

FSD Beta gives an error with message similar to can't complete maneuver please take over (obviously i'm paraphrasing.

 

[powertoold]

FSD Beta gives an error with message similar to can't complete maneuver please take over (obviously i'm paraphrasing.

I haven't seen that lately, but ya, I vaguely remember it before.

As for the driveway thing, I still disagree with you. Private driveways are not publicly accessible. I gave examples of private areas that are intended to be publicly accessible.

As for your elderly example, that's subjective.

 

[S4WRXTTCS]

If there were any benchmarks or performance comparisons, the sae definition would make that clear in a 35-page description. Instead, you are here making your own interpretation of it, which is the opposite of what that document is trying to do. It seems they were aware people would make performance comparisons, so they specifically added a line to clear that up.

After all, your idea of driver manageable is different than Lewis Hamilton's, is different than mine, different with a drunk person, etc. That's exactly what the SAE wanted to avoid.

What sets the expectations for the ODD for L5?

We know it can't be limited as that's what L4 is.

We know it can't be unlimited as that's not realistic.

What could possibly be used to set the expectations for an L5 autonomous driving robocar? If only we knew who or what was driving cars around today we could use that as a reference point. So designers of such a system knew how to set the ODD design intent.

 

[powertoold]

expectations for an L5 autonomous driving robocar

If we are to use the sae definitions, the expectation is no ODD, no requirement that the user intervene or take over. Black and white definition.

Yes, there are other subjective expectations for a level 5 car, but that's separate from the sae definition.

 

[S4WRXTTCS]

If we are to use the sae definitions, the expectation is no ODD, no requirement that the user intervene or take over. Black and white definition.

Yes, there are other subjective expectations for a level 5 car, but that's separate from the sae definition.

Yet, the SAE Level 5 allows for exceptions as stated:

"However, there may be conditions not manageable by a driver in which the ADS would also be unable to complete a given trip"

All these pesky "drivers" being used to determine the expected ODD limits, and ruining our black and white definition.

I would have preferred:

"Neither snow nor rain nor heat nor gloom of night stays these autonomous beast from the swift completion of their appointed trips."

Maybe go with a Cybertruck in a snow storm picture next to it.