Primer on SAE Levels of Autonomy

[EVNow]

That's wrong though. Design intent refers to the actual design of the software / ADS. If the design is within the level specification, then it's that level, regardless of performance. That's the point of the design-intent passage.

Yes - specifically, Tesla doesn't yet - by design - handle a lot of things. So can't be L5.

Actually none of them - since they are all atleast geo-fenced to certain countries.

 

[Daniel in SD]

That's wrong though. Design intent refers to the actual design of the software / ADS. If the design is within the level specification, then it's that level, regardless of performance. That's the point of the design-intent passage.

Why do they use the word "intent" then?

The level of a driving automation system feature corresponds to the feature’s production design intent. This applies regardless of whether the vehicle on which it is equipped is a production vehicle already deployed in commerce, or a test vehicle that has yet to be deployed. As such, it is incorrect to classify a level 4 design-intended ADS feature equipped on a test vehicle as level 2 simply because on-road testing requires a test driver to supervise the feature while engaged, and to intervene if necessary to maintain safe operation.

 

[powertoold]

Why do they use the word "intent" then?

You have to read and absorb the definition as a whole to fully understand it. If you take things out of context, it's a lot of mumbo jumbo'ed ideas that contradict each other.

Here's some more context wrt design intent:

The level assignment rather expresses the
design intention for the feature and as such tells potential users or other interested parties that the feature can be expected to function such that the roles of the user versus the driving automation system while the feature is engaged are consistent with the assigned level, as defined in this document. The level assignment is typically based on the manufacturer’s knowledge of the feature’s/system’s design, development, and testing, which inform the level assignment. An ADS feature’s capabilities and limitations are expected to be communicated to prospective users through various means, such as in an owner’s manual, which explains the feature in more detail, such as how it should and should not be used, what limitations exist (if any), and what to do (if anything) in the event of a DDT performance-relevant system failure in the driving automation
system or vehicle.


Tldr currently fsd beta is level 2. In no universe or interpretation can it be level 5.

 

[diplomat33]

That's wrong though. Design intent refers to the actual design of the software. If the design is within the level specification, then it's that level, regardless of performance. That's the point of the design-intent passage.

Mahmood is an engineer who works with autonomous vehicles. He literally works with this every day. I would trust him to be right over you.

 

[powertoold]

I would trust him to be right over you.

No problem, I'm backed by logic and the definition itself. If you guys want to believe fsd beta is level 5, then sobeit. I can start claiming that Tesla is the first to achieve level 5 lol

 

[diplomat33]

No problem, I'm backed by logic and the definition itself. If you guys want to believe fsd beta is level 5, then sobeit. I can start claiming that Tesla is the first to achieve level 5 lol

You misunderstand. Nobody is saying that FSD beta is L5. Mahmood is simply saying that based on some interpretations of "design intent" that in theory, Tesla could try to argue that it is L5. But it would be disputed.

 

[powertoold]

You misunderstand. Nobody is saying that FSD beta is L5. Mahmood is simply saying that based on some interpretations of "design intent" that in theory, Tesla could try to argue that it is L5. But it would be disputed.

There's no dispute in a taxonomy or engineering definition. Fsd beta is designed to require a fallback driver. If we say it's level 5, it would contradict my quote from the definition above.

Any belief we have about the definition needs to be internally consistent with every passage in the definition.

I would go on to say that an engineer who works on AVs misunderstanding the definition shows how useless the definition is.

 

[Daniel in SD]

No problem, I'm backed by logic and the definition itself. If you guys want to believe fsd beta is level 5, then sobeit. I can start claiming that Tesla is the first to achieve level 5 lol

You can claim that Tesla has made the first vehicle with Level 5 production design intent.
Tesla defines "achieve" as a safety level twice as good as a human driver. I think that makes sense and I don't think they've done that.

 

[Daniel in SD]

There's no dispute in a taxonomy or engineering definition. Fsd beta is designed to require a fallback driver. If we say it's level 5, it would contradict my quote from the definition above.

Any belief we have about the definition needs to be internally consistent with every passage in the definition.

I would go on to say that an engineer who works on AVs misunderstanding the definition shows how useless the definition is.

How is that any different from a Cruise vehicle with a safety driver?

 

[diplomat33]

There's no dispute in a taxonomy or engineering definition. Fsd beta is designed to require a fallback driver. If we say it's level 5, it would contradict my quote from the definition above.

Any belief we have about the definition needs to be internally consistent with every passage in the definition.

Look. I agree that FSD beta is not currently L5.

The point that Mahmood is making is that some argue over the meaning of "production design intent". Does "production design intent" refer to the current design or the intended design when it goes into production? If we say that it refers to the current design then FSD Beta is clearly not L5. But if we say that "production design intent" refers to the intent of the finished product, then we could possibly argue that FSD beta is "L5 in development" since Tesla intends the finished product to be L5.

The fact is that Tesla intends the finished product to be L5 so Tesla could try to say that FSD beta is "L5 in development" if they wanted to. They don't because then they would be subject to the CA DMV regulations.

 

[powertoold]

"Production design intent" appears only twice in the definition:

The level of a driving automation system feature corresponds to the feature’s production design intent. This applies regardless of whether the vehicle on which it is equipped is a production vehicle already deployed in commerce, or a test vehicle that has yet to be deployed. As such, it is incorrect to classify a level 4 design-intended ADS feature equipped on a test vehicle as level 2 simply because on-road testing requires a test driver to supervise the feature while engaged, and to intervene if necessary to maintain safe operation.


Here's what it means:

The developer already designed the feature to be level 4 (fulfills all specifications for level 4 as defined), but the software is still in development, so the developer (not the feature itself) requires a safety driver for road testing. That's it. Notice the definition is careful to say road testing requires a safety driver, not the feature requires... The developer defines the requirements for road testing, not the feature itself, because the feature is already level 4, as stated in the wording.

 

[Bitdepth]

Here's what it means:

the developer already designed the feature to be level 4 (fulfills all specifications for level 4 as defined), but the software is still in development, so the developer (not the feature itself) requires a safety driver for road testing. That's it.

Is this not what several people explained to you in the other thread when you tried to erroneously claim that a car that still needs a safety driver is not level 4?

Further, something that is "capable" of L4 isn't yet L4, as the Waymo driverless cars in Chandler "are" L4. Let's wait until Mobileye announces that they don't require safety drivers to say they're L4.

Nah, you don't understand what design intent means here. Even if the intent is L4 (i.e., eventually when the software matures), but there is still expectation that the driver take over, it's L2 or L3 max. Design intent relates to the design of the feature itself. A feature is level 4 if the software designers intend it to be so, by following the requirements in the definition. This means you can have a very poor performing L4 feature, as long as the designers (software developers) intended it to be L4.

If a feature requires a safety (fallback) driver, then it's not level 4, regardless of what the developer intends for it to be in the future.
Let's wait until Mobileye announces that they don't require safety drivers to say they're L4.

 

[powertoold]

Yes - specifically, Tesla doesn't yet - by design - handle a lot of things. So can't be L5.

This is not true either!

The main reason fsd beta is level 2 is because the ADS itself requires a fallback driver:

Wheel tug
DMS
Seat sensor

If Tesla disabled all the fallback driver aspects of fsd beta, then it'd be a very dangerous level 5 feature, but level 5 nonetheless.

Whether or not a ADS satisfactorily executes the entire DDT is relevant to its level. The whole term DDT is just the SAE's attempt at breaking down the task of driving. What we simply call "driving" is broken down into different tasks. Fsd beta can clearly drive and execute the entire DDT in a sustained manner.

 

[EVNow]

If Tesla disabled all the fallback driver aspects of fsd beta, then it'd be a very dangerous level 5 feature, but level 5 nonetheless.

Whether or not a ADS satisfactorily executes the entire DDT is relevant to its level. The whole term DDT is just the SAE's attempt at breaking down the task of driving. What we simply call "driving" is broken down into different tasks. Fsd beta can clearly drive and execute the entire DDT in a sustained manner.

Seems to be not a lot of features are yet designed for. Ofcourse FSD Beta can't drive in most countries now ... so is geofenced.

Wiki - Tracking FSD Feature Complete

Background : In the Ark podcast Musk said they are planning to be feature complete by the end of the this year (2019). Feature complete is a specific software term that means set of features that are planned for a particular project / release have been coded & tested to be working to meet the...

teslamotorsclub.com

ps : I've always argued the SAE levels are worthless because the "standards" have really not been defined. Its vague and left to the companies to decide.

 

[Daniel in SD]

Here's what it means:

the developer already designed the feature to be level 4 (fulfills all specifications for level 4 as defined), but the software is still in development, so the developer (not the feature itself) requires a safety driver for road testing. That's it.

Fine. I don't think it needs to be feature complete for the production design intent to be L5. What difference does it make whether FSD Beta is L2 or L5? None, because it's a beta.

The main reason fsd beta is level 2 is because the ADS itself requires a fallback driver:

Wheel tug
DMS
Seat sensor

I think all L4 test vehicles now have safety driver monitoring (because of the Uber incident). Making sure the safety driver is alert becomes more important the more capable the system becomes.

https://twitter.com/x/status/1439042334155497474

 

[powertoold]

Ofcourse FSD Beta can't drive in most countries now ... so is geofenced.

It's not geofenced. It just performs very poorly as shown in the Ukraine video. I'd personally perform poorly in Ukraine as well, and I'm a level 5 human!

 

[Daniel in SD]

Seems to be not a lot of features are yet designed for. Ofcourse FSD Beta can't drive in most countries now ... so is geofenced.

A system geo-fenced to single country can still be Level 5.

Practical Considerations Regarding Level 5
There are technical and practical considerations that mitigate the literal meaning of the stipulation that a level 5 ADS must be capable of ‘operating the vehicle on-road anywhere that a typically skilled human driver can reasonably operate a conventional vehicle,’ which might otherwise be impossible to achieve. For example, an ADS-equipped vehicle that is capable of operating a vehicle on all roads throughout the US, but, for legal or business reasons, cannot operate the vehicle across the border in Canada or Mexico can still be considered level 5, even if geo-fenced to operate only within the US. The rationale for this exception is that the geo-fenced limitation (i.e., US, only) is not due to limitations on the technological capability of the ADS, but rather is due to legal or business constraints, such as legal restrictions in Canada and Mexico/Central America that prohibit level 5 deployment, or the inability to make a business case for expansion to those markets.

 

[Bitdepth]

This is not true either!

The main reason fsd beta is level 2 is because the ADS itself requires a fallback driver:

Wheel tug
DMS
Seat sensor

If Tesla disabled all the fallback driver aspects of fsd beta, then it'd be a very dangerous level 5 feature, but level 5 nonetheless.

Whether or not a ADS satisfactorily executes the entire DDT is relevant to its level. The whole term DDT is just the SAE's attempt at breaking down the task of driving. What we simply call "driving" is broken down into different tasks. Fsd beta can clearly drive and execute the entire DDT in a sustained manner.

FSD beta is not level 5 because Tesla says so, not because of wheel tug, or DMS or Seat Sensor. The levels are determined by the manufacturers of the system based on their evaluation of the system and it's capabilities as it pertains to the role the human plays when the system is engaged.

J3016 (p30)

As discussed above, the level of driving automation is based on the functionality of the driving automation system feature, as determined by an allocation of roles in DDT and DDT fallback performance between that feature and the (human) user (if any). The manufacturer of a driving automation system feature determines that feature’s requirements, operational design domain (ODD), and operating characteristics, including the level of driving automation, as defined below. The manufacturer also defines the proper use of that feature.

(p36)

By itself, this document imposes no requirements, nor confers or implies any judgment in terms of system performance. Therefore, while it may be appropriate to state, for example, that a given ADS feature does not meet the definition of Level 4 because it occasionally relies on a remote fallback-ready user to perform the fallback (and is therefore a Level 3 feature), it is not appropriate to conclude that the feature in question is therefore “non-compliant” or “unsafe.”

8.2 Levels are Assigned, Rather than Measured, and Reflect the Design Intent for the Driving Automation System Feature as Defined by its Manufacturer

 

[powertoold]

L4 test vehicles now have safety driver monitoring (because of the Uber incident)

That's why only Waymo and Cruise have confirmed level 4. They have videos of their ADS enabled on public roads without a driver in the seat. All other companies claiming to have level 4 "capable" cars shouldn't be trusted yet.

 

[EVNow]

A system geo-fenced to single country can still be Level 5.

I'd say inability of FDS Beta to operate in most countries is not because of this ... but a technical limitation that the CNN has not been trained to recognize those roads, signs nor the planning software designed to obey local laws.

The rationale for this exception is that the geo-fenced limitation (i.e., US, only) is not due to limitations on the technological capability of the ADS, but rather is due to legal or business constraints,