Primer on SAE Levels of Autonomy

[diplomat33]

Just wanted to share Dr. Koopman's excellent primer on the SAE levels where he also busts some common myths:

SAE J3016 User Guide

I like this expanded table that shows what the driver and what the ADS do for each level:

 

[jebinc]

Just wanted to share Dr. Koopman's excellent primer on the SAE levels where he also busts some common myths:

SAE J3016 User Guide

I like this expanded table that shows what the driver and what the ADS do for each level:

@diplomat33 Thanks for keeping this thread fresh and current!

 

[diplomat33]

@diplomat33 Thanks for keeping this thread fresh and current!

You are very welcome. I saw Dr. Koopman's primer in my Twitter feed this morning and immediately thought that it would be perfect to share in this primer thread.

 

[jebinc]

You are very welcome. I saw Dr. Koopman primer in my Twitter feed this morning and immediately thought that it would be perfect to share in this primer thread.

@diplomat33 Been hanging out in the S forum lately; are you still driving a 3? Hope all is well.

 

[diplomat33]

@diplomat33 Been hanging out in the S forum lately; are you still driving a 3? Hope all is well.

Hi! I am doing well, thank you. Hope you are also doing well.

Yes, I am still driving a Model 3. I still love the car. It's fun and very easy to drive. It's been very reliable so far. In 3 years of ownership, I've only had 2 issues that required service: some condensation in a tail light and a side repeater camera that needed changing. Both issues were resolved easily and for free by mobile service.

 

[jebinc]

Hi! I am doing well, thank you. Hope you are also doing well.

Yes, I am still driving a Model 3. I still love the car. It's fun and very easy to drive. It's been very reliable so far. In 3 years of ownership, I've only had 2 issues that required service: some condensation in a tail light and a side repeater camera that needed changing. Both issues were resolved easily and for free by mobile service.

@diplomat33 My 3 is/was wonderful. Zero issues. Only SC trip was to replace rear glass that I cracked during a trip to Costco... If I had a four car garage, I wouldn't have traded it on the Plaid, but rather would have kept it. The car was/is perfect. Tesla gave me nearly what I paid for it in trade ($51,800, plus $5,200 tax benefit), so that made things a little less painful.

Edit: Wait, I also had one mobile visit; to replace the charging port as a pin broke.

 

[diplomat33]

Here are 3 helpful tutorial videos:

What is SAE J3016:

What is DDT:

What is ODD:

 

[diplomat33]

 

[powertoold]

This guy understands the levels pretty well. I think he should have also addressed the other misconceptions about the levels:

1) Role of remote assistance and what it means to drive / control the vehicle
2) The levels apply solely to the software design / feature, not the vehicle or developer's role in restricting its use

The best video IMO is the first one, "What is .... for?," where he says that the levels only describe the responsibilities of driver and software in executing the sustained DDT. That's the main misconception and why the levels are useless.

 

[Daniel in SD]

This guy understands the levels pretty well. I think he should have also addressed the other misconceptions about the levels:

1) Role of remote assistance and what it means to drive / control the vehicle
2) The levels apply solely to the software design / feature, not the vehicle or developer's role in restricting its use

The best video IMO is the first one, "What is .... for?," where he says that the levels only describe the responsibilities of driver and software in executing the sustained DDT. That's the main misconception and why the levels are useless.

The SAE levels are included in AV regulations here in California so they’re useful if you want to operate an AV here. I’m guessing it’s the same for other jurisdictions.
I definitely agree with him that they should have called the levels classes and not used numbers.
I still have no idea what you’re referring to in your second misconception. Can you give a real world example? Is this a hypothetical or is it something that has actually happened?

 

[powertoold]

I still have no idea what you’re referring to in your second misconception. Can you give a real world example? Is this a hypothetical or is it something that has actually happened?

Here's how I'd explain the common misconception:

The levels apply solely to the software design / feature (aka, "driving automation system", "ADS").

The levels have no performance criteria for executing the DDT on a sustained basis. The levels also don't define a minimum time to be "sustained." They only say that automatic emergency braking systems aren't "sustained."

Putting it together: the levels simply describe how the roles of the driver and ADS are designed within the software feature. If the software feature is designed to not require a fallback driver, then it can be level 4 or 5 (as long as it can perform the DDT on a sustained basis, which again, there's no minimum time defined for "sustained"). It doesn't matter how good or bad the software is at performing the DDT... the levels' purpose is solely to describe ADS autonomy, not performance. A stupid analogy would be animal species. A newborn baby is a human, and a 90 yr old is also a human, but we know they're very different in their capabilities or experience, etc.

As for the developer, I've said this before, but what the developer markets or says publicly about their ADS doesn't matter at all. If a developer says their ADS is level 5, but the software requires a fallback driver, then it's level 3 max. It doesn't matter if they're testing it or not. Again, if the software requires a fallback driver, the feature can't be level 4/5, no matter what! The levels apply solely to the ADS / software, not how the developer markets or describes their system.

 

[Daniel in SD]

Here's how I'd explain the common misconception:

The levels apply solely to the software design / feature (aka, "driving automation system", "ADS").

The levels have no performance criteria for executing the DDT on a sustained basis. The levels also don't define a minimum time to be "sustained." They only say that automatic emergency braking systems aren't "sustained."

Putting it together: the levels simply describe how the roles of the driver and ADS are designed within the software feature. If the software feature is designed to not require a fallback driver, then it can be level 4 or 5 (as long as it can perform the DDT on a sustained basis, which again, there's no minimum time defined for "sustained"). It doesn't matter how good or bad the software is at performing the DDT... the levels' purpose is solely to describe ADS autonomy, not performance. A stupid analogy would be animal species. A newborn baby is a human, and a 90 yr old is also a human, but we know they're very different in their capabilities or experience, etc.

As for the developer, I've said this before, but what the developer markets or says publicly about their ADS doesn't matter at all. If a developer says their ADS is level 5, but the software requires a fallback driver, then it's level 3 max. It doesn't matter if they're testing it or not. Again, if the software requires a fallback driver, the feature can't be level 4/5, no matter what! The levels apply solely to the ADS / software, not how the developer markets or describes their system.

If a developer were to misrepresent the capability of their system then wouldn't they lose their right to operate on public roads? How can the SAE address this problem?
It is true that there is no SAE standard for the performance of automation systems. One issue is that performance is very hard to quantify and has many dimensions. Another issue is that the required performance is a societal question. It's just not part of the scope of J3016. Perhaps in the future there will be some sort of performance standard.

 

[diplomat33]

The best video IMO is the first one, "What is .... for?," where he says that the levels only describe the responsibilities of driver and software in executing the sustained DDT. That's the main misconception and why the levels are useless.

Stop saying that the SAE levels are useless. They are not useless. Defining the responsibility of the driver and the software with respect to the sustained DDT is important and very useful. They just don't do what you care about. But that does not make them useless. Performance is outside the scope of the SAE J3016. You can't fault a standard for not addressing something that is outside it's scope! If you are more interested in reading about safety metrics, then read "Best Practice for Metrics and Methods for Assessing Safety Performance of an Automated Driving System (ADS)". I have attached the file for your convenience.

 

[powertoold]

Stop saying that the SAE levels are useless. They are not useless. Defining the responsibility of the driver and the software with respect to the sustained DDT is important and very useful. They just don't do what you care about.

When it comes to autonomous driving, all we care about is performance.

Likewise, knowing someone is "human" is useless. You want to know what they've done, what they're good at, what kind of person they are, especially if you're gonna hire them to do something for you (like driving).

 

[diplomat33]

When it comes to autonomous driving, all we care about is performance.

Wrong. Performance is important of course but it is not the only thing we care about. We also care about what kind of autonomous driving it is. Does it require a human fall-back? Can it perform the entire DDT? What is the ODD? These are important questions too. These are the questions that SAE J3016 tries to answer.

If all I know is performance but I don't know if the system requires a fall-back, I am missing vital piece of information. If I am choosing an ADS, it is pretty important to know if I need to do some of the driving tasks, if I need to be the fall-back, where/when can it be used. For example, I don't want to pick a ADS and read a book while it is driving when I am not supposed to, because I did not know that I was supposed to pay attention to the road. That is why J3016 is useful.

Likewise, knowing someone is "human" is useless. You want to know what they've done, what they're good at, what kind of person they are, especially if you're gonna hire them to do something for you (like driving).

You need to know if it is a human or a dog first. That why we have species taxonomies. Likewise, SAE J3016 tells us the "species" of ADS. It tells us if it is a driver assist or an autonomous driving system. You need to know that before you know about performance.

And like I said, there will be other standards that will address the questions of safety and performance that you care about. They are very important questions. They are just outside the scope of J3016. It is not fair to call J3016 "useless", just because it doesn't address questions that are outside its scope.

 

[powertoold]

Also doesn't help that few people here understand the levels, and many kept arguing that I don't know what I'm talking about lol. It just shows that the definition is difficult to understand, even though it's made to seem simple.

 

[Bitdepth]

Also doesn't help that few people here understand the levels, and many kept arguing that I don't know what I'm talking about lol. It just shows that the definition is difficult to understand, even though it's made to seem simple.

You say you understand it yet you don't understand why J3016 exists. It does not exist to determine the capability of the ADAS or ADS system. Its sole purpose is to define the role of the primary actor when an ADS is engaged. Anything else is outside the scope of what they are trying to define. The first 7 pages of that document reiterates it. You can't set a standard of how advanced something is in this field that is still rapidly changing, that is why disengagement reports are required to show how long (sustained) a given system drives before it needs human intervention. You know, kind of like the sustained control you keep harping on about, but you don't like disengament reports either because somehow everyone is faking it. The language is carefully chosen.

sus·tained
/səˈstānd/
adjective
continuing for an extended period or without interruption.

synonyms
constant · continued · continuous · uninterrupted

How sustained an ADS remains engaged is not dependent on the level assigned to a system. A sufficiently advanced L2 ADAS can remain engaged for almost 90% of a planned trip without needing a human intervention. In that same regard, a sufficiently advanced L3 traffic jam system can remain engaged for however long the traffic persists within its ODD.

The confusion comes in when staked parties try to obfuscate and make promises and sell products that don't exist. Kind of like how Elon Musk has been claiming for years that autonomous driving has been solved and only needs regulatory approval. Nobody else is selling said capability because they all understand that they have not yet reached a level that a lay person can understand as a finished vehicle driving automation system that perform part or all of the dynamic driving task on a sustained basis at a reliable safety level. The only confused people are those who cling to the word of Elon Musk and believed Autopilot was just an update creep away from achieving L5, pending regulatory approval.

 

[Daniel in SD]

When it comes to autonomous driving, all we care about is performance.

Sure, but first you have to categorize and define terms for vehicle automation systems. That's what SAE J3016 does.

And you didn't answer my question regarding developer misrepresenting the SAE level of their system.

As for the developer, I've said this before, but what the developer markets or says publicly about their ADS doesn't matter at all. If a developer says their ADS is level 5, but the software requires a fallback driver, then it's level 3 max. It doesn't matter if they're testing it or not. Again, if the software requires a fallback driver, the feature can't be level 4/5, no matter what! The levels apply solely to the ADS / software, not how the developer markets or describes their system.

If a developer were to misrepresent the capability of their system then wouldn't they lose their right to operate on public roads? How can the SAE address this problem?

 

[diplomat33]

Short answer: L4 and L5 can still have safety drivers. The presence of a safety driver does not downgrade a system to L2.

Interestingly, he says that, in theory, Tesla actually could say that FSD Beta is L5 based on some interpretations of "design intent".

I would argue that the role of the safety driver is very different in L2 and L4/5. In L2, the safety driver is a driver. They need to perform some or all of the driving tasks. In L4/5, the safety driver does not need to perform any driving tasks. They are more of a supervisor for the purposes of testing and development. So they should intervene for safety reasons but their primary role is to provide feedback to the development team on what needs to be improved in the software.

I would argue that the term "safety driver" is confusing for L4/L5 since they don't require a driver. Perhaps that is why Waymo and others do not use the term "safety driver". Waymo uses the term "autonomous specialist". Sounds fancier too.

 

[powertoold]

Interestingly, he says that, in theory, Tesla actually could say that FSD Beta is L5 based on some interpretations of "design intent".

That's wrong though. Design intent refers to the actual design of the software / ADS. If the design is within the level specification, then it's that level, regardless of performance. That's the point of the design-intent passage.