Autonomous Car Progress

[EVNow]

There is a big difference between L3 and L4. L3 generally does not need as much testing as L4.

Whether it is L3 or L4 - you need to figure out the disengagement rate. Yes, L4 needs more than L3 - because rates for L4 are lower than L3. But you still need a lot of miles .... and not "5 minutes" of testing or testing on a testing track.

For example, a disengagement rate of say 1 per 100 miles would be terrible for L4 but could be perfectly acceptable for L3 if every disengagement was a safe one where the driver took over correctly.

1 in 100 miles would be terrible for L3 and will result in a LOT of accidents. People are expecting the car to drive atleast as well as a human would - and humans don't have 1 accident in 100 miles on a highway.

 

[diplomat33]

The problem is - the "tests" mentioned are not comprehensive. To properly test
(1) you need to figure out every permutation of scenarios and test them OR

4.1.2 is simply mentioning one specific test scenario. Yes, you would need to do a lot of permutations for each scenario. That is why you would not do just one 5 mn test run, you would do thousands of test runs for each scenario. The reg is simply recommending that each individual test run should be a minimum of 5 minutes. It could certainly be more than 5 minutes. And you would do many test runs for each scenario. The total testing would be far more than 5 minutes.

And AV companies like Waymo do comprehensive testing that includes thousands of hours of testing on tracks + billons of miles of simulations + millions of miles of real world driving. It is not just one test.

(2) you just test for millions of miles, because statistically you would have covered a lot of real world scenarios

Since (1) is not possible, responsible companies (which includes every company testing in CA, for eg.) test in real world conditions for millions of miles.

Yes, the responsible companies do millions of miles of real world testing. They don't do just 5 minutes of testing. Respectfully, I think your claim that the reg only calls for 5 minutes of testing is a strawman.

 

[diplomat33]

Whether it is L3 or L4 - you need to figure out the disengagement rate. Yes, L4 needs more than L3 - because rates for L4 are lower than L3. But you still need a lot of miles .... and not "5 minutes" of testing or testing on a testing track.

Again, nobody is suggesting 5 minutes of testing. The reg 4.1 is not suggesting only doing 5 minutes of total testing.

1 in 100 miles would be terrible for L3 and will result in a LOT of accidents. People are expecting the car to drive atleast as well as a human would - and humans don't have 1 accident in 100 miles on a highway.

That is "L4" thinking. You are thinking about a system that is expected to handle all driving without human intervention. So yes, L4 should drive at least as well as a human and not get into accidents when there is a "disengagement". But remember L3 asks the driver to take over on purpose. It is part of the design. The driver is expecting to take over. L3 does not need to drive as well as the human for long periods of time, it only needs to be handle a certain ODD and ensure the driver can take over when prompted. 1 in 100 miles would not necessarily mean an accident every 100 miles for L3. It could just mean that every 100 miles on average, the car asks the driver to take over without incident. In L3, many disengagements are simply proper hand overs back to the driver. If the driver takes over properly, there would not be an accident for every disengagement.

 

[Daniel in SD]

Disengagement rate is a useless metric for testing both L3 and L4 because humans will disengage even when it is not necessary. There are many scenarios where it is not safe to wait and see whether the system will be able to handle the situation.
Anyway, no one is going to use 500M miles of testing before they deploy. Everyone just extrapolates and hopes they're right.

 

[Doggydogworld]

1 in 100 miles would be terrible for L3 and will result in a LOT of accidents. People are expecting the car to drive atleast as well as a human would - and humans don't have 1 accident in 100 miles on a highway.

You can't equate disengagements and accidents that way.

Highway traffic jam fender benders are extremely common. We don't know exactly how common because many are non-reportable -- just exchange insurance cards and drive on. Let's say one happens every 200k miles. 10x that without an accident would be unlikely to be random chance, 100x almost impossible. So figure 2-20m miles of testing without an accident.

The human factors part is the hardest to test. You can't test how a movie-watching driver will respond in a real car in real traffic, only in simulators.

The term disengagement isn't very useful for L3. The car requests handover more often than every 100 miles, of course, because you reach your exit or the end of the mapped area, or traffic speed increases beyond 60 kph, etc. Those aren't edge cases. The issue is whether the system can avoid other cars suddenly stopping or veering into their lane and such. Does it detect if the lane is temporarily closed and request handover far enough in advance. Does it avoid road debris? Does it phantom-brake every five miles? System failure rate is the issue, not disengagement rate.

 

[Bitdepth]

I've quoted it in this thread ! How hard is it to search ?

If they find 64 pages "demanding" - how demanding would 500 Million miles of testing be ?



View attachment 801397

That does not say it requires 5 minutes of testing, it says testing a given scenario as outlined in B, C, and D to assess the performance of the ALKS must last a minimum of 5 minutes. Several permutations of that will be done to make sure it is repeatable by the manufacturer. That is why a company like Mercedes can confidently take liability in case an accident occurs while it is under the control of their L3 automated driving system.

These are some of the testing scenarios outlined on pages 60 - 64 in action.

 

[EVNow]

That does not say it requires 5 minutes of testing, it says testing a given scenario as outlined in B, C, and D to assess the performance of the ALKS must last a minimum of 5 minutes.

LOL. Read that again.

ps: In case you are being pedantic, obviously they won’t say you have to test for max of 5 minutes. Duh.

 

[EVNow]

You can't equate disengagements and accidents that way.

Highway traffic jam fender benders are extremely common.

With OEM testing you can - anyway they can remove non-serious disengagements (in highway traffic jam, what would they be, anyway).

Highway fender benders are not that common at all. You can check that in the Virginia tech reports. They are much better than city fender benders at 1 every 10k miles.

Would you be happy with a 1 in 100 mile fender bender L3, like @diplomat33 ? Definitely not me.

Anyway - I find it so ironic that in a Tesla forum, some people only trust legacy ICE makers. They don’t have to test much, they always tell the truth etc. etc.

 

[Bitdepth]

LOL. Read that again.

ps: In case you are being pedantic, obviously they won’t say you have to test for max of 5 minutes. Duh.

I am not being pedantic, you are just not comprehending what you are reading. Read the documents again and then again. It is a battery of tests to assess the capability of the system. It is not a single 5 minutes test. Pages 60 through 64 list all the scenarios and test parameter variations that need to be tested. How are all those going to fit within a single 5 minutes test time as you incorrectly state?

What the document states, and is easy to understand is that when a given scenario or combination thereof is being assessed, the test must be done with a minimum test duration of 5 minutes. Meaning that the test may not be executed less than 5 minutes into a given scenario that is being assessed. Moreover, the manufacturer is obliged to provide documentation of their own internal testing to be used to assess their system.

Duration: the time during which something continues or lasts.


Test Specifications for ALKS
1. Introduction
This annex defines tests with the purpose to verify the technical requirements
on ALKS.
Until such time that specific test provisions have been agreed, the Technical
Service shall ensure that the ALKS is subject to at least the tests outlined in
Annex 5. The specific test parameters for each test shall be selected by the
Technical Service and shall be recorded in the test report in such a manner that
allows traceability and repeatability of the test setup.
Pass- and Fail-Criteria for tests are derived solely from the technical
requirements in paragraphs 5 to 7 of the Regulation. These requirements are
worded in a way that they allow the derivation of pass-fail-criteria not only for
a given set of test parameters, but for any combination of parameters in which
the system is designed to work (e.g. operating speed range, operating lateral
acceleration range, curvature range as contained in the system boundaries).
The test specifications in this document are meant to be a minimum set of tests,
the technical service authorities may perform any other test within the system
boundaries and may then compare the measured results against the
requirements (concrete: expected test outcome).

3.2. Test parameter variation
The manufacturer shall declare the system boundaries to the Technical Service.
The Technical Service shall define different combinations of test parameters
(e.g. present speed of the ALKS vehicle, type and offset of target, curvature of
lane) in order to cover scenarios in which a collision shall be avoided by the
system as well as those in which a collision is not expected to be avoided,
where applicable.

If this is deemed justified, the Technical Service may test additionally any
other combination of parameters.

If a collision cannot be avoided for some test parameters, the manufacturer
shall demonstrate either by documentation or, if possible, by
verification/testing that the system doesn’t unreasonably switch its control
strategy.

How can anyone read all this and conclude that they only require a 5 minutes test to validate and approve the system?

 

[Daniel in SD]

Anyway - I find it so ironic that in a Tesla forum, some people only trust legacy ICE makers. They don’t have to test much, they always tell the truth etc. etc.

Why do you think the same people wouldn’t trust a Tesla L3 system?
I 100% believe that Mercedes thinks their system is safe because if it’s not there’s no way to avoid liability. If I had a vehicle equipped with it I would use it. However, if it had a top speed of 70mph I would wait until others had used it for 100 million miles just to be safe. I don’t trust standardized testing of AVs (or any ADAS either).

 

[diplomat33]

Would you be happy with a 1 in 100 mile fender bender L3, like @diplomat33 ? Definitely not me.

I never said I would be ok with 1 in 100 miles fender benders. Please do not misrepresent what I said. I was very clear that I was talking about 1 hand over back to the driver without incident every 100 miles.

 

[diplomat33]

Anyway - I find it so ironic that in a Tesla forum, some people only trust legacy ICE makers. They don’t have to test much, they always tell the truth etc. etc.

Uh, nobody said that ICE makers always tell the truth or that they don't have to test much. That's a strawman.

Companies like Mercedes releasing L3 would do a LOT of testing. You are the one that keeps claiming falsely they only do 5 minutes of testing based on a misreading of the reg.

 

[nvx1977]

I don't intend to be a concern troll here (I will admit I still know very little about how Merc's L3 system works), but I worry in general about L3, people's understanding of what it actually is, and the reaction time required to take over in a situation where the system needs help. Specifically, if people have fallen for the marketing that L3 is more proficient at operating autonomously in its ODD than another L2 system (which is not guaranteed), they will be even less situationally aware at any moment, so the time it takes them to get focused on a driving task can be significantly lengthened, esp during a system failure where a takeover is required.

Now, people have been making similar arguments about Tesla's L2 systems stemming from their naming (autopilot, FSD), and from my extensive personal experience using both, I think it's overblown, so maybe it is the same with L3 systems. But this really is uncharted territory. If L3 starts getting a bad reputation for being more unsafe than L2, I think that's a good thing. L3 is this fuzzy area between driver assist and full autonomy. Maybe we need it as a transitional state, but the quicker we transition past it, the better.

 

[diplomat33]

Specifically, if people have fallen for the marketing that L3 is more proficient at operating autonomously in its ODD than another L2 system (which is not guaranteed), they will be even less situationally aware at any moment, so the time it takes them to get focused on a driving task can be significantly lengthened, esp during a system failure where a takeover is required.

If the L3 is designed properly, it will be better at operating autonomously in its ODD than L2. Remember that L2 is not autonomous ever, L3 is autonomous in its ODD (but with the driver as back-up).

Also, L3 is not like what we get with Tesla's AP/FSD where out of the blue, we can get a red hands on wheel alert and we need to take over immediately. If the L3 is designed properly, the driver will have enough time to resume control. The UI should be intuitive so the driver knows what is going on, and the system will give proper escalating alerts to take over, using haptic feedback, audio and visual alerts. So it will never be a case of suddenly, "oh crap, I need to take over now and I wasn't ready."

L3 is this fuzzy area between driver assist and full autonomy. Maybe we need it as a transitional state, but the quicker we transition past it, the better.

I suspect a few companies like Mercedes will deploy L3, mostly to get bragging rights. But I think L3 will be limited in its usage. You are right that L3 is a sort of transitional level and you've probably noticed that AV companies are focusing exclusively on L4. That's because the real goal is autonomous driving that does not require a human driver at all. So L4 and, eventually L5 some day, is the real objective.

 

[nvx1977]

Also, L3 is not like what we get with Tesla's AP/FSD where out of the blue, we can get a red hands on wheel alert and we need to take over immediately. If the L3 is designed properly, the driver will have enough time to resume control. The UI should be intuitive so the driver knows what is going on, and the system will give proper escalating alerts to take over, using haptic feedback, audio and visual alerts. So it will never be a case of suddenly, "oh crap, I need to take over now and I wasn't ready."

I agree that the implementation can and should give ample warning where possible. Just thinking that accidents occur suddenly and when least expected, so even if the car tries its best to give ample warning, sometimes the real world only gives you a split second to avoid.

So it's better if the car learned to avoid, rather than try to hand over to human. This is why I really hope "bragging rights" don't suck a lot of manufacturers into this L3 realm. IMO it's the most unsafe level of autonomy.

 

[diplomat33]

I agree that the implementation can and should give ample warning where possible. Just thinking that accidents occur suddenly and when least expected, so even if the car tries its best to give ample warning, sometimes the real world only gives you a split second to avoid.

So it's better if the car learned to avoid, rather than try to hand over to human. This is why I really hope "bragging rights" don't suck a lot of manufacturers into this L3 realm. IMO it's the most unsafe level of autonomy.

L3 should handle the "split second" cases on its own without human intervention. The only time L3 should ask the driver to take over is for the cases where there is plenty of time. For example, a cyclist suddenly cutting in is a case L3 should handle on its own, approaching the off ramp exit is a case where the L3 will hand it back to the driver. That is another reason why Mercedes can say that the driver does not need to supervise since they have validated that the L3 can handle the split second cases and will only ask the driver to take over when there is time. At least that is the idea. We will see how well the L3 works in practice.

Note that the Mercedes L3 will have cameras, lidar and radar and HD maps, similar to other autonomous vehicles. That is precisely to help the car detect and react to other objects as reliably as possible. And the car will have sophisticated perception, prediction and planning to react to "split second" cases. Could the car still fail to react in time? Sure, just like any autonomous vehicle. But the whole point of having all those sensors and sophisticated perception, prediction and planning is to make the car as safe and reliable as possible.

If the L3 is built right where it is really reliable FSD that can safely handle situations on its own and it only asks the driver to take over when it is leaving the ODD and there is plenty of time for the driver to resume control, then I don't think L3 would be inherently less safe than any other SAE level.

 

[nvx1977]

If the L3 is built right where it is really reliable FSD that can safely handle situations on its own and it only asks the driver to take over when it is leaving the ODD and there is plenty of time for the driver to resume control, then I don't think L3 would be inherently less safe than any other SAE level.

But the SAE levels don't quantify the level of safety. Naturally a mfr wouldn't want to assume liability if their system is unsafe. But the lack of specificity here worries me. As a consumer, I'd rather know objectively how competent the system is rather than how much risk the mfr is willing to take.

 

[Daniel in SD]

But the SAE levels don't quantify the level of safety. Naturally a mfr wouldn't want to assume liability if their system is unsafe. But the lack of specificity here worries me. As a consumer, I'd rather know objectively how competent the system is rather than how much risk the mfr is willing to take.

How would they quantify safety? Maximum achievable and relative to human safety varies depending on where and when you’re driving.

 

[diplomat33]

Motional plans to launch their first driverless service with Lyft next year:

https://twitter.com/x/status/1524084396923498496

 

[EVNow]

I never said I would be ok with 1 in 100 miles fender benders. Please do not misrepresent what I said. I was very clear that I was talking about 1 hand over back to the driver without incident every 100 miles.

That is not the same.

Can you show me your math on how many miles to test so that handover happens correctly ?