Welcome to Tesla Motors Club
Discuss Tesla's Model S, Model 3, Model X, Model Y, Cybertruck, Roadster and More.
Register

Autonomous Car Progress

This site may earn commission on affiliate links.
I am not being pedantic, you are just not comprehending what you are reading. Read the documents again and then again. It is a battery of tests to assess the capability of the system. It is not a single 5 minutes test. Pages 60 through 64 list all the scenarios and test parameter variations that need to be tested. How are all those going to fit within a single 5 minutes test time as you incorrectly state?

What the document states, and is easy to understand is that when a given scenario or combination thereof is being assessed, the test must be done with a minimum test duration of 5 minutes. Meaning that the test may not be executed less than 5 minutes into a given scenario that is being assessed. Moreover, the manufacturer is obliged to provide documentation of their own internal testing to be used to assess their system.

Duration: the time during which something continues or lasts.


Test Specifications for ALKS
1. Introduction
This annex defines tests with the purpose to verify the technical requirements
on ALKS.
Until such time that specific test provisions have been agreed, the Technical
Service shall ensure that the ALKS is subject to at least the tests outlined in
Annex 5. The specific test parameters for each test shall be selected by the
Technical Service and shall be recorded in the test report in such a manner that
allows traceability and repeatability of the test setup.

Pass- and Fail-Criteria for tests are derived solely from the technical
requirements in paragraphs 5 to 7 of the Regulation. These requirements are
worded in a way that they allow the derivation of pass-fail-criteria not only for
a given set of test parameters, but for any combination of parameters in which
the system is designed to work (e.g. operating speed range, operating lateral
acceleration range, curvature range as contained in the system boundaries).
The test specifications in this document are meant to be a minimum set of tests,
the technical service authorities may perform any other test within the system
boundaries and may then compare the measured results against the
requirements (concrete: expected test outcome).


3.2. Test parameter variation
The manufacturer shall declare the system boundaries to the Technical Service.
The Technical Service shall define different combinations of test parameters
(e.g. present speed of the ALKS vehicle, type and offset of target, curvature of
lane) in order to cover scenarios in which a collision shall be avoided by the
system as well as those in which a collision is not expected to be avoided,
where applicable.

If this is deemed justified, the Technical Service may test additionally any
other combination of parameters.

If a collision cannot be avoided for some test parameters, the manufacturer
shall demonstrate either by documentation or, if possible, by
verification/testing that the system doesn’t unreasonably switch its control
strategy.


How can anyone read all this and conclude that they only require a 5 minutes test to validate and approve the system?
We all understand each test needs to be done for 5 minutes.

You would be happy if the test for lane keeping happens for 5 minutes but doesn’t take any variations in weather, traffic conditions, marking conditions, light conditions, curvatures, huge trucks next to you etc etc etc etc are not tested ?
 
We all understand each test needs to be done for 5 minutes.

You would be happy if the test for lane keeping happens for 5 minutes but doesn’t take any variations in weather, traffic conditions, marking conditions, light conditions, curvatures, huge trucks next to you etc etc etc etc are not tested ?
The regs say that any possible permutation you can dream up within the parameters MUST be tested, and the testers can add as many permutations of tests as they want to.

So all of those things you mentioned etc etc etc are covered in the regs.
 
That's not possible, unless the number of parameters is very low.
Not every possible permutation, any you can think up (basically).

"the technical service authorities may perform any other test within the system
boundaries"


If I'm running the tests to certify, I'm checking everything I can think of, all weather conditions, lighting, traffic, edge cases. I'm not going to be blamed for missing something.
 
  • Like
Reactions: diplomat33
We all understand each test needs to be done for a minimum duration of 5 minutes.
Fixed that.
You would be happy if the test for lane keeping happens for 5 minutes but doesn’t take any variations in weather, traffic conditions, marking conditions, light conditions, curvatures, huge trucks next to you etc etc etc etc are not tested ?
Is this a trick question? The document calls for and allows all of the above variables to be used in consideration when evaluating the system if it falls within the ODD of the vehicle. And the document even allows the tester to test things outside the ODD as defined by the manufacturer.

Moreover, this is not a discussion of what I personally would be happy or not be happy with. It is rather ironic for you to ask this question considering your willingness to allow what in my opinion is a poorly tested software that is prone to making a lot of mistakes, to drive you around. Autopilot runs under trucks, runs into stationary vehicles, can abruptly disable itself if it cannot handle a weather or road conditions, and runs into guardrails, yet you are willing to let it drive you around.
 
  • Like
Reactions: diplomat33
Is this a trick question? The document calls for and allows all of the above variables to be used in consideration when evaluating the system if it falls within the ODD of the vehicle. And the document even allows the tester to test things outside the ODD as defined by the manufacturer.
You aren't getting the most important problem.

It is not about "allowing" - its about "mandating".

Do we know whether Merc tested considering all those variables ? No .... because the standard "allows" - it doesn't mandate.

ps : Anyway, you can't test for all combinations and permutations. Thats why you need to test in real-world conditions for millions of miles. Only then you can get statistically valid results.
 
Not every possible permutation, any you can think up (basically).

"the technical service authorities may perform any other test within the system
boundaries"


If I'm running the tests to certify, I'm checking everything I can think of, all weather conditions, lighting, traffic, edge cases. I'm not going to be blamed for missing something.
Judging from the shenanigans played in emissions testing (most famous being VW diesel gate) I doubt the testing by the certifiers will be much beyond whatever is explicitly laid out.
 
You aren't getting the most important problem.
Are you not understanding that this is a legally binding document that is required by law for all manufacturers to follow if they want to be approved to operate in member countries?
It is not about "allowing" - its about "mandating".
Mandating what? The document is there as a minimum guideline on how to certify and approve a system to be sold and used on the public roads in order to ensure it is performing how the manufacturer states it does and based on how it expected to behave

Do we know whether Merc tested considering all those variables ? No .... because the standard "allows" - it doesn't mandate.
Strawman? Do you know how many tests Tesla runs internally before they push out FSD beta updates to their beta testers to use on public roads? No, because the law allows them to simply release any L2 systems because liability is not on them.

Liability is the key here.
ps : Anyway, you can't test for all combinations and permutations. Thats why you need to test in real-world conditions for millions of miles. Only then you can get statistically valid results.
The onus of testing for millions or however miles is up to the manufacturer who is liable for any damages and injuries caused by their system. The governing body that certifies these systems for road use administers its own tests to make sure it works the way the manufacturer states it works and also based on a standard set by the governing body on how a model system should behave.

The DMV does not test an applicant for millions of miles before they approve you to drive on the public road, they aren't going to get enough statistically relevant data to gauge your readiness from watching you drive for 1hr. They test you on a limited scenario to make sure you have competency in driving. This document is a framework for a system to make sure it is performing exactly how the manufacturer says it performs while judging it based on how an ideal model would or should perform in various scenarios. It is more rigorous than what the DMV does as it should be.

Judging from the shenanigans played in emissions testing (most famous being VW diesel gate) I doubt the testing by the certifiers will be much beyond whatever is explicitly laid out.

The scandal is not because the testing body (EPA et al) did not do its due diligence, they did, it is because the manufacturer installed a cheating mechanism to detect when testing is being performed and engage a different engine mode to be compliant with federal emissions levels vs when an end-user is driving the car. It was these same agencies that discovered some discrepancies and investigated to find out that VW had a cheating mechanism in place. So we have these certifiers to thank for doing their job well.
 
  • Like
Reactions: daktari
Are you not understanding that this is a legally binding document that is required by law for all manufacturers to follow if they want to be approved to operate in member countries?
And its a bad testing document. It is "old school" and doesn't understand the statistical way of testing millions of miles in order to cover a lot of conditions.

I don't care about legal blah blah blah.

It is simply an inadequate way to test - let alone being "demanding" as Merc claims (which was the original point).
 
ps : Anyway, you can't test for all combinations and permutations. Thats why you need to test in real-world conditions for millions of miles. Only then you can get statistically valid results.

As we've seen with FSD Beta, sometimes updating models with new training data that are meant to fix issues cause regressions in different areas. Are you also suggesting that any updates to your L2/L3 software be continuously put through another millions of miles of testing?
 
Strawman? Do you know how many tests Tesla runs internally before they push out FSD beta updates to their beta testers to use on public roads? No, because the law allows them to simply release any L2 systems because liability is not on them.
What Tesla does in the United States for L2 FSD is not relevant to what Mercedes (or anyone else) does in the EU for L3 certification.

The DMV does not test an applicant for millions of miles before they approve you to drive on the public road, they aren't going to get enough statistically relevant data to gauge your readiness from watching you drive for 1hr.
Also not relevant to certification of an L3 machine in the EU.
 
The scandal is not because the testing body (EPA et al) did not do its due diligence, they did, it is because the manufacturer installed a cheating mechanism to detect when testing is being performed and engage a different engine mode to be compliant with federal emissions levels vs when an end-user is driving the car. It was these same agencies that discovered some discrepancies and investigated to find out that VW had a cheating mechanism in place. So we have these certifiers to thank for doing their job well.
It is actually academic researchers from West Virginia University that caught them. Note that it is not EPA that does the testing directly, they do confirmatory testing (random sampling), but most manufacturers hire third party labs to do the testing.

My point is that such labs are unlikely to do testing beyond what is explicitly specified in the standard, nor am I aware of precedent of labs being put liable for test results if they do everything "by the book" (certainly didn't hear of labs sued for it related to diesel gate for example). They don't really have an incentive to do things beyond the standard, especially if they have a good amount of testing workload (in fact they are would rather minimize the amount of time required for certification testing, an example of this are methods that have been used to reduce the amount of time required for EV range testing).

It's primarily the automakers themselves that would be concerned about liability, and they would do their own calculations on what risks they are willing to take.
 
It is actually academic researchers from West Virginia University that caught them. Note that it is not EPA that does the testing directly, they do confirmatory testing (random sampling), but most manufacturers hire third party labs to do the testing.

My point is that such labs are unlikely to do testing beyond what is explicitly specified in the standard, nor am I aware of precedent of labs being put liable for test results if they do everything "by the book" (certainly didn't hear of labs sued for it related to diesel gate for example). They don't really have an incentive to do things beyond the standard, especially if they have a good amount of testing workload (in fact they are would rather minimize the amount of time required for certification testing, an example of this are methods that have been used to reduce the amount of time required for EV range testing).

It's primarily the automakers themselves that would be concerned about liability, and they would do their own calculations on what risks they are willing to take.
This is all true, but I don't think Dieselgate is a perfect analogy. Diesel engines were well understood at the time. If they were trying to pilot a new fuel, that might be closer. FSD is brand new, everyone is blazing new ground almost every time they test a new implementation.

I'm actually shocked at the wiggle room in the regs. I work in documenting EV compliance with regulations (for Ebikes and scooters mostly), this is mind boggling. As a manufacturer, I would be pushing for better mandatory test requirements.
 
It is actually academic researchers from West Virginia University that caught them.


In 2014, the California Air Resources Board (CARB) commissioned from the International Council on Clean Transportation (ICCT) a study on emissions discrepancies between European and US models of vehicles, summing up the data on 15 vehicles from three sources. Among those recruited to this task was a group of five scientists at the West Virginia University Center for Alternative Fuels Engines and Emissions (CAFEE), who used a Japanese on-board emission testing system and detected additional emissions during live road tests on two out of three diesel cars.[32][33] ICCT also purchased data from two other sources. The new road testing data and the purchased data were generated using Portable Emissions Measurement Systems (PEMS) developed by multiple individuals in the mid-late 1990s and published in May 2014.[34][35][36]
 
As we've seen with FSD Beta, sometimes updating models with new training data that are meant to fix issues cause regressions in different areas. Are you also suggesting that any updates to your L2/L3 software be continuously put through another millions of miles of testing?
That is what makes testing all this so difficult. In general changes have to be highly controlled to reduce scope of testing. Think how Windows hotfix testing can be done in days but a full release testing would take a year. Obviously NN is different - but usually only some parts of the NN are touched - not all.
 
The onus of testing for millions or however miles is up to the manufacturer who is liable for any damages and injuries caused by their system.
That looks like a libertarian view. Its like how they don't want surgeons to be certified - because "market will take care of it".

Explain to me how Honda Legend was sold (in small numbers) as an L3 and obviously wasn't tested properly.
 
Last edited:
That looks like a libertarian view. Its like how they don't want surgeons to be certified - because "market will take care of it".
Wouldn't a libertarian view be that this law does not exist at all? Doesn't the mere fact that they are requiring stringent tests and a certification process to allow this on the public road makes it the antithesis of the libertarian view? So it is exactly the opposite of a libertarian view because it calls for governmental oversight.

Explain to me how Honda Legend was sold (in small numbers) as an L3 and obviously wasn't tested properly.
Your question does not make sense.