FSD / AP Rewrite - turning the corner?

[Glan gluaisne]

The Tesla NCAP rating was brought siren just because of the name otherwise performance wise it came out on top.

Not sure I understand the meaning of "brought siren just because of the name", any chance of a translation, please?

This is what NCAP said about the Model 3:

"Tesla’s pioneering role in self-driving technology is reflected in its top scores for Vehicle Assistance and Safety Backup. However, its “Autopilot” system does little to keep the driver engaged. Its distinct steering strategy gives the impression that either the car is driving itself or the driver has full control, and the system is more authoritarian than cooperative. Grading wise, the car is badly let down by its performance in Driver Engagement and ends up ranked 2 ‘Moderate’. On the upside, Tesla offers over-the-air updates and can rapidly introduce performance improvements, as they have shown to be doing in recent years."

The table of results seems reasonable. I've driven three of the cars in this list, and would say they've probably judged things fairly well, given the criteria they set:


1 Mercedes GLE - 174 (85% AC and 89% SB)
2 BMW 3 Series - 172 (82% AC and 90% SB)
3 Audi Q8 - 162 (78% AC and 84% SB)
4 Ford Kuga - 152 (66% AC and 86% SB)
5 Volkswagen Passat - 137 (76% AC and 61% SB)
6 Tesla Model 3 - 131 (36% AC and 95% SB)
7 Nissan Juke - 124 (52% AC and 72% SB)
8 Volvo V60 - 120 (71% AC and 49% SB)
9 Renault Clio - 105 (62% AC and 43% SB)
10 Peugeot 2008 - 101 (61% AC and 40% SB)

(AC = Assistance Competence, SB = Safety Backup)

 

[firetrax]

Not sure I understand the meaning of "brought siren just because of the name", any chance of a translation, please?

This is what NCAP said about the Model 3:



The table of results seems reasonable. I've driven three of the cars in this list, and would say they've probably judged things fairly well, given the criteria they set:


1 Mercedes GLE - 174 (85% AC and 89% SB)
2 BMW 3 Series - 172 (82% AC and 90% SB)
3 Audi Q8 - 162 (78% AC and 84% SB)
4 Ford Kuga - 152 (66% AC and 86% SB)
5 Volkswagen Passat - 137 (76% AC and 61% SB)
6 Tesla Model 3 - 131 (36% AC and 95% SB)
7 Nissan Juke - 124 (52% AC and 72% SB)
8 Volvo V60 - 120 (71% AC and 49% SB)
9 Renault Clio - 105 (62% AC and 43% SB)
10 Peugeot 2008 - 101 (61% AC and 40% SB)

(AC = Assistance Competence, SB = Safety Backup)

Typo. Should have said brought down. Lack of driver engagement means that they penalised Tesla because it doesn't monitor the driver when the internal camera currently. Tesla was also heavily penalised for calling it autopilot. It scored top marks for actual performance.

 

[Glan gluaisne]

Typo. Should have said brought down. Lack of driver engagement means that they penalised Tesla because it doesn't monitor the driver when the internal camera currently. Tesla was also heavily penalised for calling it autopilot. It scored top marks for actual performance.

There doesn't seem to be any penalty, AFAICS. The Model 3 driver assistance system scored very highly in terms of safety, but I don't think anyone, for one moment, could say that the Tesla system communicates its intentions clearly to the driver at the moment. Frequent system failings, like sudden braking, and steering intervention, often with no real hazard, and with nothing that informs the driver as to the reason, are a real issue. They are common enough to be very regularly discussed here.

Who can honestly say that they've not had at least one "WTF!!!" moment when driving a Tesla?

Other manufacturer's systems rarely ever seem to exhibit behaviour like this, which is probably one reason why they all scored far more highly in Assistance Competence. Arguably, the Safety Backup score should have a greater weight in the overall score, as that clearly is of more importance, but I don't think anyone could really argue that Tesla's system, as it currently stands, is competent at informing the driver, safely, about what's going on around the car.

 

[GeorgeSymonds]

It would be a strange world if the reason NCAP rated the Tesla system moderate, 2 full levels behind BMW, Merc and Audi was because of the name Tesla gave it. It’s not even a functional attribute. NCAP would be undoing all the credibility they’ve built up by doing so.

I think someone is missing the point of a driver assistance feature, which is what these will all be for the foreseeable time, if it doesn’t work well with the driver,

 

[firetrax]

There doesn't seem to be any penalty, AFAICS. The Model 3 driver assistance system scored very highly in terms of safety, but I don't think anyone, for one moment, could say that the Tesla system communicates its intentions clearly to the driver at the moment. Frequent system failings, like sudden braking, and steering intervention, often with no real hazard, and with nothing that informs the driver as to the reason, are a real issue. They are common enough to be very regularly discussed here.

Who can honestly say that they've not had at least one "WTF!!!" moment when driving a Tesla?

Other manufacturer's systems rarely ever seem to exhibit behaviour like this, which is probably one reason why they all scored far more highly in Assistance Competence. Arguably, the Safety Backup score should have a greater weight in the overall score, as that clearly is of more importance, but I don't think anyone could really argue that Tesla's system, as it currently stands, is competent at informing the driver, safely, about what's going on around the car.

It would be a strange world if the reason NCAP rated the Tesla system moderate, 2 full levels behind BMW, Merc and Audi was because of the name Tesla gave it. It’s not even a functional attribute. NCAP would be undoing all the credibility they’ve built up by doing so.

I think someone is missing the point of a driver assistance feature, which is what these will all be for the foreseeable time, if it doesn’t work well with the driver,

z1

See this
Tesla Autopilot gets top marks for performance but trashed for driver engagement in new industry-wide tests - Electrek

They took particular issue with the name of Tesla’s driver-assist system, the promotional material, lack of head-up display, and lack of driver monitoring system.

While Tesla is scoring at the bottom for driver engagement, Autopilot has crushed all other systems for on the actual performance:

The M3 autopilot got just 10 out of 25 points for consumer information because of the name and full marks for the capability and performance metrics. So its overall score was brought down just by the name.

 

[Glan gluaisne]

z1

See this
Tesla Autopilot gets top marks for performance but trashed for driver engagement in new industry-wide tests - Electrek

They took particular issue with the name of Tesla’s driver-assist system, the promotional material, lack of head-up display, and lack of driver monitoring system.

While Tesla is scoring at the bottom for driver engagement, Autopilot has crushed all other systems for on the actual performance:

I read that opinion piece back when it came out.

I agree wholeheartedly with NCAP (and have posted here before about this very subject) that "Autopilot" is a misleading name for what is intended to only be a driver assist aid. If Tesla had chosen to call FSD Autopilot then I think that may have been more appropriate, as FSD should, once it's perfected, behave pretty much as a true autopilot system.

A head up display would be one way of better informing the driver as to what's going on, but there are other ways. Right now, trying to second guess what the car has done is an everyday activity - witness threads here debating why some of the odd things happen, ranging from shadows possibly causing emergency braking events to them being caused by something related to overtaking high sided vehicles.

The issue with driver monitoring is that no one knows if the Tesla system really does it or not. We know that that the steering wheel "hands on" system is a bit flaky, and recently it's been rumoured that, since the internal camera was activated, there may be something that now looks at things like the driver's eye movements. However, Tesla haven't said anything, so all anyone can do is try and guess what may be going on.

Overall, I can't really criticise the NCAP conclusion. NCAP could have read this forum and obtained, from Tesla owners posting here, all the evidence needed to show the effectiveness of the Assistance Competence elements of their assessment method. Much as I love the car, I'm in no doubt at all that there are serious shortcomings in the driver assist elements at the moment. They could probably be put right fairly easily, and maybe they will be as a part of the FSD rewrite.

 

[firetrax]

I read that opinion piece back when it came out.

I agree wholeheartedly with NCAP (and have posted here before about this very subject) that "Autopilot" is a misleading name for what is intended to only be a driver assist aid. If Tesla had chosen to call FSD Autopilot then I think that may have been more appropriate, as FSD should, once it's perfected, behave pretty much as a true autopilot system.

A head up display would be one way of better informing the driver as to what's going on, but there are other ways. Right now, trying to second guess what the car has done is an everyday activity - witness threads here debating why some of the odd things happen, ranging from shadows possibly causing emergency braking events to them being caused by something related to overtaking high sided vehicles.

The issue with driver monitoring is that no one knows if the Tesla system really does it or not. We know that that the steering wheel "hands on" system is a bit flaky, and recently it's been rumoured that, since the internal camera was activated, there may be something that now looks at things like the driver's eye movements. However, Tesla haven't said anything, so all anyone can do is try and guess what may be going on.

Overall, I can't really criticise the NCAP conclusion. NCAP could have read this forum and obtained, from Tesla owners posting here, all the evidence needed to show the effectiveness of the Assistance Competence elements of their assessment method. Much as I love the car, I'm in no doubt at all that there are serious shortcomings in the driver assist elements at the moment. They could probably be put right fairly easily, and maybe they will be as a part of the FSD rewrite.

Well NCAP actually have it top marks for actual performance which you seen to be not happy with and their only real issue was the name and driver monitoring. That's in their actual scores and not an opinion piece.

 

[GeorgeSymonds]

The table of results has been posted above - it’s comes LAST on assistance competence.

You are cherry picking the odd metric as if it’s the only one that matters in true fanboy style.

Have you tried any of the other actual system being tested?

 

[Glan gluaisne]

Well NCAP actually have it top marks for actual performance which you seen to be not happy with and their only real issue was the name and driver monitoring. That's in their actual scores and not an opinion piece.

That's not correct. The overall performance rating was only medium, and was badly let down by the poor Assistance Competence mark. No one is disputing the excellent Safety Backup score, but the overall performance mark is made up from both elements, not just one. Until we have FSD, driver assistance aids, such as those presently fitted to the Model 3, must assist the driver, and not act in a way that makes driving less safe.

As mentioned before, I doubt there is a single Tesla Model 3 owner who would say that the Assistance Competence performance of the car was good. It frequently performs manoeuvres, like sudden hard braking and rather harsh steering corrections when there is no hazard. There are times when it does this and actually creates a hazard; I've had to force the steering back on track (when not using AP) a few times, when the car has decided to swerve for no reason at all that I could see. The car fails to accurately inform the driver as to why it's doing this, and that lack of driver engagement is one major reason that it scored badly.

 

[Wol747]

Mine's an S, which is arguably less advanced in the driver information department than the 3.

Having seen today's videos of the displays on both with what is presumably the new coding, I have to say that the display is less than ergonomic.
It's all very Star Wars, video games stuff but doesn't appear to me to be helpful in assessing the car's behaviour regarding hazards, signs etc.
I think Tesla need to engage seriously with aircraft avionics' software designers to work out what is necessary and what is there to show off how advance it is.

 

[Zakalwe]

Mine's an S, which is arguably less advanced in the driver information department than the 3.

Having seen today's videos of the displays on both with what is presumably the new coding, I have to say that the display is less than ergonomic.
It's all very Star Wars, video games stuff but doesn't appear to me to be helpful in assessing the car's behaviour regarding hazards, signs etc.
I think Tesla need to engage seriously with aircraft avionics' software designers to work out what is necessary and what is there to show off how advance it is.

It's beta software and almost certainly not representative of the finished product.

 

[firetrax]

That's not correct. The overall performance rating was only medium, and was badly let down by the poor Assistance Competence mark. No one is disputing the excellent Safety Backup score, but the overall performance mark is made up from both elements, not just one. Until we have FSD, driver assistance aids, such as those presently fitted to the Model 3, must assist the driver, and not act in a way that makes driving less safe.

As mentioned before, I doubt there is a single Tesla Model 3 owner who would say that the Assistance Competence performance of the car was good. It frequently performs manoeuvres, like sudden hard braking and rather harsh steering corrections when there is no hazard. There are times when it does this and actually creates a hazard; I've had to force the steering back on track (when not using AP) a few times, when the car has decided to swerve for no reason at all that I could see. The car fails to accurately inform the driver as to why it's doing this, and that lack of driver engagement is one major reason that it scored badly.

Look at the report again (attached).
40 out of 40 for adaptive cruise control performance
35 out of 35 for steering assistance
25 out of 25 for system failure
50 out of 50 for collision avoidance
91.7% for vehicle assistance
The other categories marked down for the name and non performance related subjective factors.

 

[LongRanger]

NCAP need to test their tests. Fail.

 

[Dilly]

Not much of a problem where I live, but I’ve come across plenty of city junctions where the road signage is just plain overkill. One I recall had so many signs they could have been Apple’s T&C’s. Quite how any system will identify the wheat from the chaff must be a nightmare.

 

[Glan gluaisne]

Look at the report again (attached).
40 out of 40 for adaptive cruise control performance
35 out of 35 for steering assistance
25 out of 25 for system failure
50 out of 50 for collision avoidance
91.7% for vehicle assistance
The other categories marked down for the name and non performance related subjective factors.

I have read the report, I read it when it came out. The above scores all relate to the excellent Safety Backup rating, and I believe it's fairly common knowledge that Tesla's are amongst the safest cars on the road, in part due to those features.

The issue has nothing to do with NCAP biasing results because they don't "like" Tesla, if anything, NCAP has consistently shown just how safe Tesla vehicles are, when compared to others.

The issue is about the way the driver assistance features communicate with the driver. The hard fact is that, much of the time, the driver hasn't got a clue as to why the car has done something. That failure to communicate clearly and unequivocally is the root cause of the poorer mark. As stated before, how many Tesla Model 3 owners have never experienced the car doing something unexpected, and leaving them wondering why it's behaved as it has? I doubt there are any.

My very first drive in my car, coming back from West Drayton on the M3, resulted in two, seemingly random, events, once where the car sounded a loud alarm and swerved to the right, for no reason that I could see, and once where it braked hard when I was using TACC, again for no apparent reason. I've since learned that the car just does this from time to time, yet after nearly a year of ownership I'm no wiser as to WHY it does it, as the car still doesn't pass enough information to me, the driver, about what it believes to be the hazard(s) that create these false alarms. As others have mentioned here many times, the long standing phantom braking issue alone is potentially dangerous, so it comes as no surprise to find that NCAP may view it the same way.

The analogy with aeroplanes is a reasonable one to try and describe the driver communication issue that NCAP have picked up on. With two people in the cockpit, there is a long standing procedure to make it absolutely clear who has control. The person handing over control says these exact words: "you have control" and the person taking over says these exact words: "I have control". The procedure for making it clear as to whether the car's driver assist functions are "in control", or the driver is, need to be similar, and at the moment they just aren't.

 

[firetrax]

I have read the report, I read it when it came out. The above scores all relate to the excellent Safety Backup rating, and I believe it's fairly common knowledge that Tesla's are amongst the safest cars on the road, in part due to those features.

The issue has nothing to do with NCAP biasing results because they don't "like" Tesla, if anything, NCAP has consistently shown just how safe Tesla vehicles are, when compared to others.

The issue is about the way the driver assistance features communicate with the driver. The hard fact is that, much of the time, the driver hasn't got a clue as to why the car has done something. That failure to communicate clearly and unequivocally is the root cause of the poorer mark. As stated before, how many Tesla Model 3 owners have never experienced the car doing something unexpected, and leaving them wondering why it's behaved as it has? I doubt there are any.

My very first drive in my car, coming back from West Drayton on the M3, resulted in two, seemingly random, events, once where the car sounded a loud alarm and swerved to the right, for no reason that I could see, and once where it braked hard when I was using TACC, again for no apparent reason. I've since learned that the car just does this from time to time, yet after nearly a year of ownership I'm no wiser as to WHY it does it, as the car still doesn't pass enough information to me, the driver, about what it believes to be the hazard(s) that create these false alarms. As others have mentioned here many times, the long standing phantom braking issue alone is potentially dangerous, so it comes as no surprise to find that NCAP may view it the same way.

The analogy with aeroplanes is a reasonable one to try and describe the driver communication issue that NCAP have picked up on. With two people in the cockpit, there is a long standing procedure to make it absolutely clear who has control. The person handing over control says these exact words: "you have control" and the person taking over says these exact words: "I have control". The procedure for making it clear as to whether the car's driver assist functions are "in control", or the driver is, need to be similar, and at the moment they just aren't.

Nope. The adaptive cruise control performance rating relates to the actual performance of autopilot.

 

[firetrax]

Mine's an S, which is arguably less advanced in the driver information department than the 3.

Having seen today's videos of the displays on both with what is presumably the new coding, I have to say that the display is less than ergonomic.
It's all very Star Wars, video games stuff but doesn't appear to me to be helpful in assessing the car's behaviour regarding hazards, signs etc.
I think Tesla need to engage seriously with aircraft avionics' software designers to work out what is necessary and what is there to show off how advance it is.

That's a beta test. The final version will not be like that.

 

[firetrax]

The table of results has been posted above - it’s comes LAST on assistance competence.

You are cherry picking the odd metric as if it’s the only one that matters in true fanboy style.

Have you tried any of the other actual system being tested?

Read the report again or feel free to keep on hating. The overall score has been brought down by subjective minor things like the name whereas the Tesla obtained full ratings for actual perfomance ratings.

 

[Glan gluaisne]

@firetrax , I get your enthusiasm and desire to never see anything critical written about Tesla, but can you honestly say that the Model 3 doesn't have major failings when it comes to the way the driver assistance functions behave?

I honestly cannot believe that any Model 3 owner can think that the driver assist functions work well, as far as helping, rather than hindering, the driver in everyday driving conditions (i.e. not using FSD). This forum, like every other Tesla forum, is full of reports from owners about potentially serious safety issues, like phantom braking and random steering inputs, things that happen for no discernible reason. The fact that the only visible indication is a brief warning, flashed up well to the left of the driver's sight line, doesn't help, as when something like this happens the last thing you're able to do is look down and left to try and see what the car's doing.

The car is fantastic to drive, but there's never a moment when I'm not thinking about the need for me to take rapid corrective action when the car decides to do something unpredictable. There's never any prior warning when these things happen, either, and that seems to be one of the concerns that NCAP had, the disconnect between the car systems and the driver. The first the driver knows about something going awry is when the car takes action, often quite aggressively, and simultaneously sounds the alarm gongs. I've driven other cars with driver assist functions, and none have behaved in this way, and certainly none have ever done things like the aggressive swerve for no reason that my Model 3 seem to do from time to time (even now I'm not sure what causes it to do this).

 

[FEERSUMENDJIN]

Sorry storm in a tea cup. Subjective category pulls down entire score. Political BS.
Btw : I like my missus to be quite authoritarian