Welcome to Tesla Motors Club
Discuss Tesla's Model S, Model 3, Model X, Model Y, Cybertruck, Roadster and More.
Register

Consumer Reports: Latest Autopilot “far less competent than a human”

This site may earn commission on affiliate links.
That video doesn't actually address the issues they raise though. For example they point out that AP doesn't see brake lights or turning signals, and doesn't see very far back for vehicles approaching at high speed. In the video he is doing 50 and the other cars not much more.

My experience is the exact opposite. It sees brake lights and turn signals too well, and panic brakes when it does. :D
 
This article reminds me of the shocking article written by a roboticist 2 years ago:

Tesla Autopilot Review: Bikers will die

It's shocking because the scholar didn't disclose that Autopilot has been beta and the scholar was clueless of many of its limitations including its inability to avoid hitting cyclists.

Similarly, Consumer Reports should have informed potential buyers that they get what they pay for: a beta product that acts like an unfinished product so be a good sitter to the system if they wanted to pay for that.
 
Hmm, that video just cemented my view of why I dont like NoA.

His claim at 1:15 just proves his mirrors must be adjusted incorrectly that he didn't see that truck coming and concludes that car is better than a human. I also would not sit with my blinker on while 4 cars pass me waiting to make a lane change, although that is probably a regional thing.

Despite that guys claim, I have had NoA cut someone off. It did it by taking sooooo long to change lanes that by the time it was done the guy in the faster lane was inches from my bumper and tailgating me. It is like a passive aggressive slo mo cut off.

What happens at 2:35ish in the video is absolutely how NoA is going to get me rear ended. His car slowed down from 70 to 56 in the faster lane BEFORE making the lane change. That is one of the biggest reasons why I don't use NoA.

One of the things at issue here is not just regional differences, but also personality differences.

Like I agree with you in that I wouldn't sit there with my blinkers on while 4 cars pass me. I've had that happen when NoA is too slow, and not aggressive enough at changing lanes.

The video creator was simply proving a point that the car did in fact see further back than CR reported.

The difference in how NoA changes lanes versus how I change lanes is why I never use NoA unless there is very little traffic. Ironically in very little traffic it's better, and more consistent at lane changes than I am. It does bug me to say that, but that's what the evidence says. Ughh.

The other reason I don't use NoA in areas with lots of traffic is it simply doesn't have good situational awareness.

As to the CR reports article? I didn't really care for it as I didn't see any videos. In this day, and age it's kinda pointless to talk about NoA without a reference video. Having a video is critical in showing different interpretations of events. One persons normal lane change is too aggressive for someone else.

I'm not sure why CR can't hire a millennial to make a video.

CR also failed to really identify the major failure points of NoA, and failed to understand that the lane changes to the right only happen in mad max. One would think they'd at least try the other modes.

So all in all I think CR was being lazy. That's disappointing as Tesla has been really responsive to CR complaints in the past. So it sucks to see CR fail so badly.
 
Last edited:
This is how competent a human driver is. AP is still Beta, but in a lot of ways it is already superior to humans.
 

Attachments

  • 2018-11-05 09.48.43.jpg
    2018-11-05 09.48.43.jpg
    653.4 KB · Views: 61
No, Elon said NOA was FSD on the highway and that its feature complete. That was his exact words, so either we take him at his words and use his logic and perspective or we completely reject it. You can't cherry pick, this isn't burger king.

We can debate what Elon means when he says FSD if you want. I can tell you what Nav on AP is because I have it. With Nav on AP, I can input a destination in my nav and the car will automatically merge onto the highway, steer to keep me in the lane, slow down or speed up with traffic, make a lane change to pass a slower car, make a lane change to get back in the right lane, make a lane change to stay in the right lane for a highway change, and take the right exit based on my nav and then notify me to take back control. And all I have to do is hold the wheel. Is it perfect? No, of course not. There are situations that Nav on AP cannot do well. But it is very good and will get better.
 
Last edited:
A really nice rebuttal to the CR article:


My experience with NOA has been like in this video. The auto lane changes are safe and smooth.
I thought this was also a great response. CR obviously has a beef with Tesla and they seem to be taking every opportunity to trash them at this point.

Anyone can objectively point out some short-comings of Tesla's autopilot in it's current state. Even I can do this and I love my car and the autopilot features. I can openly admit that it's not perfect and has a lot of room for growth and improvement. But this CR report is borderline fictitious in order to poke Tesla in the eye. It's not a good look for CR.
 
I thought this was also a great response. CR obviously has a beef with Tesla and they seem to be taking every opportunity to trash them at this point.

Anyone can objectively point out some short-comings of Tesla's autopilot in it's current state. Even I can do this and I love my car and the autopilot features. I can openly admit that it's not perfect and has a lot of room for growth and improvement. But this CR report is borderline fictitious in order to poke Tesla in the eye. It's not a good look for CR.

Tesla's approach of putting beta versions on the road is not well received by the industry. CR shares this opinion. This is where they are coming from:
Consumer Reports: Tesla Must Prove Safety Before Claiming “Self-Driving” Ability
 
  • Informative
Reactions: 1 person
Looks like CR is not happy with how other media outlets are mischaracterizing their report:
Consumer Reports Not Thrilled By Misleading Navigate On Autopilot Headlines

CR did reach out to say that they do in fact like Nav on AP in general, they just don't like the "no confirmation" setting.

The other articles about the CR report definitely have very click bait headlines.
 
  • Like
Reactions: mongo
Hmm, that video just cemented my view of why I dont like NoA.

His claim at 1:15 just proves his mirrors must be adjusted incorrectly that he didn't see that truck coming and concludes that car is better than a human. I also would not sit with my blinker on while 4 cars pass me waiting to make a lane change, although that is probably a regional thing.

Despite that guys claim, I have had NoA cut someone off. It did it by taking sooooo long to change lanes that by the time it was done the guy in the faster lane was inches from my bumper and tailgating me. It is like a passive aggressive slo mo cut off.

What happens at 2:35ish in the video is absolutely how NoA is going to get me rear ended. His car slowed down from 70 to 56 in the faster lane BEFORE making the lane change. That is one of the biggest reasons why I don't use NoA.

Yes, agreed. Currently, NoA is not nearly good enough about speed matching to the target lane, even with a high max speed set, and it also signals too often when people are right there. If it is not crowded, just signal a second or two before changing lanes when the coast is clear, then go! No need to make people nervous about you changing lanes into them.

Fortunately, I do think this is not a hardware limitation and will probably be improved before NoA becomes a production release (not beta).

How reliably the car can see traffic coming from behind, I do not know.
 
I haven't had time to watch this yet. Will listen to it in the car tomorrow.


Some of my thoughts from watching the video:

You can customize several options in Nav on AP. You can set the frequency of lane changes. If you want a lot of lane changes, you can set it to Mad Max. If you want less or no lane changes, you can set it to mild or disabled. My guess is that CR had it set to Mad Max because yes, in that setting, the car will try to lane change a lot. Mad Max is basically "get me around every car in front of me" setting. So the car will try to constantly lane change left or right to get around the vehicle in front of you. if you don't want to drive like that or you think Nav on AP is bad at it, then change the setting down to average or mild. The no confirmation is simply an option that Tesla added because they've apparently gotten millions of successful auto lane changes from the system so they felt like it was good enough for drivers to use if they want to. But if you don't like it, don't use it!! It's completely optional. CR should have talked more about how it is optional.

I tested "mild" and the car did almost no lane changes at all. Instead, it drove much slower and just stayed behind cars that I would have normally passed. Personally, I don't like Mad Max mode because it does try to do way too lane changes that don't make any sense to me. I like average or even mild. It does make road trips more comfortable. No, you're not constantly having to watch the system more than normal driving if you apply those settings. You will cruise along and only make occasional lane changes when you are stuck behind a real slow poke. You can even disable auto lane changes completely and you can initiate all auto lane changes yourself with the stalk. You can also customize if you want lane change confirmation. So if you don't like the car doing lane changes without confirmation, you can stay with the old system of requiring a stalk confirmation. I wish CR would have talked about this more. They are criticizing Mad Max Mode which is fair but they should tell people that they can solve many of these issues and have a more comfortable experience by just changing a couple settings. It is certainly possible in my experience to have a less stressful driving experience if you minimize lane changes. I feel like CR is being misleading when they say that Nav on AP requires more supervision than normal driving. That can be true but it depends a lot on your settings and it depends on the traffic conditions you are in.

AP and FSD are literally works in progress. Nav on AP that we have now is not the finished product. CR should be more clear to people about that. It's why Tesla calls it Beta and why we get software updates that are constantly tweaking, changing, and adding new features. It's why we get the warning to keep our hands on the wheel or the warnings when we enable auto lane change that the feature is not autonomous driving. Tesla is very clear that the driver needs to be watchful. While some of the features like auto lane change without confirmation are flirting with self-driving, the software is not FSD yet. In some ways, Tesla owners, especially if they use the newer features like auto lane without confirmation, are basically unpaid safety drivers for Tesla's FSD development. Tesla is developing their FSD by developing features in house, testing them internally, and then releasing them as driver assist features to a small group first and then to a larger group later, to refine and improve the feature. Tesla is building FSD piece by piece this way.

Right now, I feel like we are in a sort of pre-FSD twilight zone because Tesla has more advanced AP3 NN that is meant to be the real FSD software but we don't have it yet. Right now, the fleet has the AP2 NN which has improved greatly but is lagging behind the AP3 NN. So we are seeing some pre-FSD features like auto lane change without confirmation that might seem like almost FSD but we are not at FSD yet. Many of the weaknesses CR mentions in Nav on AP, are because we are still using the AP2 NN. We are not using the AP3 NN that Tesla is working on for FSD. I think when Tesla does release the AP3 NN to the fleet, we will see a much more robust system that will give us a much better picture of what Tesla's FSD will be like.

Also, many of the issues that CR mentions like the car possibly cutting off a fast car coming up behind you, are issues that will be addressed in future updates. I am confident that better NN in future updates will allow the rear camera to track fast cars better and improve that aspect of driving. And future updates will improve the assertiveness of auto lane changes to make them better as well. Elon even conceded during the Autonomy Investor Day event that Nav on AP is too conservative at lane changes and needs to be more assertive in some situations.
 
Some of my thoughts from watching the video:

You can customize several options in Nav on AP. You can set the frequency of lane changes. If you want a lot of lane changes, you can set it to Mad Max. If you want less or no lane changes, you can set it to mild or disabled. My guess is that CR had it set to Mad Max because yes, in that setting, the car will try to lane change a lot. Mad Max is basically "get me around every car in front of me" setting. So the car will try to constantly lane change left or right to get around the vehicle in front of you. if you don't want to drive like that or you think Nav on AP is bad at it, then change the setting down to average or mild. The no confirmation is simply an option that Tesla added because they've apparently gotten millions of successful auto lane changes from the system so they felt like it was good enough for drivers to use if they want to. But if you don't like it, don't use it!! It's completely optional. CR should have talked more about how it is optional.

I tested "mild" and the car did almost no lane changes at all. Instead, it drove much slower and just stayed behind cars that I would have normally passed. Personally, I don't like Mad Max mode because it does try to do way too lane changes that don't make any sense to me. I like average or even mild. It does make road trips more comfortable. No, you're not constantly having to watch the system more than normal driving if you apply those settings. You will cruise along and only make occasional lane changes when you are stuck behind a real slow poke. You can even disable auto lane changes completely and you can initiate all auto lane changes yourself with the stalk. You can also customize if you want lane change confirmation. So if you don't like the car doing lane changes without confirmation, you can stay with the old system of requiring a stalk confirmation. I wish CR would have talked about this more. They are criticizing Mad Max Mode which is fair but they should tell people that they can solve many of these issues and have a more comfortable experience by just changing a couple settings. It is certainly possible in my experience to have a less stressful driving experience if you minimize lane changes. I feel like CR is being misleading when they say that Nav on AP requires more supervision than normal driving. That can be true but it depends a lot on your settings and it depends on the traffic conditions you are in.

AP and FSD are literally works in progress. Nav on AP that we have now is not the finished product. CR should be more clear to people about that. It's why Tesla calls it Beta and why we get software updates that are constantly tweaking, changing, and adding new features. It's why we get the warning to keep our hands on the wheel or the warnings when we enable auto lane change that the feature is not autonomous driving. Tesla is very clear that the driver needs to be watchful. While some of the features like auto lane change without confirmation are flirting with self-driving, the software is not FSD yet. In some ways, Tesla owners, especially if they use the newer features like auto lane without confirmation, are basically unpaid safety drivers for Tesla's FSD development. Tesla is developing their FSD by developing features in house, testing them internally, and then releasing them as driver assist features to a small group first and then to a larger group later, to refine and improve the feature. Tesla is building FSD piece by piece this way.

Right now, I feel like we are in a sort of pre-FSD twilight zone because Tesla has more advanced AP3 NN that is meant to be the real FSD software but we don't have it yet. Right now, the fleet has the AP2 NN which has improved greatly but is lagging behind the AP3 NN. So we are seeing some pre-FSD features like auto lane change without confirmation that might seem like almost FSD but we are not at FSD yet. Many of the weaknesses CR mentions in Nav on AP, are because we are still using the AP2 NN. We are not using the AP3 NN that Tesla is working on for FSD. I think when Tesla does release the AP3 NN to the fleet, we will see a much more robust system that will give us a much better picture of what Tesla's FSD will be like.

Also, many of the issues that CR mentions like the car possibly cutting off a fast car coming up behind you, are issues that will be addressed in future updates. I am confident that better NN in future updates will allow the rear camera to track fast cars better and improve that aspect of driving. And future updates will improve the assertiveness of auto lane changes to make them better as well. Elon even conceded during the Autonomy Investor Day event that Nav on AP is too conservative at lane changes and needs to be more assertive in some situations.


I think a first time user, like them will always feel more overwhelmed by the system, compared to everyday users who have already gotten used to the quirks.

As for why they didn't emphasize that this can be turned off or this is beta is exactly that. CR, Google, Volvo and all the others realized the safety issue related human attention keeping in monitoring a very well automated system. The better the system, the less attention they pay which leads to accidents. Elon is well aware of this issue but still decided to deploy it in production cars because of his own ego. Let's be the first, who cares about casualties.
Tesla can hide behind the screen of "The driver is responsible all the time" but after all they are releasing a product that doesn't take some human behavior in account. Making a comfortable seat is just one part of matching humans with machines but one can't stop there.
 
New and POSSIBLY relevant insight: check out this photo from the article, used to illustrate unsafe lane changes. I suspect the CR drivers were clueless on how to interpret the AP display, which may explain the whole disconnect between their experiences and everyone else:

D04E5D12-AC9E-4E3A-A498-66298D910EE3.jpeg


Note that the AP lane change is waiting on the approaching car in the destination lane, as indicated by the red highlight of that car icon and the lane, and will not in fact change lanes until at least that car goes by. HOWEVER, since this is labeled “ready to pass on left”, and the gist of the article is that the AP chooses to cut off approaching cars when doing such lane changes, it very well might be that the drivers got only this far, misread the cues entirely, and disabled AP.

Someone not familiar at all with the AP, which might include 100% of the CR drivers, might look at that display with a path into that left lane and think “holy smokes it is going to change lanes in front of that approaching car!” This would be a crazy level of incompetence for CR testing, but, well, it DOES explain the total disconnect, and CR did not include any video, so maybe ...
 
New and POSSIBLY relevant insight: check out this photo from the article, used to illustrate unsafe lane changes. I suspect the CR drivers were clueless on how to interpret the AP display, which may explain the whole disconnect between their experiences and everyone else:

View attachment 411709

Note that the AP lane change is waiting on the approaching car in the destination lane, as indicated by the red highlight of that car icon and the lane, and will not in fact change lanes until at least that car goes by. HOWEVER, since this is labeled “ready to pass on left”, and the gist of the article is that the AP chooses to cut off approaching cars when doing such lane changes, it very well might be that the drivers got only this far, misread the cues entirely, and disabled AP.

Someone not familiar at all with the AP, which might include 100% of the CR drivers, might look at that display with a path into that left lane and think “holy smokes it is going to change lanes in front of that approaching car!” This would be a crazy level of incompetence for CR testing, but, well, it DOES explain the total disconnect, and CR did not include any video, so maybe ...

That's a good explanation of the disconnect. Because there is clearly a disconnect between CR's characterization and reality.
 
New and POSSIBLY relevant insight: check out this photo from the article, used to illustrate unsafe lane changes. I suspect the CR drivers were clueless on how to interpret the AP display, which may explain the whole disconnect between their experiences and everyone else:

Note that the AP lane change is waiting on the approaching car in the destination lane, as indicated by the red highlight of that car icon and the lane, and will not in fact change lanes until at least that car goes by. HOWEVER, since this is labeled “ready to pass on left”, and the gist of the article is that the AP chooses to cut off approaching cars when doing such lane changes, it very well might be that the drivers got only this far, misread the cues entirely, and disabled AP.

Someone not familiar at all with the AP, which might include 100% of the CR drivers, might look at that display with a path into that left lane and think “holy smokes it is going to change lanes in front of that approaching car!” This would be a crazy level of incompetence for CR testing, but, well, it DOES explain the total disconnect, and CR did not include any video, so maybe ...

Wrong. Watch the video above. She mentioned that the car started to move to the next lane and she had to take over.
 
Wrong. Watch the video above. She mentioned that the car started to move to the next lane and she had to take over.

Thanks, I totally missed that CR video. Still having a hard time wrapping my head around the disconnect between what the the CR testers are saying happened, and what the ALL Electric video in response demos as happening, along with his claims that he has never seen what the CR folks describe:

Are one of these two parties simply wrong, or is there something else going on?
 
Also there was the illegal pass on the right that it made multiple times.

I'm really confused. In what state is it illegal to "pass on the right" when on a freeway or interstate? Granted, I've virtually lived my whole life in the "armpit of the midwest", but I'm not aware that it is illegal to pass on the right when on a freeway or interstate highway? In fact, I make it a habit to "stay right unless passing", which I learned in Driver's Ed back in 1968 and it appears that nowadays, 85% (or more) of the current drivers have forgotten that or ignore it. It seems most drivers come down the on ramp and immediately go into the middle or left lane and virtually "park" their car there forcing people to pass them on the right. They seem to think it is their "right" to choose an interstate lane and drive at whatever speed they choose ignoring that they are in fact breaking the law by not yielding to faster drivers. What they don't realize is that the faster driver trying to get around them might just be having an emergency and is driving to the hospital because someone had a heart attack, someone got stabbed or shot, or their wife is about to deliver. I ALWAYS yield to faster drivers just because I have NO IDEA why they are driving faster. True, they might just be idiots that like to drive faster, but do you really want to jeopardize a life because you SUSPECT they might not have a reason for driving fast? Yield to them, move to the right. If they have no reason for speeding, they just might find karma a mile or two up the road when a trooper pulls them over.