Navigate on Autopilot is Useless (2018.42.3)

[clydeiii]

I actually used to drive that particular area somewhat regularly (10x year or so) a while back... and man, I could not imagine trusting NoA in its current state to handle things there. Unless it's just dense traffic where it has no opportunity or reason to even suggest a lane change (seems likely given the location). In that case, you're just using lane keeping anyway, which after 3 years of development is finally mostly usable on highways with AP2+, and isn't NoA.

You'll have to imagine a little harder because NoA is so great on that stretch, which isn't super dense traffic. I honestly think it's the dense traffic where NoA is worse, tbh, but luckily my commute doesn't feature that. I'll upload a video eventually so you can see.

For NoA to be even close to "flawless", IMO it'd need to do at least the following:

• Hold the lane well while conforming to traffic ahead (normal AP + TACC)
• Make and execute sane lane change decisions without delay (as in, I should never have to cancel one)
• Make lane changes to overtake without decelerating, and not merging to the new lane at a speed lower than the current flow of traffic in the target lane.
• Never attempt or suggest a lane change into an obstacle (it does this frequently in my testing)
• No unnecessary braking (the phantom braking issue is very real, and a huge issue in every vehicle I've tested NoA in)
• Take the correct exits/interchanges as needed for a route without error and in a timely manner (merging to the right to be in the correct lane 3 miles before an exit, slowing pace by 20 MPH to do so, is not flawless)
• I shouldn't have to question every decision it attempts to make as if I'm stressfully babysitting a teenage student driver. The decisions it makes should make sense in the vast majority of cases.
  • For example, I shouldn't have to override a decision within the first few minutes of engaging NoA, which is pretty common. In fact, I override lane changes it tries to make about 2/3 times in practice. This does not up my confidence in the system at all.

It holds the lane well. It mostly makes sane lane change decisions (sometimes I am questioning of it, and yes, I will cancel, because this is how you teach the neural network). These days, it mostly makes lane changes without deceleration. It's been getting slowly better and better at this sort of thing. I'm not 100% sure what you mean by it attempts lane changes into obstacles but like I said, most of the lane changes are very sane and very smooth. I have experienced phantom braking maybe 3 times in probably 200 commutes. It absolutely takes the correct exits and in a timely manner. TBH, I sometimes think it's too timely and it gets over too early. I never feel like I'm babysitting a teenage student and most of the time I tune out driving and focus on Washington Post and New York Times podcasts.

I honestly think this simply comes down to our personalities. You're different than I am, clearly, and have different reactions to the same technology.

If I were so inclined, I could quite easily just setup a few GoPros and make a massive compilation of NoA fails with very overall few miles required to do so. It really does astound me that people are singing such high praises for this feature in its current state.

You should! I'm definitely going to film my commute.

 

[DrDabbles]

It's comments like these that make me really scratch my head. I can't tell if these are cases of crowd delusion because people want to believe so much, or if they're cases where AP somehow actually works flawlessly. If it's the former, then I can understand, but if it's the latter I really have a hard time. What is it about these magical commute paths that makes AP and NoAP work so perfectly when it fails so badly in so many cases??

It holds the lane well. It mostly makes sane lane change decisions

AP holds the lane in that it doesn't cross lines normally, sure. But in this latest update 36.2.1, I've got ping-ponging in the lane that has come back noticeably. NoA lane change decisions very clearly relate to the level of traffic nearby, so that's not really something worth discussing except to say that NoA suggested I change lanes into the slow lane while a pickup truck was beside me, and then wanted to slow down to get behind the pickup that was going just slightly slower than I was. It's not the first time that kind of dumb choice has been offered. And that's why I turned off NoA and only turn it on to test different stretches of highway now.

These days, it mostly makes lane changes without deceleration

Again, this is total chance. I have times where it does and times where it doesn't. There's no obvious rhyme or reason except that sometimes NoA offers a lane change early when approaching a slow moving car and sometimes it waits until I'm at distance 1 before offering. Whatever is powering that logic needs serious work, BTW. It's one of the worst features of NoA.

It's been getting slowly better and better at this sort of thing.

Slowly is an understatement. NoA is basically the same for me since the first day I got it. There have been extremely slight improvements and there have been multiple cycles of regressions. But by and large, it's in no way ready to drive on its own, unaided or unchecked by operators. And remember, this is our sneak preview of FSD that's going to be "feature complete" by the end of next month.

I have experienced phantom braking maybe 3 times in probably 200 commutes.

Many of us can reproduce phantom braking with 100% reliability. You must either not drive under overpasses on your commute, or those overpasses have been whitelisted. If they're whitelisted, be extra careful. In heavy traffic, if cars are stopped under and overpass that is whitelisted, you'll keep driving right into them. Not a very elegant solution, but it has been reported twice that I can recall already.

Phantom braking is such a large concern that Tesla blogged about it in 2016, and they still haven't solved it. And again, FSD "feature complete" end of December 2019.

 

[clydeiii]

It's comments like these that make me really scratch my head. I can't tell if these are cases of crowd delusion because people want to believe so much, or if they're cases where AP somehow actually works flawlessly. If it's the former, then I can understand, but if it's the latter I really have a hard time. What is it about these magical commute paths that makes AP and NoAP work so perfectly when it fails so badly in so many cases??

I can assure you I am not delusional. I think I'm just more forgiving and understanding of this baby brain. It's learning and it keeps getting better and better. A year from now? Thing's gonna be rock solid.

 

[rnortman]

I can assure you I am not delusional. I think I'm just more forgiving and understanding of this baby brain. It's learning and it keeps getting better and better. A year from now? Thing's gonna be rock solid.

I've heard this before, in 2017. And 2018. And throughout all of 2019.

But nevermind that, you have specifically saying that NoA is "flawless" for you, right now. Not in a year -- it is flawless right now, according to your prior posts. Now you're saying that you are very forgiving of it right now, and it will be "rock solid" in a year. Is "rock solid" better or worse than "flawless"? Does "flawless" require forgiveness and understanding?

 

[clydeiii]

I've heard this before, in 2017. And 2018. And throughout all of 2019.

But nevermind that, you have specifically saying that NoA is "flawless" for you, right now. Not in a year -- it is flawless right now, according to your prior posts. Now you're saying that you are very forgiving of it right now, and it will be "rock solid" in a year. Is "rock solid" better or worse than "flawless"? Does "flawless" require forgiveness and understanding?

You heard that about NoA in 2017?

Quick q: did NoA actually exist in 2017?

Dude, I use NoA every day. I can see the improvements with each release. Compared to how it performed when it was released only months ago, it's so much more robust. It is vastly better at figuring out when to change lanes and it do so with more confidence. Could it be better? Absolutely. And I'm sure it'll get there.

For MY commute, it works almost flawlessly (as video evidence will prove whenever I have time to go get a tripod for my phone). For your commute, if it isn't nearly flawless right now, it will get there.

 

[eli_]

Locally I haven't seen it change much since it released in late October 2018. It still gets tripped up on the same highway forks as before. The biggest noticeable change to the feature was in April this year they started allowing unconfirmed lane changes and shipped a new vision NN, but I keep ULC off because there are too many unnecessary route-based lane changes that it's a big hassle to keep canceling them. And then in May they updated the car's Nav map and that may improve NoA routing I guess. Since then I haven't noticed any changes to the planner it uses or anything like that.

 

[timewasted]

I can assure you I am not delusional. I think I'm just more forgiving and understanding of this baby brain. It's learning and it keeps getting better and better. A year from now? Thing's gonna be rock solid.

Ahh yes, the tried and true battle cry of the Tesla faithful: “just wait until next year when it gets better!” Even if it takes multiple “next year”s, it’ll definitely be better next year.

I’m sure there’s some medical term for “paid for something useless and now has to rationalize it to avoid the harsh reality of the situation”. Anyone know what it is?

 

[Buckarooo]

Ahh yes, the tried and true battle cry of the Tesla faithful: “just wait until next year when it gets better!” Even if it takes multiple “next year”s, it’ll definitely be better next year.

I’m sure there’s some medical term for “paid for something useless and now has to rationalize it to avoid the harsh reality of the situation”. Anyone know what it is?

Endowment effect

This bias occurs when we overvalue something that we own, regardless of its objective market value (Kahneman et al., 1991).

 

[ItsNotAboutTheMoney]

Ahh yes, the tried and true battle cry of the Tesla faithful: “just wait until next year when it gets better!” Even if it takes multiple “next year”s, it’ll definitely be better next year.

I’m sure there’s some medical term for “paid for something useless and now has to rationalize it to avoid the harsh reality of the situation”. Anyone know what it is?

There's probably also a name for dismissing other people's experiences that differ from your own.

 

[DrDabbles]

There's probably also a name for dismissing other people's experiences that differ from your own.

Anecdotes should be dismissed. Even mine. What matters is data, and Tesla shares non with us. But we do seem to have a narrative forming from all the owner surveys and forum posts about NoA's behavior. People tend to love it, and not trust it to do the right thing. Owners tend to claim that AP has saved them from what they think would have been a collision, but also believe AP has done dangerous things.

And, given how a NN works, that's pretty much exactly what we'd expect. Seemingly random behavior as the network is trained and approaches local minimums on its descent to a global minimum in some as yet undiscovered area of the graph. The issue that concerns me is that people use adjectives like "perfect" when we know with absolute certainty that the system isn't perfect. We can conclude that not based on anecdotes, but on Tesla's own statement about the system.

 

[clydeiii]

Ahh yes, the tried and true battle cry of the Tesla faithful: “just wait until next year when it gets better!” Even if it takes multiple “next year”s, it’ll definitely be better next year.

I’m sure there’s some medical term for “paid for something useless and now has to rationalize it to avoid the harsh reality of the situation”. Anyone know what it is?

And magically, it's actually true when it comes to deep learning neural networks that are constantly getting feedback from hundreds of thousands of cars in the fleet.

 

[clydeiii]

Anecdotes should be dismissed. Even mine. What matters is data, and Tesla shares non with us. But we do seem to have a narrative forming from all the owner surveys and forum posts about NoA's behavior. People tend to love it, and not trust it to do the right thing. Owners tend to claim that AP has saved them from what they think would have been a collision, but also believe AP has done dangerous things.

Both can definitely be true. That's how you train the NN. And you have to stop and ask yourself, are two brains better than one? I think yes. Clearly some think no.

BTW, I purchased a tripod for my phone. Will hopefully record my PM commute today. It'll be slightly different than normal (starting from a non-standard place) but it is mostly the same thing, just slightly shorter and with one fewer interchange.

 

[clydeiii]

FUN TIMES

 

[DrDabbles]

And magically, it's actually true when it comes to deep learning neural networks that are constantly getting feedback from hundreds of thousands of cars in the fleet.

That isn't how these are being trained. In fact, the stream of data to Tesla can't be constant, since it would swamp the network and would be impossible to process. They collect batches of data from the fleet when they connect to wifi (which you can see if you monitor your network), and those batches are only data that they're somehow interested in.

They also kick off training of the network, so it's not like there's a constant training process happening.

Both can definitely be true. That's how you train the NN. And you have to stop and ask yourself, are two brains better than one? I think yes. Clearly some think no.

I do not understand what you're saying here.

 

[rnortman]

You heard that about NoA in 2017?

Quick q: did NoA actually exist in 2017?

Really? You're going to respond to me by taking a ridiculous interpretation of what I said and pretending that's what I meant? Clearly I meant I have heard the refrain "it will be better in a year" over and over again, with respect to various different AP features. NOA is just the latest one. Meanwhile, TACC continues to get worse every year for me. What I would give to be able to revert to late-2018's TACC...

 

[diplomat33]

Meanwhile, TACC continues to get worse every year for me. What I would give to be able to revert to late-2018's TACC...

Do TACC and AP behave differently? I only use AP. I never use TACC alone. So I can't speak for TACC alone. But AP is noticeably better at "speed control". It used to be jerky at slowing down for cars, brake way too late for stopped cars, take too long to resume acceleration and stop unnecessarily in the middle of the road for cross traffic and then wait too long to resume even when the cross traffic was clear. That behavior is almost completely fixed now. And it is very smooth now in my experience.

 

[clydeiii]

Really? You're going to respond to me by taking a ridiculous interpretation of what I said and pretending that's what I meant? Clearly I meant I have heard the refrain "it will be better in a year" over and over again, with respect to various different AP features. NOA is just the latest one. Meanwhile, TACC continues to get worse every year for me. What I would give to be able to revert to late-2018's TACC...

It's so strange. I'd love to see a video of your issues with it. I have basically no issues with it. It's incredible. Maybe we're talking about a different thing? I don't know.

 

[clydeiii]

That isn't how these are being trained. In fact, the stream of data to Tesla can't be constant, since it would swamp the network and would be impossible to process. They collect batches of data from the fleet when they connect to wifi (which you can see if you monitor your network), and those batches are only data that they're somehow interested in.

They also kick off training of the network, so it's not like there's a constant training process happening.

I realize about the batching, but my point still stands: the more cars in the fleet constantly using NoA will ultimately train the NN. Even if you hate NoA, you should use it as much as possible.

 

[scottf200]

Do TACC and AP behave differently? I only use AP. I never use TACC alone. So I can't speak for TACC alone. But AP is noticeably better at "speed control". It used to be jerky at slowing down for cars, brake way too late for stopped cars, take too long to resume acceleration and stop unnecessarily in the middle of the road for cross traffic and then wait too long to resume even when the cross traffic was clear. That behavior is almost completely fixed now. And it is very smooth now in my experience.

I occasionally just use TACC if I'm in traffic on a non-highway where all lanes are going > 5 MPH over the speed limit (ie. 10). It seems to work exactly the same to be as far as stopping and going. I think it was 36.12 where the take off from a stoplight while behind another vehicle changes considerably. It used to be way to gentle and I could see cars behind you being annoyed. Now it jack rabbits ... almost too fast but is pretty good. Braking has definitely improved on my 2.0 hardware (older radar) based car and my feeling was it was able to use the camera more to just the distance and slow down at the right pace. Occasionly it acts like my previous AP1 car which was borderline scary in late slow downs (so I'd tap the -5 MPH myself a couple times to help). Cross traffic is very noticeably improving. So nothing really new compared to what you said but I 100% agree on the subtle improvements that I've witness and pay very very close attention to.

 

[DrDabbles]

Do TACC and AP behave differently? I only use AP. I never use TACC alone. So I can't speak for TACC alone. But AP is noticeably better at "speed control". It used to be jerky at slowing down for cars, brake way too late for stopped cars, take too long to resume acceleration and stop unnecessarily in the middle of the road for cross traffic and then wait too long to resume even when the cross traffic was clear. That behavior is almost completely fixed now. And it is very smooth now in my experience.

We'd need someone that has disassembled the firmware to comment. But basically AP as far as I can tell is the combination of lane keep assist, and TACC. They all behave the same to me, and I frequently use TACC rather than AP in areas where AP just makes terrible decisions. TACC also fails to operate when the AP computer crashes, which would again seem to indicate that it's using the same inputs and is almost certainly the same system.

I realize about the batching, but my point still stands: the more cars in the fleet constantly using NoA will ultimately train the NN. Even if you hate NoA, you should use it as much as possible.

I used to believe that, too. And then I noticed that Tesla has sole more cars in the past year than it has ever sold in its entire history. And strange enough, AP hasn't made the leaps and bounds improvements everybody said it would. I'm not going to use a feature that makes me annoyed at my car. I'll use EAP, I've disabled NoA automatic engagement, and if the highway is totally empty, I'll engage it occasionally. Otherwise, I'll be using it to test its improvements over releases, and won't be using it for actual driving until it's better than "almost doesn't suck". To quote Elon.