Navigate on Autopilot is Useless (2018.42.3)

[TaoJones]

This ain't good.

As one who has also found AP2.x to be almost completely underwhelming relative to AP1, as well as dangerous (see phantom braking that persists to this day), I cannot wait until I find another AP1 car in order to get out of my current AP2 sled.

My thinking remains that the AP1 car will get me through the next 2-3 years, at which point Tesla will have either embraced MobilEye technology again (highly doubtful to nigh unto impossible), figured out their (still-largely-silent E)AP performance versus promises and representations made for the past 2-4 years, or another manufacturer will surpass their efforts (in UI/UX/AP only - nobody's even close for infrastructure yet).

After reading the reviews of others, I opined recently that the v9.x downgrade ^H^H^H update was two-thirds of a turd. Now that I have driven with it for a bit in two different Model S, evidently I was overly optimistic in even that somewhat generous estimation.

So now we have what, the new SoC/board retrofit next year and v10 to which to look forward in 1.5 years? Woo. "Smart Summon" without stop sign or even speed limit sign reaction (with the currently very broken speed limit db as backup) is just not compelling.

I want what I was sold (and what was clearly represented) in the December 2016 video: stop sign reaction while navigating a selected route. No less, no more (since I doubt the current technology will work worth a tinker's damn with traffic signals, at least at a good number of the intersections hereabouts).

Encountered a SpaceX engineer the other day who had sold his Model S some time ago due to various headaches and hassles. That day he was considering a Model 3. At least he got the first part right.

I hope wk is successful in his upcoming limited-hour effort.

And as a shareholder, I would strongly advise Tesla to reconsider the hiring criteria for that fellow. His efforts and contributions provided, say, 90% remotely would be well worth paying for. It's 2018. Figure it out. That said, I wouldn't wish that environment upon anybody, although maybe the working remotely part would provide enough of a buffer to keep things somewhat in the realm of sanity.

 

[aronth5]

I certainly expected far better when NoA was released. For now I've disabled lane changes and find basic EAP works pretty well with some improvement over the past 6 months.
What I've never understood is why the visualization panel shows cars so far in front of you when what everybody wants is to see are vehicles beside you and far enough behind you to make "good" decisions on lane changes.
If you watch the rear view mirror and compare the mirror to the display vehicles only show up when they are no longer viewable in the mirror. That is too close and unsafe to make a lane change decision. Why not move the car up in the visualization panel so I can see vehicles further behind me. Is their anyone who wouldn't welcome that enhancement? Most of my other comments echo the Ops observations.
I'd like to see Tesla pick 1 or 2 specific items each release make them work and move on. And Tesla be much more specific in the release notes on what is being improved/changed.

 

[DrDabbles]

Rain sensing works fine as long as it isn't misting or dark out :-D

Automatic wipers work even better in my garage, where the wiper work really hard to make sure they clear all that rain that isn't falling on my windshield.

The rain sensing is actually one of the things that annoys me the most- a sensor that already works extremely well in all sorts of scenarios, but they decided to go another route and teach a computer what precipitation looks like. And it turns out it looks like the ceiling of my garage. Which is fantastic.

 

[DrDabbles]

Strange that a beta product is partially baked.

This is Tesla's saving grace, in my opinion. The product is marked Beta, so I cut them slack and I stay ready constantly for it to make a mistake. I don't hold that against Tesla, and I'm not angry at them about it. I'd prefer to take part in the development in whatever way I can.

But there are a lot of people that use EAP as though it's a robotic Lyft driver. The instructions clearly tell you not to do that, the warnings continuously tell you that you're responsible and need to be alert, but we all know that isn't how the average user looks at the system unfortunately.

My worry is that some day something terrible will happen to someone using the system improperly, and a court will find Tesla responsible in some way. All it takes is one judge that doesn't get how this all works, and a loose or non-existent legal framework and we're set back for years.

 

[timewasted]

Everyone else has pretty much summed up my less than impressed feelings of Nav on AP, so I won’t rehash that much. However I will say that for a feature whose sole purpose is making sure you’re in the proper lane, it is ASTONISHINGLY bad at actually changing lanes.

It doesn’t know how to get onto the freeways that I frequent (four lanes condensing down to one). It rarely suggests a lane change that’s actually worthwhile. It can’t gracefully handle two lanes condensing to one or one lane expanding to two. It doesn’t know how to get off the freeways that I frequent (“Unsupported exit” is an interesting thing to see pop up).

This is the first time I’m actively disappointed in a new feature.

 

[Twiglett]

I just gave unmodified, unhacked, fully stock 2018.42.3 Navigate on Autopilot a good try with an open mind on a ~300 mile drive this weekend.

I've seen some videos, tweets, posts, etc praising the feature. I hadn't really had a chance to try it on a longer drive myself as of yet, though, until this weekend.

TLDR version: This is the most useless thing I've ever seen. I've seen some whoppers, but this takes the cake.

Let's do a rundown of what I think was improved:

• Autosteer in highway interchanges and off-ramps was improved. It would stay in the ramp without too much trouble, while prior it would freak out and demand the driver intervene for sharper curves. (We'll ignore that it was taking the turns at ~15 MPH lower than the suggested speeds, but baby steps I suppose).
• Visual indication of what travel lane was needed for upcoming interchanges was reasonable and a good addition to normal navigation.
• I do like the path visualization when lane changes are initiated.
• It does usually try to take exits without intervention (more on this later) which is a step in the right direction for on-ramp to off-ramp autopilot.

So, some improvements I suppose.

Now for the bad.

• I now fully understand why Tesla makes it require confirmation. If it had been allowed to make the suggested lane changes on its own without confirmation, I'd likely have died 10-20x if I didn't take control every time.
• AP1 and AP2 previously did *okay* when following a lane that ended and gradually merged into a single lane. While using NavOnAP this weekend, the car just wanted to make its own lane instead every time instead of merging... usually trying to run into a barrier or median, requiring intervention every time.
• The car regularly suggested lane changes directly into objects it clearly detected. It would even show the proposed path on the visualization as going directly through the other vehicle. In one instance I wondered if it really was going to let me change lanes into a semi truck, or if it would wait until it was clear. Nope, it started to move right towards it after confirmation. No red lane, nothing, while directly along side a semi. *shakes head*
• NavOnAP has no concept of "Keep Right, Pass Left". It never suggests lane changes back to the right in any of the available modes.
• Further, it randomly suggests lane changes to the left for no reason whatsoever. No traffic, no interchanges, nothing.
• I found the car randomly decelerating at least 10x during the trip with no obvious cause. More common when driving in the right lane vs left. It would also set a seemingly random max speed at times, with no speed limit changes or interchanges.
• AP2 still doesn't read speed limit signs, so the noted speed limit doesn't always match the real highway speed limit in areas where it was recently upped or lowered (happens a lot around here with places bumping to 70).
• At least once the car detected a construction zone with a popup about it (kudos on that) and then immediately proceeded to try and suggest a lane change into construction cones..... which negates this from making the "improvements" list above.
  
• Overtake suggestions are useless. On two lanes, driving in the right lane, I would approach a vehicle ahead that was traveling more slowly. No other traffic. The car would decelerate... 5.... 10.... 15 MPH.... as it sees the vehicle. Then, after matching its speed at my set following distance, a few seconds later it'd popup "Confirm lane change" to overtake. Seriously, wtf. And not just once in a while. Every single time I waited for the suggested change, it behaved this way. In every mode setting, including "Mad Max".
  • The car detects the other vehicle way in advance, even when just using the in-car visualization for reference, and could easily make the suggested lane change early enough so that no deceleration at all would be needed, even with the delay of requiring confirmation.
• On multiple occasions the car would start doing a lane change (either a confirmed one, a manually initiated one, or an automatic one for an exit), get part way through, and quickly veer back into the starting lane for no reason. About half of those times it would popup with "Lane change cancelled". In one instance I actually missed an exit because it was 2/3's into the exit ramp lane, stayed there a moment, then just jumped back to the left for no reason.... ugh.
• Even features that were usable before, like manually initiated auto lane changes, are no longer reliable.

Overall, using "Navigate on Autopilot" did not improve the experience of using Autopilot at all, with the limited exception of autosteer's new ability to mostly keep in lane on a tight interchange... with that being negated by the fact that it tries to kill you any time a lane ends. Also, it seems that the ability to take tight interchanges is mostly thanks to nav fusion, as the vision model does not appear to be properly detecting lanes in some of these situations, yet the system presses onward.

The suggested lane changes were completely useless on every mode. It would either suggest changes that weren't necessary, weren't safe, or weren't useful. It was even suggesting lane changes for an interchange upwards of 8 miles away at one point, then refusing to suggest overtake lane changes until after that interchange.

Some more notes:

• Vehicle detection to the sides and behind your vehicle is complete garbage.
  • This is super obvious when sitting still with other still vehicles all around. You'll seem them "swimming" around the visualization, colliding with each other, with you, etc.
  • Also obvious when overtaking large vehicles. Almost every single semi truck, bus, or RV I passed ended up with a twin ghost visual on the screen.
  • Finally, vehicles to the side are regularly shown overlapping my own vehicle visual, despite them being firmly in their own lane.
  • Vehicles behind your vehicle are actually detected only part of the time, apparently due to some issue with the rear cam setup in the hardware (@verygreen I believe has documented this).
• It seems very obvious that Tesla has no real data fusion whatsoever between the cameras. This results in both huge gaps in the usable data as well duplicate data (like the ghost trucks). This is like computer vision 101 stuff that I don't understand why Tesla hasn't overcome this, especially in something shipped to thousands of customers.
• Radar/vision fusion on AP2 appears to be significantly worse than AP1, with AP1 easily accurate for a few cm... AP2 easily worse than +/- 1m... very obvious when looking at the lead vehicle visualization.
• Some of the failings of NavOnAP don't even make sense. If it clearly "sees" a vehicle, it seems like a basic sanity check in the higher level code would prevent it from suggesting a lane change into it.... but this isn't what happens.

Could probably go on for quite a while, but suffice it to say I won't be using the feature any further... not at least until it's actually useful.

It doesn't improve the experience of using autopilot for me one bit. In fact, it makes it even more frustrating. This is ignoring the super frequent nags that plague the more recent firmwares, too.

I'll be sticking to my AP1 vehicles for longer trips from now on I think. In fact, I'm probably going to try and make time to make some videos/posts about AP1/AP2 modifications that are actually useful.

For example, my modded AP1 vehicle would handle the situation I noted above (overtaking a vehicle) smoothly with zero deceleration. AP1 (and AP2) can detect a vehicle ahead of you over 100m away... no excuse for the behavior of NavOnAP.

I'm just super disappointed in Tesla. Their spat with Mobileye has cost Tesla customers a huge amount of progress on the autopilot front. AP1 owners are completely screwed because they will get zero improvements. (Despite promises of ongoing improvements, AP1 hasn't had a single improvement in about two years). Meanwhile, AP1 is running on Mobileye hardware that was released nearly 5 years ago and still handles many situations better than AP2. And it's not like Mobileye has stopped. They're positioned to blow Tesla out of the water with their current hardware (EyeQ4), and off the face of the Earth with their upcoming hardware (EyeQ5). Had Tesla not screwed us all over in that regard, it's likely AP1 would still be improving and that AP2 would be running the next gen of Mobileye hardware with features well beyond what Tesla is capable of doing today. Again, just disappointing that they've decided to forsake early adopters yet again, and also give current adopters less value for their $ in the meantime.

I'm sure people will come out in force to defend Tesla, say how great NavOnAP is, etc... and by all means, do what you must. I personally own both types of vehicles (AP1 and AP2) and drive both regularly... pretty simple to tell the deficiencies of AP2. You're not going to convince me that somehow my extensive first-hand experience is somehow flawed and that things are way better than I claim. lol.

I find it hard to believe that as a one-man free-time dev crew I've been able to do better than Tesla's entire multi-million-dollar-funded AP dev team has been able to do in more than two years. I have maybe 40 hours of total work into my AP1 mods, and they've been more usable than NavOnAP for nearly two years. I just don't get it. I'm pretty good at what I do, but I can't believe I'm better-than-full-teams-with-millions-in-funding good.

To that end, when/if I get the time, I'm going to set a goal for myself of ~25 hours of work to make a hardware/software modification to an AP2 vehicle that actually does what NavOnAP is supposed to do. Start to finish, from scratch. It's been suggested that I get some basics prepped (hardware I need in-hand, for example), then keep a GoPro on a chesty running until the modifications are done and working to document the entire thing.

If doable, then I think it would be even more obvious that we have to reevaluate Tesla's progress on the driver assistance front.

Anyway, enough of that for now.

Disclaimer: I have no positions with any of the companies mentioned nor do I intend to initiate one at any point in the future.

Mostly agree with everything there, just wouldn’t say it’s completely useless though.
Just got back from a long trip to Houston and back and used NoA.
It really does head to the left lane and almost never comes back. Being able to choose a lane would be so much better.
Real highway changes were handled quite well, but many lane changes seemed to arrive just as other cars blocked it. I was waiting for the lane change to arrive which made it worse
Having said that I’m looking forward to seeing what comes next.
So while I fully agree with almost everything that @wk057 mentioned, my take is rather more positive.

 

[TradingInvest]

This is just one data point. I have a Model 3 since April. Got the latest software update 3~4 days ago. I tried several times, each time 10~30 miles. Tried both daytime and nighttime. Over all, I'm quite happy with the performance. There was one time, after my approval, the car started to switch to the other lane, but 1/3 through the process, suddenly switched back to the original lane, because the car felt there wasn't enough distance in the new lane between my car and the car in front of me. I think if you feel the distance is not enough, then don't start the lane change. Also there was a time the car mixed an exit because it suggested to exit too late. I let the car continue and took a different route to home, the remaining drive was perfect. I wouldn't be surprised if Tesla fix the issues in future releases and continue to make the feature better. I feel if the car can't see the other cars, that would be a major issue. As far as I can see, right now the car can detect other cars, it's the decision logic that needs update.

Personally, I really love this feature, I always pay attention to the road and the soundings. I can't imagine going back to a car without autopilot. All of my future cars have to have this feature.

Also regarding the windshield wipers, initially they were useless, mainly because they were too slow. Now they have improved to a point I don't need to interfere. Today it was raining and my son made the comment: the automatic wipers now work perfectly. I felt the same.

 

[Boomer19]

Audi, Caddilac, Waymo and others are ready to take responsibility for their car's actions;

what do you mean?

 

[Boomer19]

@wk057
excellent information across your posts here, and also good +/- feedback from most others.

i’m a noob (or a boob) when it comes to the tech of AP, and don’t even have the latest version yet (kinda miffed about it as well).

my question, to everyone really, is why?
why is tesla not up to speed on this? is it internal strife about a direction to take? disagreement on methodology? budget? head butting with old guard versus avant- guard?
is it that karpathy hasn’t been there long enough, to cultivate a team, set a game plan, get everyone on board, and start executing?
are they on the cusp?

any thoughts on this, i think, would be as interesting as the tech itself. FSD seems such a monumental challenge, yet people speak as if it’s so apparently going to be as plain as day...i’m just trying to grasp what forces have to align in order to find success on this front.

thanks.

 

[Snuffysasa]

Elon is promising Full Self-Driving by next year;

Is he??!!

When will Tesla release any level 4 cars?

I do not know this, but I feel it will be a long long ways away if ever.

Audi, Caddilac, Waymo and others are ready to take responsibility for their car's actions; how long can Tesla wait.

I am not sure what you are saying here?

Are you suggesting these companies have or are planning on releasing products / services that are beyond level 2 / Driver assist, and they will be responsible for accidents caused by the vehicles, rather than the driver?

I agree. However, that is not the case with the Super Cruise in the Cadillac. But perhaps will be the case with future GM/Cadillac systems.

 

[Ketchups]

Tell us how you really feel Jason...

 

[PhaseWhite]

@wk057 Thank you for taking the time to write up your detailed observations. I hope someone on the AP team is reading this post and takes these criticisms seriously.

Lets hope the issues are in the control systems that leverage the NN outputs to make decisions and not something fundamental in the NN approach.

You hinted that you have made some mods to AP1. That sounds interesting, would you be able to share more details?

 

[SongjinDK]

I would like to share a little bit of my experience (2018 MX AP2.5).

1. V9 (39.7 and 40) has big improvements for lane changing compared with V8. For V8, I need to check and make sure it is safe and there is no car in blind spot before turning the signal. For V9, I have tested in all different cases and I have built a trust that I will just turn the signal and let the car change it. The car will speed up or slow down or wait, etc. and made a safe lane change. There is a few cases where it has to abort in the middle of change because another car was changing to the same lane at the same time, then it wait and change to that lane later, which was very impressive (If I did it, I might have hit the car since another car changed to the same lane after I checked the blind spot). The only problem is for very heavy traffic, the car is too conservative and it may take long time so I usually take over and squeeze in another lane then re-enage AP.

2. For NoA (42), it is very useful to not have to worry about missing exit or change to a wrong freeway, this itself is very helpful to me.

3. Because NoA is not available before entering freeway, I never used it for on ramp, I always enable it when merged in.

4. I usually change lanes myself for over taking (just turn the signal, no need to check blind spot or traffic, the car will change lane. If the car suggest a lane change to overtake, I usually confirm it immediately (one tap is easier than enraging the turn signal).

5. It tends to suggest lane change for exit very early to be conservative (about 2 mile). I usually wait until about 1/2 or 1/4 before confirming it, usually it does fine. If there is too much traffic, I take over and change the lane myself. At this point I already know which lane I should be in.

6. Occupationally if there is phantom brake, I step on the accelerator to override it, this usually fix it immediately.

Overall, I found the auto lane change and NoA are very useful driver assistant tools and it reduced work load dramatically. Hope this helps.

 

[verygreen]

2. For NoA (42), it is very useful to not have to worry about missing exit or change to a wrong freeway, this itself is very helpful to me.

unless the maps are wrong and keep you in the wrong lane. then you miss the exit. I know for a couple of places like this.

4. I usually change lanes myself for over taking (just turn the signal, no need to check blind spot or traffic, the car will change lane. If the car suggest a lane change to overtake, I usually confirm it immediately (one tap is easier than enraging the turn signal).

you know you are using the system incorrectly, right? In the unlikely event that something happens during this - you are at fault.

6. Occupationally if there is phantom brake, I step on the accelerator to override it, this usually fix it immediately.

The driver behind you totally loves it when you brake-check them like that, helps them stay awake during their otherwise boring morning/evening commutes. On weekends passengers also appreciate sudden jerkiness.

 

[S4WRXTTCS]

Telsa's Motion planning and control algorithms is straight garbage among other things, in-addition to their terrible in-accurate and in-efficient network my friend. But be-careful, you can easily go from a Saint to a foe around these quarters now.

Jimmy has this entire forum hypnotized!
I have even felt strong resistance from @lunitiks and @S4WRXTTCS of all people.

What does Jimmy have to do with planning and control algorithms?

Like I said before Jimmy is a Neural Network guy.

The problem with you is you purposely try to find as much negative about AP that you even people like me who find AP problematic have to purposely say something positive to counter it. You encounter heavy resistance because some of us don't believe you're fair. You don't even own the car nor do you have any interest in one. Instead you just sit on the sidelines poking fun of Tesla while you wait for whatever it is that you plan on actually buying. You should be enjoying life in a Cadillac with super cruise at the very least.

You skew objectivity on this forum. I even made a joke about it on one thread where I'm like "Ugh, I hope Blader isn't lurking because this sucks".

What I find hilarious is I don't think AP3 is capable of L3 driving (with the current sensor suite), and you do. So in some ways you're more positive about Tesla than I am.

But, I don't go on and on about being right because I don't want to be right.

At some point you're going to have to be positive about Tesla/AP to be right about AP3 being capable of L3 driving.

 

[cwerdna]

Elon is promising Full Self-Driving by next year; if Tesla's can't full self-drive on limited access highways, what chance do they have in crowded cities.

Is he??!!

Sorta. See Elon Musk: EAP solved, on track for FSD completion in 2019 (No one else is close!). That poster points to Elon Musk: The Recode interview from 11/2/18

The other car companies ... I don’t wanna sound overconfident, but I would be very surprised if any of the car companies exceeded Tesla in self-driving, in getting to full self-driving.

You know, I think we’ll get to full self-driving next year. As a generalized solution, I think. But that’s a ... Like, we’re on track to do that next year. So I don’t know. I don’t think anyone else is on track to do it next year.

I do not know this, but I feel it will be a long long ways away if ever.

Ditto

Agreed with postpast's sentiment.

 

[Bladerskb]

What I find hilarious is I don't think AP3 is capable of L3 driving (with the current sensor suite), and you do. So in some ways you're more positive about Tesla than I am.

But, I don't go on and on about being right because I don't want to be right.

At some point you're going to have to be positive about Tesla/AP to be right about AP3 being capable of L3 driving.

L3 highway in good weather. 2021+. Yes. But dont confuse that with urban surface street environment.

 

[Silent Ludicrosy]

Are autopilot override events flagged and sent to the Tesla autopilot team for analysis? Obviously they cant look at every event, but i would hope they’re using real world feedback. Maybe even use some sort of AI to look for trends.

 

[RayW]

Great thread.

BTW, Teslas don't have to to run Tesla's software, do they?
This may be an opportunity for after market software... maybe there's a way to load from a USB and some secret left-right-up-down clicks of the thumb wheels

 

[verygreen]

Sorta. See Elon Musk: EAP solved, on track for FSD completion in 2019 (No one else is close!). That poster points to Elon Musk: The Recode interview from 11/2/18


Ditto

Agreed with postpast's sentiment.

that is a *very* misleading choice of words if you ask me.

but I would be very surprised if any of the car companies exceeded Tesla in self-driving, in getting to full self-driving.

How many of the car companies you know are working on this? On the other hand who are the contenders? Intel (MobilEye), Waymo, Uber(?), ... none of them are car companies (those prefer to license ready made solutions anyway - it's safer this way too - somebody else is taking all the R&D risks! And the car companies can concentrate on their core strengths (or at least that's what PHBs/MBA would like you to believe as the good thing)). GM bought whoever does supercruise I think, but do they even aim at FSD ATM, any autonomous miles logged anywhere?

So... re-reviewing that quote again what can we infer? No other car companies are close to what Tesla is doing (sorta true, because they are not even trying) and next year there might be (on track) a generalized solution from FSD at Tesla (whatever that means, certainly does not sounds like anything in a production car to me. Esp. coming from the "there should be zero doubt about 10k/week model 3s being produced by the end of 2018". If you ask me - the "we are on track" is a lot weaker wording than the zero doubts one. The timeframe is similar, the 0 doubts is from mid-2017).

In other words Elon is certainly not promising anything next year no matter how much you want to believe he does. He's hinting that they are not even close (in his own unique ways where he must infer strength when there's not any, of course).