Navigate on Autopilot is Useless (2018.42.3)

[sandpiper]

You are wrong. Regulations do not require hands on the wheel. BMW allows "hands off" below a certain speed on one of their cars. Cadillac allows "hands off" at any speed on one of their cars. There was a car that predated both the BMW and Cadillac by a full 4 years that allowed it as well, trying to remember the name..... oh yeah, I think it was a company called Tesla in 2015 (powered by Mobileye).

Well.... technically Tesla DID require that you keep your hands on the wheel. They just didn't enforce it much in the early incarnations of AP1. That changed due to a few high profile accidents and too many idiots posting their stupidity on youtube.

 

[croman]

To be clear only New York state requires hands on the wheel as far as I know. Other states laws just assume it is needed to not be negligent which is the catch-all.

 

[S4WRXTTCS]

To be clear only New York state requires hands on the wheel as far as I know. Other states laws just assume it is needed to not be negligent which is the catch-all.

Does anyone know if Cadillac enforces that rule for Supercruise in New York? The blurb on their website says "Only remove your hands from the steering wheel if Supercruise is engage, it is safe to do so, and is permitted by state and local laws.". That to me means they leave it up to the driver to follow local laws. Just like all cars on the road today largely leave the task of following laws to the driver.

Supercruise is only the first of many L2 cars that will use eye sensors to detect if the driver is engaged or not.

Tesla as many other L2 vehicle manufactures still use torque sensors which obviously don't work very well considering how annoying they are, and how useless they seem to be at preventing a person from falling asleep at the wheel.

The Model 3 torque sensor is so bad it fails to encourage a person to hold the steering wheel. The nag is there whether I'm holding the steering wheel correctly. or if I'm not holding it at all. The only time I don't get a nag is when I hold it in a position that isn't conducive to quickly resolving an autosteer issue.

 

[DarkMatter]

Have you tried setting the steering to 'sport'? That reduced the false alert nags with my hands on the wheel by quite a bit. Same for a friend who is a pilot. I think people who tend to track the wheel with very little force get lots of false positives, and for whatever reason Sport seems to make it more sensitive.

 

[GlmnAlyAirCar]

Have you tried setting the steering to 'sport'? That reduced the false alert nags with my hands on the wheel by quite a bit. Same for a friend who is a pilot. I think people who tend to track the wheel with very little force get lots of false positives, and for whatever reason Sport seems to make it more sensitive.

I have always driven in sport and have never had any issues with nags...so perhaps there is some truth to this.

 

[S4WRXTTCS]

Have you tried setting the steering to 'sport'? That reduced the false alert nags with my hands on the wheel by quite a bit. Same for a friend who is a pilot. I think people who tend to track the wheel with very little force get lots of false positives, and for whatever reason Sport seems to make it more sensitive.

I originally had it in Sport, and then someone told me to try comfort mode to reduce nags.

So yeah I've tried sport and comfort mode.

Whatever the mode it's in it's way worse than it was with the Model S I had. With the Model S I could drive with a single hand on the steering wheel with hardly ever getting a nag.

 

[PhaseWhite]

I originally had it in Sport, and then someone told me to try comfort mode to reduce nags.

So yeah I've tried sport and comfort mode.

Whatever the mode it's in it's way worse than it was with the Model S I had. With the Model S I could drive with a single hand on the steering wheel with hardly ever getting a nag.

I've found holding the wheel like the Tesla NOA Demo video shows gives good results on my Model 3 to eliminate nag. That is holding the wheel with both hands on the bottom at 8 and 4 o-clock position with palms facing me.

The sad truth is that the torque sensor worked fine when the nag interval was longer (higher chance of registering torque in a given time period) but when Tesla had to lower the nag interval aggressively it ended up not being sensitive enough. As with many things in life, the poor choices of a few people result in restrictions on the larger group that were following the rules already

I'm hopeful that as Autopilot improves that the nags will become slightly less aggressive.

 

[Soda Popinski]

I originally had it in Sport, and then someone told me to try comfort mode to reduce nags.

So yeah I've tried sport and comfort mode.

Whatever the mode it's in it's way worse than it was with the Model S I had. With the Model S I could drive with a single hand on the steering wheel with hardly ever getting a nag.

It could be a function of the size of the steering wheel. A larger wheel means a larger "lever" allowing you to put more torque on the wheel with the same weight. Would be nice if the sensitivity could be adjusted accordingly by Tesla.

 

[PhaseWhite]

I just gave unmodified, unhacked, fully stock 2018.42.3 Navigate on Autopilot a good try with an open mind on a ~300 mile drive this weekend.

I've seen some videos, tweets, posts, etc praising the feature. I hadn't really had a chance to try it on a longer drive myself as of yet, though, until this weekend.

TLDR version: This is the most useless thing I've ever seen. I've seen some whoppers, but this takes the cake.

Let's do a rundown of what I think was improved:

• Autosteer in highway interchanges and off-ramps was improved. It would stay in the ramp without too much trouble, while prior it would freak out and demand the driver intervene for sharper curves. (We'll ignore that it was taking the turns at ~15 MPH lower than the suggested speeds, but baby steps I suppose).
• Visual indication of what travel lane was needed for upcoming interchanges was reasonable and a good addition to normal navigation.
• I do like the path visualization when lane changes are initiated.
• It does usually try to take exits without intervention (more on this later) which is a step in the right direction for on-ramp to off-ramp autopilot.

So, some improvements I suppose.

Now for the bad.

• I now fully understand why Tesla makes it require confirmation. If it had been allowed to make the suggested lane changes on its own without confirmation, I'd likely have died 10-20x if I didn't take control every time.
• AP1 and AP2 previously did *okay* when following a lane that ended and gradually merged into a single lane. While using NavOnAP this weekend, the car just wanted to make its own lane instead every time instead of merging... usually trying to run into a barrier or median, requiring intervention every time.
• The car regularly suggested lane changes directly into objects it clearly detected. It would even show the proposed path on the visualization as going directly through the other vehicle. In one instance I wondered if it really was going to let me change lanes into a semi truck, or if it would wait until it was clear. Nope, it started to move right towards it after confirmation. No red lane, nothing, while directly along side a semi. *shakes head*
• NavOnAP has no concept of "Keep Right, Pass Left". It never suggests lane changes back to the right in any of the available modes.
• Further, it randomly suggests lane changes to the left for no reason whatsoever. No traffic, no interchanges, nothing.
• I found the car randomly decelerating at least 10x during the trip with no obvious cause. More common when driving in the right lane vs left. It would also set a seemingly random max speed at times, with no speed limit changes or interchanges.
• AP2 still doesn't read speed limit signs, so the noted speed limit doesn't always match the real highway speed limit in areas where it was recently upped or lowered (happens a lot around here with places bumping to 70).
• At least once the car detected a construction zone with a popup about it (kudos on that) and then immediately proceeded to try and suggest a lane change into construction cones..... which negates this from making the "improvements" list above.
  
• Overtake suggestions are useless. On two lanes, driving in the right lane, I would approach a vehicle ahead that was traveling more slowly. No other traffic. The car would decelerate... 5.... 10.... 15 MPH.... as it sees the vehicle. Then, after matching its speed at my set following distance, a few seconds later it'd popup "Confirm lane change" to overtake. Seriously, wtf. And not just once in a while. Every single time I waited for the suggested change, it behaved this way. In every mode setting, including "Mad Max".
  • The car detects the other vehicle way in advance, even when just using the in-car visualization for reference, and could easily make the suggested lane change early enough so that no deceleration at all would be needed, even with the delay of requiring confirmation.
• On multiple occasions the car would start doing a lane change (either a confirmed one, a manually initiated one, or an automatic one for an exit), get part way through, and quickly veer back into the starting lane for no reason. About half of those times it would popup with "Lane change cancelled". In one instance I actually missed an exit because it was 2/3's into the exit ramp lane, stayed there a moment, then just jumped back to the left for no reason.... ugh.
• Even features that were usable before, like manually initiated auto lane changes, are no longer reliable.

Overall, using "Navigate on Autopilot" did not improve the experience of using Autopilot at all, with the limited exception of autosteer's new ability to mostly keep in lane on a tight interchange... with that being negated by the fact that it tries to kill you any time a lane ends. Also, it seems that the ability to take tight interchanges is mostly thanks to nav fusion, as the vision model does not appear to be properly detecting lanes in some of these situations, yet the system presses onward.

The suggested lane changes were completely useless on every mode. It would either suggest changes that weren't necessary, weren't safe, or weren't useful. It was even suggesting lane changes for an interchange upwards of 8 miles away at one point, then refusing to suggest overtake lane changes until after that interchange.

Some more notes:

• Vehicle detection to the sides and behind your vehicle is complete garbage.
  • This is super obvious when sitting still with other still vehicles all around. You'll seem them "swimming" around the visualization, colliding with each other, with you, etc.
  • Also obvious when overtaking large vehicles. Almost every single semi truck, bus, or RV I passed ended up with a twin ghost visual on the screen.
  • Finally, vehicles to the side are regularly shown overlapping my own vehicle visual, despite them being firmly in their own lane.
  • Vehicles behind your vehicle are actually detected only part of the time, apparently due to some issue with the rear cam setup in the hardware (@verygreen I believe has documented this).
• It seems very obvious that Tesla has no real data fusion whatsoever between the cameras. This results in both huge gaps in the usable data as well duplicate data (like the ghost trucks). This is like computer vision 101 stuff that I don't understand why Tesla hasn't overcome this, especially in something shipped to thousands of customers.
• Radar/vision fusion on AP2 appears to be significantly worse than AP1, with AP1 easily accurate for a few cm... AP2 easily worse than +/- 1m... very obvious when looking at the lead vehicle visualization.
• Some of the failings of NavOnAP don't even make sense. If it clearly "sees" a vehicle, it seems like a basic sanity check in the higher level code would prevent it from suggesting a lane change into it.... but this isn't what happens.

Could probably go on for quite a while, but suffice it to say I won't be using the feature any further... not at least until it's actually useful.

It doesn't improve the experience of using autopilot for me one bit. In fact, it makes it even more frustrating. This is ignoring the super frequent nags that plague the more recent firmwares, too.

I'll be sticking to my AP1 vehicles for longer trips from now on I think. In fact, I'm probably going to try and make time to make some videos/posts about AP1/AP2 modifications that are actually useful.

For example, my modded AP1 vehicle would handle the situation I noted above (overtaking a vehicle) smoothly with zero deceleration. AP1 (and AP2) can detect a vehicle ahead of you over 100m away... no excuse for the behavior of NavOnAP.

I'm just super disappointed in Tesla. Their spat with Mobileye has cost Tesla customers a huge amount of progress on the autopilot front. AP1 owners are completely screwed because they will get zero improvements. (Despite promises of ongoing improvements, AP1 hasn't had a single improvement in about two years). Meanwhile, AP1 is running on Mobileye hardware that was released nearly 5 years ago and still handles many situations better than AP2. And it's not like Mobileye has stopped. They're positioned to blow Tesla out of the water with their current hardware (EyeQ4), and off the face of the Earth with their upcoming hardware (EyeQ5). Had Tesla not screwed us all over in that regard, it's likely AP1 would still be improving and that AP2 would be running the next gen of Mobileye hardware with features well beyond what Tesla is capable of doing today. Again, just disappointing that they've decided to forsake early adopters yet again, and also give current adopters less value for their $ in the meantime.

I'm sure people will come out in force to defend Tesla, say how great NavOnAP is, etc... and by all means, do what you must. I personally own both types of vehicles (AP1 and AP2) and drive both regularly... pretty simple to tell the deficiencies of AP2. You're not going to convince me that somehow my extensive first-hand experience is somehow flawed and that things are way better than I claim. lol.

I find it hard to believe that as a one-man free-time dev crew I've been able to do better than Tesla's entire multi-million-dollar-funded AP dev team has been able to do in more than two years. I have maybe 40 hours of total work into my AP1 mods, and they've been more usable than NavOnAP for nearly two years. I just don't get it. I'm pretty good at what I do, but I can't believe I'm better-than-full-teams-with-millions-in-funding good.

To that end, when/if I get the time, I'm going to set a goal for myself of ~25 hours of work to make a hardware/software modification to an AP2 vehicle that actually does what NavOnAP is supposed to do. Start to finish, from scratch. It's been suggested that I get some basics prepped (hardware I need in-hand, for example), then keep a GoPro on a chesty running until the modifications are done and working to document the entire thing.

If doable, then I think it would be even more obvious that we have to reevaluate Tesla's progress on the driver assistance front.

Anyway, enough of that for now.

Disclaimer: I have no positions with any of the companies mentioned nor do I intend to initiate one at any point in the future.

I think it's a good time to revisit this topic. Tonight I installed firmware 2019.8.5 3aaa23d, turned off the lane change confirmation and went for a for a drive in my Model 3. Conditions were dark with a light drizzle which previously might have required me to manually engage the wipers but to their credit, auto wipers worked perfectly.

I immediately noticed that NOA now initiates a "lane change" maneuver when merging in, something which it seemed to be missing before.

I found lane changes to be much more confident and at one point I had to merge onto a road with traffic and and exit only about 1/4 mile away on opposite side, so NOA had to cross 3 lanes of traffic and it made it. It seemed to realize the urgency and immediately initiate each lane change.

Exits have also gotten better, there were times on previous versions it would throw it's self so fast into the exit that I had to disengage because I wasn't sure if it would hit the guard rail. This still isn't perfect as it seemed to ping pong slightly on exit but I didn't feel uncomfortable in any of the cases I tested so I didn't have to disengage.

Overtaking slow cars is pretty cool but is still conservative in the sense it won't follow mad max mode if you are getting close to your exit so you still have to follow that super slow car for 2 miles. Probably a the right trade off to make until Tesla is more confident their system can get you back in the exit position even in busy traffic or perhaps they could have an algorithm that gauges traffic difficulty level and in light traffic it will be less risk adverse about overtaking maneuvers.

Also one last point, I had my hands on the wheel as is required and I did not experience a single nag and it seems like Tesla had made changes to the approach they use for nags such that they are not bothering me much if at all (And I always keep my hands on the wheel)

@wk057 I'm curious to hear your take after you get the update.

 

[avoigt]

I drove this weekend 85 km on a German Autobahn with 2 construction zones, 1 Autobahn change, 2 traffic jams and parts where I could drive full speed up to 150km/h with many lane changes.

It was a great experience and I did not have to interrupt or take over a single time.

Clearly you need to be awake and ready to step in and sometimes it was not clear to me why the car did not a lane change or why did it wanted to although it did see it can't due to other cars but beside that it was just awesome.

What a great piece of technology, love it !

 

[mhan00]

Drove 60 miles or so round trip on NOA last night. I was quite impressed. I can’t wait u til wk and green get to tear it down and give their opinions on it.

 

[diamond.g]

Still refuses to do a clover interchange (I64 to 168 Expressway).

 

[wk057]

@wk057 I'm curious to hear your take after you get the update.

(You may be sorry you asked this...)

So, I got the update on a Model 3. Decided it'd been long enough since I gave NavOnAP a good try, and it probably improved since then. Drove ~130 miles today with it.

Boy was I wrong.

Within the first mile of engaging NavOnAP with ULC, it starts a lane change from the left lane to the right. I look at the visualization, and out the window. Clearly a truck there. No way it's going to actually start moving, right? Wrong. It starts heading straight into the truck, route line drawn on the screen directly through the truck in the target lane, and the car is just happily about to smash into it. "F*** this!" I blurted as I jerked control back.

Ok ok. Was just a fluke. Will give it a go again.

Went alright for a few more miles. The car made a mostly sensible move out of the passing lane after a bit. It immediately detected a slower car a little ways ahead and started back to the left, making me look like a jackass, but, whatever. Was sane enough.

Get a bit closer to the car I'm overtaking, which is going maybe 12 MPH slower than I am... about to overtake it and BAM, nearly a full panic brake out of no where down to about 5 MPH slower than the car I'm passing. Luckily no one was behind me. The other car was clearly in their lane, nothing ahead of my car but clear open highway. Seriously, wtf...

I let it do its thing, and it phantom braked off and on the whole way past this car. It was comical. I'd get part way overtaking them, the car would slow down and get partly behind them, then pass again. The other driver must have thought I was drunk or something. Then, with the car being overtaken partially passed, the car is like, "Lane change"... again directly into the car in that lane that it is clearly aware of. It was pretty subtle and I actually didn't catch it until I was on the dashed line. Finally gave up and disabled NavOnAP, and got past that car.

About 10 miles more before the first interchange. The lane change decisions it made were okay... not the best, but at least didn't try to move into other vehicles. Few phantom brakes again... and only while NavOnAP was enabled. Tried normal AP for a bit without the braking issues, which was weird. While I did have phantom braking problems last time I did a similar trip with NavOnAP, it was definitely way worse on this newer version.

Btw, this is a different car than last time, too (newer). No hacks, root, mods, etc either.

Anyway, it tooks the interchange like a champ. Nice.

Gets back on the highway without much issue too, including the merge. Not bad. Oh, but it's still stuck at the interchange speed setting while full on the new highway now. I bump it back up to the flow of traffic with the touchscreen. Nothing, stays there. Use the accelerator pedal to override, get to speed, let off, full regen back down to 55. No errors showing, nothing. No clue why it wasn't following my set speed.

Disabled AP entirely, reengaged, and it worked as it should.

This leg of the trip went okay, and was mostly a longer straight stretch. The lane change decisions definitely need a lot of work. It was constantly switching back and forth for no good reason. Tried all four modes with similar results.

Was some random braking/slowing, even when no other cars were around. Super weird, and super annoying. And wasn't happening with NavOnAP disabled. Happened back to back a dozen times in a row one time while someone was behind me. The person sped around me, probably thought I was brake checking them or something.

Last interchange of that leg was a no go. It started to take it, then decided, "This is close enough" and tried to just drive dead center of the left white line of the exit... wow. lol.

The return trip was worse. Didn't successfully take a single interchange without intervention. Several more attempts to change lanes directly into traffic. And tons and tons of phantom braking.

I completely gave up on it about 10 miles from home and just used normal AP, which worked alright. It still doesn't like the 3 spots on I-40 where the newly redone overpasses don't have markings and tried to kill us at all 3, but okay. (psst... stock AP1 handles those spots flawlessly).

We get to the last exit and I'm like, "Ok, lets see what it does." and I tap NavOnAP back on. Starts out okay. We get into the exit (it splits after the first part).. it starts the charge to the right... we move partly into that lane... BAM, HARD LEFT!!!!. Car turned out of the correct exit lane, back into and past the other lane before I could correct it and get us back on the right path. Mind you, I'm letting it take this exit at ~20 MPH less than any sane driver would bother, too, and it still screws it up. Clear markings, unambiguous.

My g/f: "I'm never using this."

Suffice it to say, I'm not impressed. Figured I'd share this misadventure before I headed to bed.

 

[Sonic_78]

In any urban highway driving I only set lane change to mild, anything else and it’s “rule of the road” logic isn’t followed by human drivers. Basically, get me in a lane and leave me there until I gotta exit or interchange.

Different on rural or turnpikes highways / left lane passing only, right lane cruising, 99% drivers on cruise control. Mad max it there and it works really well.

And you gotta watch it because if a car nearly inches into your lane from and adjacent lane, it’ll brake hard to pull you back. Again, shouldn’t be a problem if traffic is following you at the correct distance, but who does that. Lucky if you can see the grill on most cars following you. So yeah, kinda dangerous.

 

[mhan00]

Interesting re: wk057’s experience vs my own. Looks like he’s in NC, which really does make me wonder if the map difference greentheonly mentioned is the main difference.

 

[DarkMatter]

I think that the training set may just be unhappy with road markings in some places. The experience is so different. Going to put a hundred miles on it tomorrow and expect only to have to disengage it in the section where I know the map is utterly wrong because they keep moving the lanes. Today was 40 miles with zero drama.

 

[PhaseWhite]

Interesting re: wk057’s experience vs my own. Looks like he’s in NC, which really does make me wonder if the map difference greentheonly mentioned is the main difference.

I was thinking the same thing about maps. The other thing @wk057 might do to reduce the number of lane changes is Speed based lane change setting to Average vs MadMax.

 

[DrDabbles]

Suffice it to say, I'm not impressed. Figured I'd share this misadventure before I headed to bed.

Your situation makes me think there's something wrong with the car. Perhaps an improperly calibrated EAP, a camera misaligned, or something. My car has the opposite symptoms as yours- it's paranoid of other vehicles when changing lanes even if there's a lane between us, so it won't attempt lane changes until I'm easily half a car length away or more. I get the red line and all.

The interchange speed limit thing is unbelievably annoying and they need to get rid of it. It's a stupid idea in the first place, and it's dangerous in a lot of situations where you're doing 20 MPH less than the rest of traffic. Flat out, this is a terrible feature.

The exit behavior is pretty well known. If your exit splits, you need to take over.

 

[wk057]

The other thing @wk057 might do to reduce the number of lane changes is Speed based lane change setting to Average vs MadMax.

The lane change decisions definitely need a lot of work. It was constantly switching back and forth for no good reason. Tried all four modes with similar results.

^

Your situation makes me think there's something wrong with the car.

I thought this as well, since it definitely felt odd that there would be such a regression. For unrelated purposes, I drove a completely different car (friend's late-2018 S that just got the update) this morning on part of the same route, and had pretty much the same results as yesterday.

Also, the car... both cars, actually, behave perfectly fine with NavOnAP disabled and normal AP working. No phantom braking, and obviously no changing lanes into other vehicles.

I don't feel like a calibration issue could be the cause of it attempting lane changes into vehicles it's clearly displaying on the visualization, though, since it can obviously see them if it's showing them on the screen. Like, it would draw the lane change path through the other vehicle on the screen. Going to have to setup a cabin GoPro next time I do a longer trip in the 3 to see the screen and surroundings in the frame.

Definitely seems like a YMMV situation, since other's are reporting glorious results... so, who knows. I'm definitely not losing my mind, since I had passengers today and yesterday who witnessed the same shenanigans.

 

[thewishmaster]

^
I don't feel like a calibration issue could be the cause of it attempting lane changes into vehicles it's clearly displaying on the visualization, though, since it can obviously see them if it's showing them on the screen. Like, it would draw the lane change path through the other vehicle on the screen. Going to have to setup a cabin GoPro next time I do a longer trip in the 3 to see the screen and surroundings in the frame.

Don’t think this particular visualization matters too much - mine happily draws paths through cars, but it also respects their boundaries a lot more than yours seems to ...