TMC is an independent, primarily volunteer organization that relies on ad revenue to cover its operating costs. Please consider whitelisting TMC on your ad blocker and becoming a Supporting Member. For more info: Support TMC

Navigate on Autopilot is Useless (2018.42.3)

Discussion in 'Autonomous Vehicles' started by wk057, Nov 4, 2018.

  1. wk057

    wk057 Senior Tinkerer

    Joined:
    Feb 23, 2014
    Messages:
    5,184
    Location:
    Hickory, NC, USA
    I just gave unmodified, unhacked, fully stock 2018.42.3 Navigate on Autopilot a good try with an open mind on a ~300 mile drive this weekend.

    I've seen some videos, tweets, posts, etc praising the feature. I hadn't really had a chance to try it on a longer drive myself as of yet, though, until this weekend.

    TLDR version: This is the most useless thing I've ever seen. I've seen some whoppers, but this takes the cake.

    Let's do a rundown of what I think was improved:
    • Autosteer in highway interchanges and off-ramps was improved. It would stay in the ramp without too much trouble, while prior it would freak out and demand the driver intervene for sharper curves. (We'll ignore that it was taking the turns at ~15 MPH lower than the suggested speeds, but baby steps I suppose).
    • Visual indication of what travel lane was needed for upcoming interchanges was reasonable and a good addition to normal navigation.
    • I do like the path visualization when lane changes are initiated.
    • It does usually try to take exits without intervention (more on this later) which is a step in the right direction for on-ramp to off-ramp autopilot.

    So, some improvements I suppose.

    Now for the bad.
    • I now fully understand why Tesla makes it require confirmation. If it had been allowed to make the suggested lane changes on its own without confirmation, I'd likely have died 10-20x if I didn't take control every time.
    • AP1 and AP2 previously did *okay* when following a lane that ended and gradually merged into a single lane. While using NavOnAP this weekend, the car just wanted to make its own lane instead every time instead of merging... usually trying to run into a barrier or median, requiring intervention every time.
    • The car regularly suggested lane changes directly into objects it clearly detected. It would even show the proposed path on the visualization as going directly through the other vehicle. In one instance I wondered if it really was going to let me change lanes into a semi truck, or if it would wait until it was clear. Nope, it started to move right towards it after confirmation. No red lane, nothing, while directly along side a semi. *shakes head*
    • NavOnAP has no concept of "Keep Right, Pass Left". It never suggests lane changes back to the right in any of the available modes.
    • Further, it randomly suggests lane changes to the left for no reason whatsoever. No traffic, no interchanges, nothing.
    • I found the car randomly decelerating at least 10x during the trip with no obvious cause. More common when driving in the right lane vs left. It would also set a seemingly random max speed at times, with no speed limit changes or interchanges.
    • AP2 still doesn't read speed limit signs, so the noted speed limit doesn't always match the real highway speed limit in areas where it was recently upped or lowered (happens a lot around here with places bumping to 70).
    • At least once the car detected a construction zone with a popup about it (kudos on that) and then immediately proceeded to try and suggest a lane change into construction cones..... which negates this from making the "improvements" list above.
    • Overtake suggestions are useless. On two lanes, driving in the right lane, I would approach a vehicle ahead that was traveling more slowly. No other traffic. The car would decelerate... 5.... 10.... 15 MPH.... as it sees the vehicle. Then, after matching its speed at my set following distance, a few seconds later it'd popup "Confirm lane change" to overtake. Seriously, wtf. And not just once in a while. Every single time I waited for the suggested change, it behaved this way. In every mode setting, including "Mad Max".
      • The car detects the other vehicle way in advance, even when just using the in-car visualization for reference, and could easily make the suggested lane change early enough so that no deceleration at all would be needed, even with the delay of requiring confirmation.
    • On multiple occasions the car would start doing a lane change (either a confirmed one, a manually initiated one, or an automatic one for an exit), get part way through, and quickly veer back into the starting lane for no reason. About half of those times it would popup with "Lane change cancelled". In one instance I actually missed an exit because it was 2/3's into the exit ramp lane, stayed there a moment, then just jumped back to the left for no reason.... ugh.
    • Even features that were usable before, like manually initiated auto lane changes, are no longer reliable.

    Overall, using "Navigate on Autopilot" did not improve the experience of using Autopilot at all, with the limited exception of autosteer's new ability to mostly keep in lane on a tight interchange... with that being negated by the fact that it tries to kill you any time a lane ends. Also, it seems that the ability to take tight interchanges is mostly thanks to nav fusion, as the vision model does not appear to be properly detecting lanes in some of these situations, yet the system presses onward.

    The suggested lane changes were completely useless on every mode. It would either suggest changes that weren't necessary, weren't safe, or weren't useful. It was even suggesting lane changes for an interchange upwards of 8 miles away at one point, then refusing to suggest overtake lane changes until after that interchange.

    Some more notes:
    • Vehicle detection to the sides and behind your vehicle is complete garbage.
      • This is super obvious when sitting still with other still vehicles all around. You'll seem them "swimming" around the visualization, colliding with each other, with you, etc.
      • Also obvious when overtaking large vehicles. Almost every single semi truck, bus, or RV I passed ended up with a twin ghost visual on the screen.
      • Finally, vehicles to the side are regularly shown overlapping my own vehicle visual, despite them being firmly in their own lane.
      • Vehicles behind your vehicle are actually detected only part of the time, apparently due to some issue with the rear cam setup in the hardware (@verygreen I believe has documented this).
    • It seems very obvious that Tesla has no real data fusion whatsoever between the cameras. This results in both huge gaps in the usable data as well duplicate data (like the ghost trucks). This is like computer vision 101 stuff that I don't understand why Tesla hasn't overcome this, especially in something shipped to thousands of customers.
    • Radar/vision fusion on AP2 appears to be significantly worse than AP1, with AP1 easily accurate for a few cm... AP2 easily worse than +/- 1m... very obvious when looking at the lead vehicle visualization.
    • Some of the failings of NavOnAP don't even make sense. If it clearly "sees" a vehicle, it seems like a basic sanity check in the higher level code would prevent it from suggesting a lane change into it.... but this isn't what happens.

    Could probably go on for quite a while, but suffice it to say I won't be using the feature any further... not at least until it's actually useful.

    It doesn't improve the experience of using autopilot for me one bit. In fact, it makes it even more frustrating. This is ignoring the super frequent nags that plague the more recent firmwares, too.

    I'll be sticking to my AP1 vehicles for longer trips from now on I think. In fact, I'm probably going to try and make time to make some videos/posts about AP1/AP2 modifications that are actually useful.

    For example, my modded AP1 vehicle would handle the situation I noted above (overtaking a vehicle) smoothly with zero deceleration. AP1 (and AP2) can detect a vehicle ahead of you over 100m away... no excuse for the behavior of NavOnAP.

    I'm just super disappointed in Tesla. Their spat with Mobileye has cost Tesla customers a huge amount of progress on the autopilot front. AP1 owners are completely screwed because they will get zero improvements. (Despite promises of ongoing improvements, AP1 hasn't had a single improvement in about two years). Meanwhile, AP1 is running on Mobileye hardware that was released nearly 5 years ago and still handles many situations better than AP2. And it's not like Mobileye has stopped. They're positioned to blow Tesla out of the water with their current hardware (EyeQ4), and off the face of the Earth with their upcoming hardware (EyeQ5). Had Tesla not screwed us all over in that regard, it's likely AP1 would still be improving and that AP2 would be running the next gen of Mobileye hardware with features well beyond what Tesla is capable of doing today. Again, just disappointing that they've decided to forsake early adopters yet again, and also give current adopters less value for their $ in the meantime.

    I'm sure people will come out in force to defend Tesla, say how great NavOnAP is, etc... and by all means, do what you must. I personally own both types of vehicles (AP1 and AP2) and drive both regularly... pretty simple to tell the deficiencies of AP2. You're not going to convince me that somehow my extensive first-hand experience is somehow flawed and that things are way better than I claim. lol.

    I find it hard to believe that as a one-man free-time dev crew I've been able to do better than Tesla's entire multi-million-dollar-funded AP dev team has been able to do in more than two years. I have maybe 40 hours of total work into my AP1 mods, and they've been more usable than NavOnAP for nearly two years. I just don't get it. I'm pretty good at what I do, but I can't believe I'm better-than-full-teams-with-millions-in-funding good.

    To that end, when/if I get the time, I'm going to set a goal for myself of ~25 hours of work to make a hardware/software modification to an AP2 vehicle that actually does what NavOnAP is supposed to do. Start to finish, from scratch. It's been suggested that I get some basics prepped (hardware I need in-hand, for example), then keep a GoPro on a chesty running until the modifications are done and working to document the entire thing.

    If doable, then I think it would be even more obvious that we have to reevaluate Tesla's progress on the driver assistance front.

    Anyway, enough of that for now.

    Disclaimer: I have no positions with any of the companies mentioned nor do I intend to initiate one at any point in the future.
     
    • x 67
    • x 25
    • x 20
    • x 10
    • x 7
    • x 1
  2. nnick01

    nnick01 Member

    Joined:
    Feb 8, 2017
    Messages:
    113
    Location:
    Bay Area, CA
    I don't have many of these issues. Perhaps something is wrong with your car?
     
    • Funny x 11
    • Like x 9
    • Love x 1
  3. JohnSnowNW

    JohnSnowNW Active Member

    Joined:
    Feb 13, 2015
    Messages:
    2,209
    Location:
    Minnesota
    Hmm, works for me...but, sure, it's useless :rolleyes:
     
    • Like x 10
  4. dennis_d

    dennis_d Member

    Joined:
    Jan 22, 2017
    Messages:
    296
    Location:
    CA
    :popcorn:
     
    • Funny x 4
    • Like x 1
  5. jaguar36

    jaguar36 Active Member

    Joined:
    Apr 10, 2014
    Messages:
    1,916
    Location:
    NJ
    While I agree that it is useless in its current form, what I'm excited about is how close it is to being really useful. The hard work is almost all done. There just needs to be a bit more qork with some mroe testing and some more tweaks and you'll be able to really go on a multi-hour trip without any driver intervention at all (besides for the awful nags that is).
     
    • Like x 6
  6. Navin

    Navin Member

    Joined:
    Jun 14, 2016
    Messages:
    670
    Location:
    chicago
    Reading your twitter history. It’s obvious you are just being a troll as it relates to autopilot. It’s impossible for one person to constantly have issues with autopilot. If you don’t like the car or Tesla or autopilot- get another car
     
    • Disagree x 40
    • Like x 7
    • Funny x 5
    • Love x 1
  7. strangecosmos

    strangecosmos Non-Member

    Joined:
    May 10, 2017
    Messages:
    1,032
    Location:
    The Prime Material Plane
    Why not buy a vehicle with EyeQ4?

    Also, why not start your own company or interview for a job at GM, Tesla, Mobileye, Zoox, Uber, Ford, etc.? You could probably make millions if you can do what you say.
     
    • x 8
    • x 8
    • x 2
    • x 1
    • x 1
  8. wk057

    wk057 Senior Tinkerer

    Joined:
    Feb 23, 2014
    Messages:
    5,184
    Location:
    Hickory, NC, USA
    I realize I haven't been super active here on TMC lately, but I'd suggest to the people immediately jumping to conclusions that you check out some of my post history for the past ~5 years before.

    I love Tesla's products. They're the best of what they are, currently. Hell, I've purchased quite a few Tesla's and currently have... five?... on my My Tesla account (haven't bothered to update my signature in a while, either).

    So people start with the name calling and other hating, perhaps do minimal research.
     
    • x 46
    • x 26
    • x 2
    • x 2
    • x 1
  9. mtndrew1

    mtndrew1 Member

    Joined:
    May 12, 2015
    Messages:
    613
    Location:
    Gardena, CA
    Mine is also on the verge of useless, requiring a number of rather abrupt manual takeovers as it second-guesses itself. I'd imagine there were a few adjacent drivers who assumed I was drunk or distracted over the last week. It has never tried to steer me into any other cars or objects though.

    I've found that turning off the speed-based lane change suggestions improves NoA performance. The system never accepts that you're ignoring a suggested speed-based lane change so it slows the car while leaving a huge gap with traffic ahead, assuming that at some point you will accept its suggestion. Of course other cars pile in to the gap and the cycle continues.

    For me the only time NoA is useful is when I'm in a lane that is splitting between two freeways (it successfully signals and takes the correct path each time) or if I'm already in the correct exit lane and then it will signal, take the exit, and slow. Otherwise NoA is just too "polite" for LA traffic and always second guesses its own lane changes.

    One thing I noticed is that NoA never calibrated my cameras after I got the feature; it worked immediately after the software update. I wonder if my cameras should have calibrated and it's a bug? WK did your cameras calibrate?

    As an aside "regular" Autopilot has taken another leap forward. Auto lane changes (not on NoA) are actually quite useful now as opposed to being a novelty. Lane holding is rock solid and it feels very confident. I'm looking forward to seeing where NoA goes in the future but for now it's a novelty like auto lane change was six months ago.
     
    • Like x 5
    • Informative x 3
    • Helpful x 2
  10. kavyboy

    kavyboy Member

    Joined:
    Jan 13, 2016
    Messages:
    640
    Location:
    Spring, TX
    #10 kavyboy, Nov 4, 2018
    Last edited: Nov 4, 2018
    I don't think you know OP's "street creds" when it comes to Tesla. :) I wouldn't be surprised if he knows more about Tesla tech than anyone outside the company, and possibly even within the company.

    EDIT: Might want to check out the story where he added Autopilot to a non-AP car by himself.
     
    • x 26
    • x 1
    • x 1
    • x 1
    • x 1
  11. untilcomplete

    untilcomplete Member

    Joined:
    Apr 1, 2017
    Messages:
    169
    Location:
    East Coast
    Keep up the good work man! It’s important to annotate where Tesla is lacking and hope they fix it soon.
     
    • Like x 17
  12. mechrock

    mechrock Member

    Joined:
    Nov 11, 2017
    Messages:
    7
    Location:
    Hendersonville, NC
    I have to add that his thoughts are spot on with mine right now. Also too a 100 miles trip and had the same issues. I assume his was also in NC.

    “Just drove to Newton this morning and can’t believe they even considered releasing this with automatic lane changes! It asked to change lanes so many times when someone was in my blind spot I lost count. Most of the time, the road line wasn’t even red, a couple times it didn’t even see the car.

    Noticed it activated a lane change almost every entrance ramp, even if there were no cars coming in, only once did I want / need to accept.

    I just like to cruise in the right lane and many times even though I was going 5 over and not a single person in front of me, it would ask to change lanes...

    The only good thing I can say is it takes exit ramps really well. Turns the blinker on and takes it immediately and then slows to a stop when you get off the highway.”
     
    • Helpful x 1
    • Like x 1
  13. 22522

    22522 Active Member

    Joined:
    Jun 6, 2016
    Messages:
    1,465
    Location:
    Texas
    Questions about your hardware platform?
    Model 3 116XXX October build is better on the red line about merge...
     
    • Helpful x 1
  14. Bladerskb

    Bladerskb Senior Software Engineer

    Joined:
    Oct 24, 2016
    Messages:
    1,432
    Location:
    Michigan

    [​IMG]
     
    • Funny x 10
    • Disagree x 3
    • Like x 1
    • Love x 1
  15. wk057

    wk057 Senior Tinkerer

    Joined:
    Feb 23, 2014
    Messages:
    5,184
    Location:
    Hickory, NC, USA
    If it works for you, great.

    Honestly, I don't think any of these companies would work with me on this, including Tesla (who has offered me a position in the past, surprisingly, but I refused to move to California) due to my stance on not relocating. Making my own company for this.... meh, maybe.


    There was no new calibration shown, but the car was driven ~2k miles on 2018.39, so likely calibrated in the background at that point.


    I don't toot my own horn on this very often, but yeah... you are probably right.


    That's overall kind of my point with this. Tesla needs to get on the damn ball here and stop screwing around. Things could be so much better, but they keep dropping the ball.


    Yeah this trip was from Hickory to the Raleigh area and back.


    This particular drive was with a Model 3, although I have an AP2.5 S at my disposal that has the same issues. While I hadn't taken that one on a long trip to truly confirm everything behaves identically to the 3, I have enough miles behind the wheel to be pretty confident asserting that it's not any different.
     
    • Like x 3
    • Informative x 1
  16. pilotSteve

    pilotSteve Active Member

    Joined:
    Jul 14, 2012
    Messages:
    1,128
    Location:
    Vancouver WA
    @wk057, you have solid gold cred with those of us that have read your super informative posts and articles over the years. You not only talk (and share) your insights, you actually implement almost magical improvements to Tesla HW FW and SW that advance the benchmark for Tesla and the autonomous vehicle industry as a whole.

    IMO you are this forum (and quite possibly Tesla’s) best catalyst to someone actually delvering the vehicle capabilities we would love.
     
    • Like x 30
    • Love x 2
    • Helpful x 1
    • Funny x 1
  17. djgarrett21

    djgarrett21 Member

    Joined:
    Jul 22, 2018
    Messages:
    144
    Location:
    Utah
    Great post and my experience as well.

    NoA shouldn't have made it past the Early Access Program, probably shouldn't have even made it that far to be honest. It takes more work and is more stressful than driving on AP or simply controlling the vehicle myself.
     
    • Like x 5
    • Informative x 2
    • Helpful x 1
  18. jeffro01

    jeffro01 Active Member

    Joined:
    Jan 30, 2013
    Messages:
    2,485
    Location:
    SF Bay Area
    Anyone doubting @wk057 has no idea what they are talking about... His reputation speaks for itself.

    I find NoA not only completely useless, it actually requires so much attentiveness to keep it from doing something stupid that it's more work than just regular AP... I've stopped using it all together and I keep saying that if this is a representation of where Tesla is with AP development then we're so far away from Tesla FSD that it's not even worth having the conversation which I suspect is why Tesla removed it from purchase.

    My biggest complaint is the fact the car has no idea what the speed limit actually should be, has no feel for traffic speed, and thus injects stupid and extremely dangerous autosteer speed restrictions at random times, especially on on/off/interchange ramps. Nothing like going 75 and then hitting the exit ramp and then the car deciding you can't go faster than 35 and rapidly decelerating the car for no reason which is going to get someone rear ended like I almost was multiple times. Autosteer speed restrictions should be removed entirely, they're totally stupid.

    While I don't necessarily share the same opinion when it comes to MobileEye as @wk057 does, I get his frustration.

    Jeff
     
    • Like x 18
    • Informative x 2
    • Helpful x 1
  19. wk057

    wk057 Senior Tinkerer

    Joined:
    Feb 23, 2014
    Messages:
    5,184
    Location:
    Hickory, NC, USA
    Honestly, that's kind of my hope if I'm able to get around to doing my project on an AP2 car. If me doing some amazing stuff with an AP2 car with modded hardware/software ends up being a bit of a spur to Tesla's AP team's rear...... well, then success! My hope is that afterward someone will walk into that meeting and be like, "OK, watch this, then tell me wtf have you guys been doing the past two years? (Presses play on Youtube video)"
     
    • Like x 12
    • Love x 12
    • Funny x 3
  20. skitown

    skitown Member

    Joined:
    Dec 28, 2015
    Messages:
    849
    Location:
    Central Idaho, U.S.A, Earth
    Interesting - thanks for the thoughts @wk057. I live in a rural area and haven't been able to test NoAP given our roads. That said, we also own both AP2.5 and AP1 Teslas (Model 3 and Model S). My experience around here is that the AP2.5 car is far superior to our AP1 car with regards to general Autopilot performance. I wish it read traffic speed signs like our AP1 Model S, but beyond that it's proven to be a much better performer.
     
    • Like x 4

Share This Page

  • About Us

    Formed in 2006, Tesla Motors Club (TMC) was the first independent online Tesla community. Today it remains the largest and most dynamic community of Tesla enthusiasts. Learn more.
  • Do you value your experience at TMC? Consider becoming a Supporting Member of Tesla Motors Club. As a thank you for your contribution, you'll get nearly no ads in the Community and Groups sections. Additional perks are available depending on the level of contribution. Please visit the Account Upgrades page for more details.


    SUPPORT TMC