Navigate On Autopilot: automatic lane change results

[Enginerd]

Everybody has anecdotal experience with automated lane changes using Navigate on Autopilot (NOA). We see comments such as "works for me every time", "kinda sketchy", or "merged into the fast lane without considering the guy coming at 100 mph". All of these observations are true, but don't offer a statistical way to measure quality or success rate. So I decided to take on that task.

I have a unique "commute" covering 330 miles each way, weekly, with about 263 miles on highways enabled for NOA. Most of the route is similar to I-95 in South Carolina: wide center median separating opposing traffic, 2 lanes of traffic in each direction. Given that traffic conditions will never be identical week-to-week, and each traffic interaction is potentially unique, using the same route on the same day of the week at similar times should mitigate outliers and trend towards results that are as consistent as possible.

My parameters include a 72 mph target speed (posted limit 70 mph). NOA speed based lane change setting: Mad Max (Disabled, Mild, Average, Mad Max).

Scoring system:

• Success (+1 point): Successful lane change, no negative interaction with other motorists
• Uncomfortable success (+0.5 points): Successful lane change, but negative interaction with other motorists
• AP abort (-0.5 points): Autopilot initiated but aborted lane change (did not complete)
• Driver decline (-0.75 points): NOA indicated intent to change lanes, driver declined on screeen in order to avoid: (1) negative interaction with other motorists, or (2) a poor tactical choice
• Driver abort (-1 point): Driver aborted an autopilot-initiated lane change in progress, either for safety or to avoid a negative interaction

Perfect average score for a drive would be +1.0, while a failed lane change on each attempt would yield an average score of -1.0.

Today with 2019.8.5 3aaa23d, NOA initiated 78 lane changes, with 0.647 average points per attempt (detailed results below). I plan to track this trend over time, and hopefully identify discrete improvements with relevant future firmware updates. I probably won't consider FSD until I see Tesla starting to march the 9's on this scale.

 

[gilscales]

80% success is better than I thought it would be based on my FAR smaller sample of about 5 scary times, won't use it until a few more updates I suspect, thanks to you and the bold few who provide Tesla the data though!

 

[BillO]

Great idea, but isn’t the amount of traffic going to influence the results heavily? That has been my experience—as the amount of traffic increases so does the number of mistakes. Can you factor that in somehow?

 

[Enginerd]

Today with 2019.12.1.2, NOA initiated 73 lane changes, with 0.815 average points per attempt (detailed results below).

As this was only my second drive logging the results, I think any difference was probably within the scatter of repeatability. It will take longer to establish trends and scatter bands.

Overall score:

@BillO , you're right, success may correlate with overall traffic level. All I can do is attempt repeatability as best I can. So far 78 vs. 73 attempts indicates some level of consistency. I'll track # of lane change attempts and keep an eye on the effect. What I'm really hoping for is that NOA will stop doing dumb stuff, like waiting too long as an opportunity expires and then cutting someone off.

One maneuver I noticed that this method doesn't capture the "on-ramp" half of "on-ramp to off-ramp" functionality. After engaging NOA on a freeway on-ramp, when it "merges" onto the main highway, sometimes it merges nicely, and sometimes it just kind of fumbles itself into the first lane. It either runs out of merging lane and slaloms over onto the highway, or signals to merge too late and nearly swerves into the second lane. I'm excluding this behavior for now, but I'll keep an eye on it as something they'll have to fix.

 

[Wodensthrall]

One maneuver I noticed that this method doesn't capture the "on-ramp" half of "on-ramp to off-ramp" functionality. After engaging NOA on a freeway on-ramp, when it "merges" onto the main highway, sometimes it merges nicely, and sometimes it just kind of fumbles itself into the first lane. It either runs out of merging lane and slaloms over onto the highway, or signals to merge too late and nearly swerves into the second lane. I'm excluding this behavior for now, but I'll keep an eye on it as something they'll have to fix.[/QUOTE]

This has been one of my frustrations -- NOA seems to think the shoulder is my lane when I'm trying to merge onto the freeway. I haven't been willing to wait long enough to see what it will ultimately decide -- I take control in order to not appear drunk or chaotic. I have pretty much resigned myself to not starting autopilot until after I have merged.

 

[Bet TSLA]

I'm very much looking forward to seeing more data.

 

[Enginerd]

I've made a couple of small adjustments to how I categorize the results:

1. I'm trying to avoid interrupting NOA attempting to do its job. I try to allow NOA to show me what it would really do after indicating intent to change lanes when I would not have done so. In most cases, I've learned that it will signal early while painting a red lane line and waiting, then it successfully completes the lane change when clear. This moves more of the "driver decline" results into either "success" or "driver abort" (if unsafe or rude).
2. I added a category for "no offer", where NOA failed to offer a lane change to get out of fast lane or take an exit. Categories and points are now:

Today's results:

Commentary: The latest drive today had smallest number of lane change opportunities (70% of average). Although the overall score was lower than the previous 2 drives, performance was flawless for the first half of the drive. The second half of the drive included sunset and night conditions.

Anecdotes of individual events today:

• Marginal: On entrance ramp, as my entrance lane tapered to nothingness, at the last second AP turned hard into the slow lane, but overshot a couple feet into the second lane before settling in the slow lane.
• AP abort: My lane divided into two lanes, AP tracked towards right lane, then at the last second aborted and "selected" the left. Very non-human-like.
  
• Driver abort: Slow big rig merging from freeway on-ramp. AP did not alter speed to accommodate, and there was no opportunity for me to change lanes to the left to allow space. Heavy braking was required to prevent closing off the big rig's only opportunity to merge.

Historical tracking for the categorized results:

Historical tracking for the overall score:

 

[TJtv]

Interesting data. However it shows that the performance is absoutely horrific. Every single drive that you took would have resulted in an accident if you were letting the car truly do all the lane changes it wants to do. That's not a feature that should be released to the general public.

 

[Enginerd]

Every single drive that you took would have resulted in an accident if you were letting the car truly do all the lane changes it wants to do.

I agree in principle. However, I also don't want to sensationalize or exaggerate. My personal "driver abort" threshold is well below the level that would result in an accident. It also includes NOA/AP behavior that negatively interacts with other drivers... the sort of thing that would result in an extended finger, a horn, high beams, etc. The point of the thread is that the system isn't all good or all bad. It should only be trusted within its capabilities, and this is an attempt to quantify them.

 

[TJtv]

Consumer reports just posted a review of NOA:

Tesla's Navigate on Autopilot Requires Significant Driver Intervention

Their analysis is pretty similar to yours, although they present it quite a bit less rosy.

 

[Enginerd]

Well, my first 2 long drives with 2019.16.2 were... schizophrenic. The first was pretty poor, with a handful of swervy lane changes and missed opportunities to merge right before being passed on the right. It seemed 2019.16.2 was definitely not an improvement for NOA. Then the second drive was nearly flawless, with only 1 such missed opportunity. Granted, yesterday's traffic was the lightest yet (only 44 NOA events, 59% of average), which as previously noted, could indicate a correlation (see third plot below).

In other notes, I believe there is a new steering oscillation which is hard to pinpoint, with a period of maybe 3-4 seconds. Although it appeared several times, it didn't seem to persist more than a few cycles, and I pondered whether the highway grooves could be a contributing factor. I'll try to identify it with my phone's accelerometer on the next drive.

 

[Enginerd]

The 2 lane changes that I aborted on the most recent drive occurred on a 4 mile section of I-20 that isn't part of my normal route. The first was a lane overshoot when merging onto I-20. The second was an attempt to exit from I-20 E onto I-95 S. The car had already changed into the correct exit lane. Then after it was no longer an option, it attempted to get back on I-20 E.

Categorized results to date:

Weighted scores to date:

The chart below reflects cumulative results with 2019.16.2 to date.

Has anyone else noticed this feature? When I perform a manual lane change with the turn signal, the car won't offer any NOA automatic lane changes for about a minute. Most often this occurs when I leave the right lane to give a stalled truck more room. I'm guessing this is intentional, so that it doesn't seem like NOA is trying to undo lane changes recently executed by the driver.

 

[Enginerd]

Friday's drive (#9 on the chart) was uncharacteristically poor. I actually suspect perhaps something didn't get initialized properly at startup or something:

• would not close garage door (this was the first/only time)
• dashcam did not work until steering wheel button reboot during Supercharger stop
• seemed loathe to change lanes to the right (out of the fast/passing lane) [falls into the "no offer" category]

That said, I'm starting to feel like although there is some variation due to unique conditions on each drive, this data will serve as a good baseline to track progress in future software updates.

Overall NOA lane change success rate with firmware 2019.16.2 is 87% based on 409 events to date. Stated another way, given an unsatisfactory result in about 1 out of every 8 NOA lane changes, my wife would not trust it in it's current state. But I'm happy to witness the development first hand.

 

[lakemirror]

My parameters include a 72 mph target speed (posted limit 70 mph).

In Florida, that would get me zero opportunities to pass anybody.

 

[Enginerd]

In Florida, that would get me zero opportunities to pass anybody.

Fair. It's not for everyone. However, I'm still averaging about 62 lane change events per trip. Quick rationale:

1. When FSD becomes a thing, I'd be upset if it is programmed/allowed to exceed speed limits like humans do. The automation had better play it safe for obvious reasons. I'm training myself for that future.
2. I'm driving approximately 34,000 miles per year with my current living situation. Realizing that statistics are against me reaching my target 2.5 years in this situation without an incident, I'm doing what I can to skew the odds in my favor, especially at night. Autopilot plays a big part in this.

 

[Enginerd]

Well, 2019.20.1 doesn't appear substantially different. Performance on drive #11 was below average, but above average on #12... which is why we have averages.

• On drive #11, I had 2 instances of a NOA dynamic reboot, where I got the red hands and an AP/NOA unavailable message, which healed itself to full functionality within about 2 minutes.
• Although I've decided under normal circumstances not to "driver decline" an offered lane change, I did so twice on drive #11. One was to avoid making a big rig anxious regarding my intentions (blinking turn signal). The other was when NOA expressed an interest in changing from the stopped left lane into the right lane which was moving around 40 mph. Due to the high speed differential, it didn't seem like a good idea.
• Of the (only) 4 unsatisfactory events on drive #12, one was a lane overshoot entering the interstate from an on-ramp, and the other was missing the I-20E to I-95S interchange (again). On-ramps/off-ramps aren't a strong point yet, so this isn't a surprise.
• Although I'm not officially collecting data on AP allowing traffic from an on-ramp to merge without drama, I noted 4 impressive examples on drives 11 & 12... the kind where I take my hand off the wheel just long enough to briefly applaud for AP's courtesy and forethought. Subjectively this seems to be an area of improvement.

 

[Enginerd]

Well, no big surprises in 2019.20.2.1. NOA left me hanging out in the passing lane while faster cars passed on the right, a total of 9 times this weekend. It's an embarrassing faux pas (and I subsequently avoid eye contact with those drivers), but... it's for science.

Both "driver aborts" on drive 14 were at the same interchanges where NOA previously failed:

1. Entering I-20E from SC-340: attempted to merge into lane #1, overshot, and I aborted.
2. I-20E to I-95S: NOA wanted to continue on I-20E instead of taking I-95S exit.

Random data note: The northbound drives (chart: odd #'s) Fridays 12 noon - 6pm tend to have 35% more lane change events than southbound drives (even #'s) Sundays 3pm - 9pm.

 

[Didgie]

Love this data Enginerd. How are you pulling this together?

 

[animorph]

Has anyone else noticed this feature? When I perform a manual lane change with the turn signal, the car won't offer any NOA automatic lane changes for about a minute. Most often this occurs when I leave the right lane to give a stalled truck more room. I'm guessing this is intentional, so that it doesn't seem like NOA is trying to undo lane changes recently executed by the driver.

Yes, I have noticed that it will just follow a slower car soon after a manual lane change when I would have expected it to pass the slower car.

 

[Enginerd]

Love this data Enginerd. How are you pulling this together?

Old school. I tried keeping a tally on paper, but found it too distracting.