Navigate On Autopilot: automatic lane change results

[Didgie]

Collaborating with Enginerd to introduce some data into my very subjective videos. Big thanks Enginerd.

 

[Didgie]

This update logs the first 2 drives with the 2019.36.2.1 update. While there are some nice new enhancements, drive 48 included a handful of disappointing performances with basic maneuvers.

• Drive 47 had no driver aborts, or as Elon says, at no time did the driver have to intervene. Unfortunately, I had to intervene 3 times on drive 48 (described below).
• Wide berth for semi trucks: now works with two lanes in same direction (e.g. #1 & #2 lane). Previously on 2019.32.11.1 this feature was only enabled on highways with 3 lanes or more. This feature also now applies to some smaller vehicles like utility vans and big pickups.
• I'm fairly certain the same feature also positions at the far side of the lane when passing a stopped vehicle on the shoulder. It's subtle, but I'm fairly certain I saw it several times. Not sure this has been reported elsewhere.
• Strange event where NOA signalled and began to maneuver for an unnecessary exit. NOA aborted the maneuver and returned to the #2 lane. Quickly, NOA signalled left and moved into the #1 lane for no apparent good reason... like I wouldn't notice the lane change mistake if it looked like it was doing something on purpose? Then NOA switched back to the #2 lane. I logged this cluster as 1 AP abort and 2 successful lane changes, even though all 3 maneuvers were unnecessary.
• Drive 48, driver abort #1: NOA had taken the #1 lane (of 2) due to slower traffic entering the highway. When these recent entrants had sorted out their speeds, NOA wanted to switch out of passing lane #1, although a truck was gaining in lane #2. While it was shaping up to be a borderline jerk move, I let NOA do it. However, NOA got all swervy, and after the second swerve towards lane #2, I took over to preserve what was left of my dignity.
• Drive 48, driver abort #2: Entering I-20E from SC-340. I've had mixed results at this location in the past. Recently NOA had seemed to be performing better. After the merging zone, the onramp vanishes to a point. If the car finds itself at the end of the vanishing lane, sometimes it just jumps over without signaling. When the lane runs out after signaling, the car seems to want to jump 2 lanes. When it was clear that we were overshooting into the #1 lane, I aborted. It seems that the car can successfully initiate the real lane change while less than about 50% over the lane line. Initiating the real lane change when already more than 50% over the lane line results in overshooting. [Video doesn't really illustrate the event, which fell between file splices.]
  
• Drive 48, driver abort #3: Passing a big rig (only a few mph slower), NOA did not properly detect a Camaro approaching from the rear at maybe +10 mph. While I had hoped NOA would detect it at the last second and show a red lane line, it did not. Had I not aborted, I think there would have been brake lights, tire smoke, perhaps a horn, and most certainly an extended digit.

View attachment 478199

Weighted score:
View attachment 478200

Driver intervention required?
View attachment 478201

So far with 2019.36.2.1:
View attachment 478203

Performance not looking good for 36.2.x

 

[Fernand]

Personally, I think the personality of 36.2.x is very different from the last V9 releases. Much more "human", more assertive, and thus more prone to some tentative moves. The numbers might not convey that, but I think it's a better driver in many subtle ways, and I'm counting on a little time to zero it in. By the next wave of releases it should be more polished.
.

 

[mikes_fsd]

Performance not looking good for 36.2.x

I single drive with problems and a single drive without any problems is not enough to get a trend.

 

[Didgie]

I single drive with problems and a single drive without any problems is not enough to get a trend.

Agreed, let's see how it improves over more drives.

 

[strangecosmos2]

Thank you for doing this rigorous data collection! It helps to have a quantitative sense of how Navigate on Autopilot is doing. The improvements in V10 are encouraging.

 

[Enginerd]

single drive with problems and a single drive without any problems is not enough to get a trend.

I confess that I do keep my fingers crossed with each new software load... that this is the one where everything just works. Rest assured that I will always let the numbers speak for themselves, regardless of the story they tell. It serves us well to remember that the Concorde had the best safety record of any airliner (zero accidents per flight hour)... until the first accident. At that point, it became statistically the least safe (because of such a small fleet and limited flight hours), and ultimately led to the retirement of the whole fleet.

 

[Enginerd]

Just one drive on the books this week, while using the ICE minivan to haul furniture. I'm so not used to turning on the wipers myself, manual headlights, manual steering, manipulating dumb cruise control to accommodate traffic. Missing the torque. Spending $35 on gas instead of $5 on electricity does not bring me joy.

• Another disappointing score this week. I've been thinking about how the change of seasons and "falling back" have put a larger portion of my drive in darkness. While I don't have a sense that NOA/AP performance is worse at night, it's certainly a possibility. That said, the poor scores recently are mainly in daylight. Most of the negative marks seem to collect on I-95 in the noon to 3pm timeframe, with good lighting. Drive 49 includes 4 marginal lane changes and 3 AP aborts (of 77 total attempts).
• Drive 49: I-95N @ South Carolina exit 22. Last week NOA performed a strange unnecessary exit at this location. This week at the same spot, I think I felt the car do a tiny jerk right then left, then the red hands AP disengage. Why? If anyone is a whiz with OpenStreetMaps, let me know what you think. I took a quick look, and while I didn't see anything obvious, I'm not knowledgeable about road mapping data.
• Drive 49: Driver abort when NOA wanted to change lanes ahead of a rapidly advancing semi. (Sorry, no video.)

Full data:

Weighted score:

So far with the current mainstream firmware:

 

[Fernand]

I've had an atypical (relative to previous versions) miss recently. Car didn't lane change in time (for no clear reason), and being in rightmost lane was heading onto an unwanted exit. Maybe Nicki wanted to explore. Not a hazardous situation had it exited, as far as I know, but it wasn't planned.

A few months ago, I forget the version, I was on HW880 approaching the Tesla factory in Fremont, and Nicki spontaneously moved out of the fast lanes, over to the right, as if getting ready to exit there! My wife and i were totally cracking up. Then the car apparently realized it wasn't going to get to visit with its makers and buddies there. It got back into the fast lanes and resumed normal driving ;-)
.

 

[Falkirk]

I've noticed on the same typical local roads that the car just won't change lanes much more often now. It's now a making me think to myself each time I signal...will it actually decide to change lanes and if it does in a reasonable amount of time? Its been pretty embarrassing having the car signal for 10-20 secs and then nothing just turns off. Lately I'm giving it 5 seconds if the family is in the car and just take over.

At least I haven't had a phantom break incident in months! Knocking on a lot of wood right now.

 

[DopeGhoti]

Sounds like the phantoms are taking a break.

 

[Fernand]

Got 2019.36.12.3, just in time for Thanksgiving. I do, I do.

 

[Enginerd]

Drive #50 included heavy post-Thanksgiving, hey-everyone-let's-go-back-to-Florida-on-Sunday-afternoon traffic. Performance deficiencies this time seemed focused in the same type of maneuver.

• On-ramp 1: On-ramp merged into lane #2, however there was a car in lane #1. In my defensive posture against lane overshoots I've experienced in this location, I may have wrested steering control before NOA/AP had a chance to demonstrate the quality of its lane change. I called this one a "driver decline".
• On-ramp 2: On-ramp merged into lane #2. With no conflicting traffic, I decided to see what it would do. Turns out, maybe the worst possible thing? It made one continuous double-lane change across lane #2 all the way into lane #1. Yikes. My closest category: "marginal".
• On-ramp 3: On-ramp merged into lane #2, again with nearby traffic in lane #1. I definitely felt the steering overshooting towards/into lane #1. I took control without crossing the line. This was the only "driver abort" for drive #50.

While I still have 2019.36.2.1, I've heard comments and seen videos about lane change improvements in 2019.40.1.1. I look forward to testing, and anticipate better results, but I'm not sure if it will be for the right reasons. In all 3 on-ramp errors listed above, the car waited the full 5-6 signal blinks before initiating steering. This delay caused the steering input to occur late, in the narrowing lane, after the left wheels had already crossed into lane #2. In 2019.40.1.1, it looks like the signal delay is shortened to 3-4 blinks. This would probably make all 3 on-ramp scenarios successful. But if the merging lane was shorter than 3-4 blinks, I'm guessing we might see the same overshoot behavior in 2019.40.1.1. I'd prefer the NN team would work on recognizing the end of the merging lane and executing the lane change without overshoot.

Weighted score:

Cumulative results with this firmware:

 

[Enginerd]

My first drives with FW 2019.36.2.4 were not very impressive. Drive 51 was the worst score I've logged since June, and drive 52 was not any better. I fear (and the data shows) that the NOA success rate may have peaked back with FW 2019.32.12.2. I'm wondering whether it's possible for the neural network to unlearn things as they're adding new features.

• Drive 51: I-95 narrows from 3 lanes to 2 as it crosses the bridge from Georgia to South Carolina. Lane #3 ends as a wide, combined double lane for a couple hundred yards. Typically the car just centers itself in this wide lane, which is weird but not unsafe. However, on drive 51, it centered itself, then began changing from lane #2 to lane #1 to pass. AutoSteer aborted the lane change and disconnected AS with a red hands warning, leaving the car pointing at the shoulder. This was a disappointing unforced error, and I classified it as an "AP abort".
• Drive 51: While slowly passing a slow semi, a fast semi approached from the rear and impatiently tailgated. When my pass was finally complete, NOA initated the merge back to the right. As soon as the car started to move out of the way, the semi behind me began accelerating to pass. About 30% of the way into the lane change, AP swerved back towards lane #1. I yanked the wheel right to complete the lane change. This was especially disappointing because NOA aborted a safe lane change to put the car in the path of a much larger overtaking vehicle. I classified this debacle as both an "AP abort" and a "driver abort".
• Drive 52 at night: Speed matching while passing a semi. Upon reaching the cab of the truck, the car slowed to match the semi's speed. With other cars behind me wanting to pass, and the truck not steering very accurately within his lane, this is a place where I didn't want to hang out. Similar to previously observed "speed matching" behavior.
• Drive 52 at night: A cluster of 4 events where NOA signaled a lane change into an empty lane, aborted at 0% to 30% completion. This happened 3 times in a row before successful lane change completion. This cluster counted for 3 "AP abort" events, which really dragged down the score.

Full results:

Weighted score:

So far with 2019.36.2.4:

 

[Didgie]

My first drives with FW 2019.36.2.4 were not very impressive. Drive 51 was the worst score I've logged since June, and drive 52 was not any better. I fear (and the data shows) that the NOA success rate may have peaked back with FW 2019.32.12.2. I'm wondering whether it's possible for the neural network to unlearn things as they're adding new features.

• Drive 51: I-95 narrows from 3 lanes to 2 as it crosses the bridge from Georgia to South Carolina. Lane #3 ends as a wide, combined double lane for a couple hundred yards. Typically the car just centers itself in this wide lane, which is weird but not unsafe. However, on drive 51, it centered itself, then began changing from lane #2 to lane #1 to pass. AutoSteer aborted the lane change and disconnected AS with a red hands warning, leaving the car pointing at the shoulder. This was a disappointing unforced error, and I classified it as an "AP abort".
• Drive 51: While slowly passing a slow semi, a fast semi approached from the rear and impatiently tailgated. When my pass was finally complete, NOA initated the merge back to the right. As soon as the car started to move out of the way, the semi behind me began accelerating to pass. About 30% of the way into the lane change, AP swerved back towards lane #1. I yanked the wheel right to complete the lane change. This was especially disappointing because NOA aborted a safe lane change to put the car in the path of a much larger overtaking vehicle. I classified this debacle as both an "AP abort" and a "driver abort".
• Drive 52 at night: Speed matching while passing a semi. Upon reaching the cab of the truck, the car slowed to match the semi's speed. With other cars behind me wanting to pass, and the truck not steering very accurately within his lane, this is a place where I didn't want to hang out. Similar to previously observed "speed matching" behavior.
• Drive 52 at night: A cluster of 4 events where NOA signaled a lane change into an empty lane, aborted at 0% to 30% completion. This happened 3 times in a row before successful lane change completion. This cluster counted for 3 "AP abort" events, which really dragged down the score.

Full results:
View attachment 486623

Weighted score:
View attachment 486624

So far with 2019.36.2.4:
View attachment 486625

I have also noticed a change in performance, speed matching occurrences have increased, over running the lines and over correction during on/off ramps is significant and a general increase in issues leading to confidence reduction is noticeable, so to see it backed up in your data is telling a declining story.

 

[Didgie]

Quick video on 36.2.4 using our Enginerd’s data, then almost immediately after posting it I received 40.2.1 (typical), so I’ll be out testing tomorrow.

 

[Didgie]

40.2.1 is looking good so far, less fails and more confident lane changes. Looking forward to Enginerd's test results.\

 

[Enginerd]

My first drive with FW 2019.40.2.1 was almost entirely in light to moderate rain. While this is certainly a variation on my normal test conditions, it is representative of real world conditions.

• Due to rain, NOA was unavailable for a substantial part of the drive: the number of NOA lane change events was only 59% of the average to date. To state the obvious, rain is not an abnormal condition, and an FSD car must be able to handle it.
  • If the problem is software for image processing, Tesla may work on this in the future.
    
  • If the problem is purely optical, i.e. can't digest images from rear-facing cameras due to water/mist/road spray, then there is a fundamental hardware problem with all Tesla cars.
• Failure to exit passing lane was a big problem with drive 53 in the rain. Interestingly, the car:
  • would not change lanes to the right to exit the passing lane
  • was able to change lanes to the right when traffic in that lane was moving faster
  • was able to change lanes to the right when that lane was required to navigate an interchange.
• After about 5 of failures to exit the passing lane, I exited the highway, shut the car down, and started fresh, with no apparent effect.
• Late in the drive, after several failures to exit the passing lane, a warning message posted periodically: "Right front fender camera blocked or blinded. Clean camera or wait for it to regain visibility." Interestingly, while this message posted a couple times, NOA remained engaged, although it did not make any lane changes to the right in this condition. Based on the photo below, NOA could not see the far lane line to the right. Perhaps this may have been a problem earlier in the drive, resulting in degraded performance, but not quite reaching the threshold for the warning message? I clean my car weekly, so the only schmutz on the camera was due to driving in the rain that day. The camera appeared clean upon reaching my destination.

• Speed matching: this time, it latched on to the rear of a semi trailer instead of latching onto the cab of the truck. The car was driving at about 15 mph below the target speed until I manually accelerated past the truck.
• Ironically, while the drive 53 score was the worst I've recorded, no driver abort was required for safety.
• Unrelated to NOA:
  • Driver-initiated lane changes are much more assertive with reduced delay between signal and steering. Unfortunately, this improvement was not apparently applied to lane changes initiated by NOA (yet?).
  • Ping pong bobbing and weaving is much worse than previous FW.
  • The Deep Rain neural network wipers worked well. In 330 miles of light to moderate rain, I never had to manually wipe or take the wipers off AUTO mode.
  • Adjacent Lane Speed activated a few times, and it seemed entirely appropriate… a welcome human-like change.

Full results:

Weighted score:

So far with 2019.40.2.1:

 

[navguy12]

My first drive with FW 2019.40.2.1 was almost entirely in light to moderate rain. While this is certainly a variation on my normal test conditions, it is representative of real world conditions.

• Due to rain, NOA was unavailable for a substantial part of the drive: the number of NOA lane change events was only 59% of the average to date. To state the obvious, rain is not an abnormal condition, and an FSD car must be able to handle it.
  • If the problem is software for image processing, Tesla may work on this in the future.
    
  • If the problem is purely optical, i.e. can't digest images from rear-facing cameras due to water/mist/road spray, then there is a fundamental hardware problem with all Tesla cars.
• Failure to exit passing lane was a big problem with drive 53 in the rain. Interestingly, the car:
  • would not change lanes to the right to exit the passing lane
  • was able to change lanes to the right when traffic in that lane was moving faster
  • was able to change lanes to the right when that lane was required to navigate an interchange.
• After about 5 of failures to exit the passing lane, I exited the highway, shut the car down, and started fresh, with no apparent effect.
• Late in the drive, after several failures to exit the passing lane, a warning message posted periodically: "Right front fender camera blocked or blinded. Clean camera or wait for it to regain visibility." Interestingly, while this message posted a couple times, NOA remained engaged, although it did not make any lane changes to the right in this condition. Based on the photo below, NOA could not see the far lane line to the right. Perhaps this may have been a problem earlier in the drive, resulting in degraded performance, but not quite reaching the threshold for the warning message? I clean my car weekly, so the only schmutz on the camera was due to driving in the rain that day. The camera appeared clean upon reaching my destination.

View attachment 488083​

• Speed matching: this time, it latched on to the rear of a semi trailer instead of latching onto the cab of the truck. The car was driving at about 15 mph below the target speed until I manually accelerated past the truck.
• Ironically, while the drive 53 score was the worst I've recorded, no driver abort was required for safety.
• Unrelated to NOA:
  • Driver-initiated lane changes are much more assertive with reduced delay between signal and steering. Unfortunately, this improvement was not apparently applied to lane changes initiated by NOA (yet?).
  • Ping pong bobbing and weaving is much worse than previous FW.
  • The Deep Rain neural network wipers worked well. In 330 miles of light to moderate rain, I never had to manually wipe or take the wipers off AUTO mode.
  • Adjacent Lane Speed activated a few times, and it seemed entirely appropriate… a welcome human-like change.

Full results:
View attachment 488079

Weighted score:
View attachment 488080

So far with 2019.40.2.1:
View attachment 488081

Great data!

I'm glad I'm not the only one experiencing the ping-ponging that has returned.

As for assertive nature of the lane changes, I had to change from Mad Max mode to the average mode to calm the amount of lateral g in the lane changes for passenger comfort.

IMO, the lane change assertiveness should be situational.

If one is weaving in heavy traffic, then by all means one can be much more aggressive in lane changes to keep with the flow.

But on a lonely stretch of freeway, when about to pass a single slower moving vehicle, the lane change should be less aggressive.

Also, IMO, if the road surface has any amount of slickness to it (slush, ice, etc) the current assertive lane change behavior could be problematic.

 

[Enginerd]

As for assertive nature of the lane changes, I had to change from Mad Max mode to the average mode to calm the amount of lateral g in the lane changes for passenger comfort.

I'm pretty sure that the settings (mild / average / mad max) only have to do with the decision to change lanes, not the character of the maneuver itself. I think the metric is how much below your target speed NOA will tolerate, perhaps with a duration component as well. Once the logic determines that a lane change should be made, I think the steering command is likely the same in any mode. Most of my experience is in Mad Max, so if I'm wrong about this, let me know. I do agree, though, that the aggressiveness of the lateral acceleration should at least consider road conditions (wet, ice), and perhaps leisurely vs. immediate need.