Navigate On Autopilot: automatic lane change results

[Fernand]

Ah, the bead game method.

Could try an abacus but the carries get confusing.
You a bona fide enginerd, Man. Coolio.

My only ever panic abort, on 12.1, was as much view into another
vehicle's suspension as I ever care to see.

 

[Enginerd]

Here's my second report on V10 NOA. There are some mild hints of progress, but again I would say there are no obvious substantial improvements in V10 decision-making. Notes:

• A new V10 behavior that doesn't properly fit in one of the existing categories. NOA activates the turn signal and then just sits there for 10-15 blinks until it cancels the lane change, never having initiating any steering into the adjacent open lane. Once on drive #37 this happened 3 times in succession before successfully completing a lane change.
• Drive #37 was the second drive to date with no major NOA infractions. That is: no driver aborts or AP aborts, and no uncomfortable/marginal lane changes... only failures to get out of the passing lane before being passed on the right. Drive #37 included 80 NOA lane change events, whereas the previous "no major infraction" drive on 2019.16.2 was in light traffic with only 44 NOA lane change events during the trip.
• Drive #38 included an AP red hands abort, where NOA initiated the lane change and steered to cross the lane line. Then the red hands displayed and AP completely disengaged. I didn't notice any abnormal conditions, so this one is a mystery.
• Wide berth for semi trucks: I witnessed this behavior on a different 3-lane route. There has been no evidence of this behavior on the 2-lane routes logged for this test.

Full results:

Weighted score:

Cumulative results with V10 to date:

 

[Fernand]

Good work, thank you!

 

[Routybouty]

Anxiously waiting 32.12.x results as for my situation, the success rate took a nose dive. Specifically around on/off ramps. I would go back to 32.11 if I could.

 

[Fernand]

Anxiously waiting 32.12.x results as for my situation, the success rate took a nose dive. Specifically around on/off ramps. I would go back to 32.11 if I could.

I too liked 32.11 better. With 12.1 and now 12.2 the whole NOA experience is different, that's always been the case, that it gets a different personality with updates. Granted, this one is more aggressive, more mature, more confident. But I'm not. It's made more WTF pointless lane changes I've aborted - that used to only happen once in a blue moon. It tends to hug edges more instead of centering. And it did one I don't really want to talk about. I'm going in Monday to have the machine checked. I don't like it, no Sir.
.

 

[Enginerd]

First update with 2019.32.12.2:

• 4 drives in a row under V10 with no driver-aborted lane changes. Longest prior streak before V10 was 2.
• Drive #39: 6 instances of "no offer" to exit the fast lane before being passed on the right. "No offer" was the only type of deficient auto lane change on this drive.
• Drive #40: 2 marginal/uncomforable lane changes, where NOA switched into fast lane with a faster car approaching from the rear, giving that car an uncomfortably short amount of time to adjust.
• Drive #40: highest weighted score to date (0.985). Total number of lane change events was 66 (96% of the average to date), indicating an approximately average level of traffic.

One hypothesis I have about the "no offer" events (failure to exit the fast lane) involves light traffic conditions. This particular condition probably doesn't exist for West Coast drivers. My theory is that one contributing component of the "exit fast lane" logic is another car approaching in the fast lane from behind. I think that when there's no approaching traffic, sometimes there's less motivation to exit the fast lane. While "no offer" also happens in heavier traffic at times, I wonder if this hole in the logic was an unintended byproduct of testing primarily in California. When NOA does get passed on the right, it clearly seems to recognize the behavioral error, as most of the time it then quickly exits the fast lane. So while it corrects its mistake, I'd guess that the neural net training does not penalize this behavior, which may be why it keeps happening.

One potential minor improvement observed in 2019.32.12.2:
Previously NOA would signal as soon as it found itself trapped behind a slower car, and then wait until an opening appeared. Sometimes it would wait for several cars to pass while signaling, using the turn signal more as a request than a statement. I've seen at least a couple of times with 2019.32.12.2 where NOA allowed itself to be trapped at up to 10 mph slower than the lead car, while waiting for the opening to appear before signaling. This is much more human (and less pathetic) behavior.

Anxiously waiting 32.12.x results as for my situation, the success rate took a nose dive. Specifically around on/off ramps.

One behavior seems a little more obvious with 2019.32.12.2 than it was previously. Frequently, NOA leaves the slow lane (AKA merging lane) when crossing another numbered highway. Presumably this would reduce the number of interactions with exiting and entering drivers. However, NOA now seems to return to the slow/merging lane before the on-ramp merge point, which kind of leaves one scratching their head about apparently unnecessary lane changes.

One final note. Is it possible that NOA now changes lanes away from a stopped emergency vehicle on the side of the road? When this happened today, the rationale for the lane change was slightly ambiguous. There was a slower truck in the distance, which also changed lanes to give the highway patrol some space. The displayed message was clearly the standard "changing lanes to pass slower traffic" message. However, the lane ahead was basically clear. Anyone else have an example of this?

Weighted score:

 

[Didgie]

Here's my second report on V10 NOA. There are some mild hints of progress, but again I would say there are no obvious substantial improvements in V10 decision-making. Notes:

• A new V10 behavior that doesn't properly fit in one of the existing categories. NOA activates the turn signal and then just sits there for 10-15 blinks until it cancels the lane change, never having initiating any steering into the adjacent open lane. Once on drive #37 this happened 3 times in succession before successfully completing a lane change.
• Drive #37 was the second drive to date with no major NOA infractions. That is: no driver aborts or AP aborts, and no uncomfortable/marginal lane changes... only failures to get out of the passing lane before being passed on the right. Drive #37 included 80 NOA lane change events, whereas the previous "no major infraction" drive on 2019.16.2 was in light traffic with only 44 NOA lane change events during the trip.
• Drive #38 included an AP red hands abort, where NOA initiated the lane change and steered to cross the lane line. Then the red hands displayed and AP completely disengaged. I didn't notice any abnormal conditions, so this one is a mystery.
• Wide berth for semi trucks: I witnessed this behavior on a different 3-lane route. There has been no evidence of this behavior on the 2-lane routes logged for this test.

Full results:

View attachment 465810

Weighted score:
View attachment 465814

Cumulative results with V10 to date:
View attachment 465812

Still loving your work

 

[Fernand]

Graphs give me headaches, but yours are very nice.

 

[Enginerd]

Additional drives are showing evidence of V10 improvement.

• Drive 41: Another instance of NOA initiating a lane change with no traffic, which times out before ever steering out of the lane. I believe this happened 3 times sequentially, before finally succeeding on the 4th attempt. It's possible this occurred in the same spot on I-95N near Coosawhatchie where this happened before, so I'll monitor that in the future.
• Drive 42: AP lane line detection became confused over lane lines with sun glare in the front camera. Other camera views (without glare) show lane line quality should have been adequate. This highway was repaved a few weeks ago, and it looks like the work trucks tracked something shiny onto the roadway.
  
  
  
• Drive 41 marked 5 drives in a row without a single driver abort!
• A driver abort on drive 42 was complicated, involving several cars. NOA passed on the left, but hung out in the passing lane a little too long. Car #1 behind started passing on the right, shortly before NOA signaled to the right and then waited to change lanes. Car #2 behind him waited patiently for me to exit the fast lane. Then car #3, less patient, performed a high speed pass on the right. NOA initiated the lane change, which I aborted to avoid conflict with the hot-head that we had pissed off.
• I inadvertently triggered the AP stop light detection warning. With no traffic at night, approaching an intersection with AP off at 30 mph, I saw the cross-traffic signal was yellow. I engaged AP, and as it accelerated toward 45 mph and toward the (still) red light, it posted the red hands warning with the red traffic light symbol. The traffic light turned green and there was no further drama. This was an unintentional test that I believe I conducted safely, confirming that the safety feature works.
• On a personal note, I think I've reached an evidence-based milestone. The past 7 drives with V10 have had only 2 driver aborts out of 536 events (0.37%). Prior to V10, the driver abort average was 2.03% (42 driver aborts out of 2071 events). This improvement indicates that while AP is not yet perfect, the Tesla team is committed to the future success of FSD, and is demonstrably advancing on that goal. The next time Elon threatens to raise the price on the $3k upgrade from Enhanced Autopilot to FSD, I'll be putting my money down. I will plan to continue this tracking effort as any new features become available.

Weighted score

 

[Enginerd]

The last 2 drives show solid NOA performance, although V10 has definitely shown some "interesting" artifacts.

• NOA in Mad Max mode seems to be more patient when following someone who is going slightly below the target speed. In two instances, I spent about 5 minutes behind a car going 2 mph slower than my target. Wanting to ensure that NOA wasn't paralyzed, I rolled the speed up +1 mph, which prompted the passing maneuver.
• I think I may have had an example of "speed matching" that others have described. I was passing a Ryder truck which was about 2 mph slower than me. When I got up in line with the cab, NOA signaled to get back in the slow lane (occupied by the truck). Then NOA slowed to match the truck's speed. We sat there for some time, in lock step with the side of the truck, blinking, and below the target speed. Initially I waited for NOA to sort itself out. When that didn't look likely, I disabled NOA on the screen, leaving AP engaged. The car accelerated and passed the truck. I re-engaged NOA, and things became normal again. The only thing I can think of is that cameras/sonar may have mistaken the front cab of the truck for the back of another truck... waiting for the space to open up enough to allow the lane change. Odd. For scoring purposes, I put this one in the "driver decline" bucket on drive #44.
• At the I-95 lane split (1 lane turns into 2) where indecision was observed previously (FW 2019.28.3.1, post #34), in nearly identical night conditions, V10 handled the lane split exactly as a human would. No unnecessary drama. No NOAUI (Navigate On Autopilot Under the Influence). Progress.
• I had an autopilot lockout of unknown origin ("cruise control unavailable for the rest of the drive" or something). I hadn't ignored warnings, and there was no rain/debris/bugs obscuring any sensors. Not wanting to drive the remaining 250 miles in that state, I did a dual thumbwheel reboot, which seemed to restore full functionality.
• V10 seems fairly prone to signal and then cancel without ever maneuvering for the lane change. If I had a category for this, which I don't, it would be 6.7% in the last 5 drives. Thus this new V10 artifact is easily the most common type of less-than-satisfactory result.

Weighted score:

 

[Didgie]

The last 2 drives show solid NOA performance, although V10 has definitely shown some "interesting" artifacts.

• NOA in Mad Max mode seems to be more patient when following someone who is going slightly below the target speed. In two instances, I spent about 5 minutes behind a car going 2 mph slower than my target. Wanting to ensure that NOA wasn't paralyzed, I rolled the speed up +1 mph, which prompted the passing maneuver.
• I think I may have had an example of "speed matching" that others have described. I was passing a Ryder truck which was about 2 mph slower than me. When I got up in line with the cab, NOA signaled to get back in the slow lane (occupied by the truck). Then NOA slowed to match the truck's speed. We sat there for some time, in lock step with the side of the truck, blinking, and below the target speed. Initially I waited for NOA to sort itself out. When that didn't look likely, I disabled NOA on the screen, leaving AP engaged. The car accelerated and passed the truck. I re-engaged NOA, and things became normal again. The only thing I can think of is that cameras/sonar may have mistaken the front cab of the truck for the back of another truck... waiting for the space to open up enough to allow the lane change. Odd. For scoring purposes, I put this one in the "driver decline" bucket on drive #44.
• At the I-95 lane split (1 lane turns into 2) where indecision was observed previously (FW 2019.28.3.1, post #34), in nearly identical night conditions, V10 handled the lane split exactly as a human would. No unnecessary drama. No NOAUI (Navigate On Autopilot Under the Influence). Progress.
• I had an autopilot lockout of unknown origin ("cruise control unavailable for the rest of the drive" or something). I hadn't ignored warnings, and there was no rain/debris/bugs obscuring any sensors. Not wanting to drive the remaining 250 miles in that state, I did a dual thumbwheel reboot, which seemed to restore full functionality.
• V10 seems fairly prone to signal and then cancel without ever maneuvering for the lane change. If I had a category for this, which I don't, it would be 6.7% in the last 5 drives. Thus this new V10 artifact is easily the most common type of less-than-satisfactory result.

View attachment 473582

Weighted score:
View attachment 473583

View attachment 473584

Hey, Enginerd, do you mind if I use some screenshots of your graphs in a video on my channel? I will of course credit the work to you.

 

[Enginerd]

Sure thing. If you have Excel, I could also just give you the file.

 

[Didgie]

Sure thing. If you have Excel, I could also just give you the file.

That would be great.

 

[Enginerd]

Drives 45 & 46 showed solid performance, with most of the point deductions due to failure to exit the passing lane before being passed on the right.

• Another "speed matching" incident on drive #46, similar to the one experienced on drive #44. This time we were attempting to pass a minivan pulling a trailer. We locked on and hung out next to the minivan, below the target speed, with the signal on, apparently wanting to occupy the same space as the minivan. Having seen this before, and with other faster cars behind me also attempting to pass, I simply accelerated out of the situation with the go pedal. Once clear of the minivan, all was normal again. Although the minivan driver used his high beams to indicate his displeasure with our interest in collocation. For scoring purposes, I put this one in the "driver decline" bucket.
• Perhaps the most compelling indication of improvement is that 8 of the past 11 drives with V10 have not required a driver abort. This implies that the NOA lane changes were sufficiently safe that no driver intervention was required for the entire drive (~263 miles). I've attempted to capture this in the last chart, which illustrates this binary result against the weighted score trend.

Weighted score:

Weighted score with binary result: 1 = no driver abort required for the entire drive.

 

[DopeGhoti]

I've not been taking data down myself, but I've finally taken the plunge, so to speak, and have turned on autonomous lane changes. I have, however, turned NoA off by default. It doesn't yet handle on-ramp merges cleanly enough for me based on when it's been prompting me so far, but everything else has been wonderful. So I get on the highway, merge and then turn NoA on for the duration. I've had exactly one issue since, and that was tonight- one car-aborted lane change even though I was the only car on that section of road at the moment.

 

[Fernand]

Just drove from Bay Area to Reno NV through the Sacramento traffic on 2019.32.12.2 and back on 2019.36.2.1 with NOA on all the time except for very brief manual sections, like places where the road has no lane markings due to snow plow scraping. Without quantifying, the improvement in smoothness and smarts in handling unanticipated situations, was significant. It's getting REALLY good. I wouldn't dream of going without. I had a Model S loaner without any automation, and it was medieval.

I normally turn on NOA during on-ramp, it handles the merge better than I do. Had one funny section where the car kept leaving the Carpool lane until I got to the right screen (Lawd, I HATE trying to tap the little "X" to dismiss a screen) and enabled using HOV lanes.

 

[Fernand]

Another NOA drive today in dense freeway traffic, and the thing that stood out is how brave my Nicki has gotten ;-) A few months ago s/he wouldn't dream of cutting in. That was polite, but a bit impractical. Now when that exit is near, and it's tight, this brave little car muscles in with the best of 'em. A lot of the "more human" touches with every update now.

Had a section of non-NOA highway today that went better than ever, including smart slowdown at the entry into curves, then speeding up, rather than strictly going by speed limit. I'm not sure where that applies, as I still scroll the speed down on some overpass ramp merges in NOA. The ability to request and execute an automatic lane change without being in NOA is great.

HOLD mode makes slowdowns in NOA more dramatic. Takes a bit to get used to it, but seeing the green regen bar is good.
.

 

[Didgie]

Hey Enginerd, any data on 2019.36 yet?

 

[Enginerd]

 

[Enginerd]

This update logs the first 2 drives with the 2019.36.2.1 update. While there are some nice new enhancements, drive 48 included a handful of disappointing performances with basic maneuvers.

• Drive 47 had no driver aborts, or as Elon says, at no time did the driver have to intervene. Unfortunately, I had to intervene 3 times on drive 48 (described below).
• Wide berth for semi trucks: now works with two lanes in same direction (e.g. #1 & #2 lane). Previously on 2019.32.11.1 this feature was only enabled on highways with 3 lanes or more. This feature also now applies to some smaller vehicles like utility vans and big pickups.
• I'm fairly certain the same feature also positions at the far side of the lane when passing a stopped vehicle on the shoulder. It's subtle, but I'm fairly certain I saw it several times. Not sure this has been reported elsewhere.
• Strange event where NOA signalled and began to maneuver for an unnecessary exit. NOA aborted the maneuver and returned to the #2 lane. Quickly, NOA signalled left and moved into the #1 lane for no apparent good reason... like I wouldn't notice the lane change mistake if it looked like it was doing something on purpose? Then NOA switched back to the #2 lane. I logged this cluster as 1 AP abort and 2 successful lane changes, even though all 3 maneuvers were unnecessary.
• Drive 48, driver abort #1: NOA had taken the #1 lane (of 2) due to slower traffic entering the highway. When these recent entrants had sorted out their speeds, NOA wanted to switch out of passing lane #1, although a truck was gaining in lane #2. While it was shaping up to be a borderline jerk move, I let NOA do it. However, NOA got all swervy, and after the second swerve towards lane #2, I took over to preserve what was left of my dignity.
• Drive 48, driver abort #2: Entering I-20E from SC-340. I've had mixed results at this location in the past. Recently NOA had seemed to be performing better. After the merging zone, the onramp vanishes to a point. If the car finds itself at the end of the vanishing lane, sometimes it just jumps over without signaling. When the lane runs out after signaling, the car seems to want to jump 2 lanes. When it was clear that we were overshooting into the #1 lane, I aborted. It seems that the car can successfully initiate the real lane change while less than about 50% over the lane line. Initiating the real lane change when already more than 50% over the lane line results in overshooting. [Video doesn't really illustrate the event, which fell between file splices.]
  
• Drive 48, driver abort #3: Passing a big rig (only a few mph slower), NOA did not properly detect a Camaro approaching from the rear at maybe +10 mph. While I had hoped NOA would detect it at the last second and show a red lane line, it did not. Had I not aborted, I think there would have been brake lights, tire smoke, perhaps a horn, and most certainly an extended digit.

Weighted score:

Driver intervention required?

So far with 2019.36.2.1: