-
Want to remove ads? Register an account and login to see fewer ads, and become a Supporting Member to remove almost all ads.
Navigate On Autopilot: automatic lane change results
- Thread starter Enginerd
- Start date
Dithermaster
Member
OMG, I love it. Probably makes the most sense, and easy enough to tally. I wouldn't have thought of it. I'd probably have done an audio recording, said very unique words for each thing, and ran it through speech-to-text later.
I really hope over the long haul with software updates we'll see the results get better. I hope your graphs will indicate where new software versions came in.
Thanks for doing it. I hope Tesla sees it and pays attention.
New fans all around Enginerd 
I made a quick video of a few things in AP for 2019.20.4.2 (right before 2019.24 hit the wild). I really feel like AP has improved lately, can't quite put my finger on it and certainly don't have your level of data but everything seems smoother and more confident (or it could just be me?).
DidgieTalks
I made a quick video of a few things in AP for 2019.20.4.2 (right before 2019.24 hit the wild
). I really feel like AP has improved lately, can't quite put my finger on it and certainly don't have your level of data but everything seems smoother and more confident (or it could just be me?).DidgieTalks
2019.20.4.2 seems par for the course as far as NOA is concerned. Results this weekend seem to indicate no noteworthy changes.
I'm debating whether to include repeatable highway entrance/exit errors. My preference would be to exclude what likely may be map errors that are substantially independent of NOA's vision/sensing/neural network algorithms. Problem spots on my route are:
I'm debating whether to include repeatable highway entrance/exit errors. My preference would be to exclude what likely may be map errors that are substantially independent of NOA's vision/sensing/neural network algorithms. Problem spots on my route are:
- I-95N to I-20E: NOA indicates intent to exit (for Florence SC Supercharger), but doesn't execute.
- I-20E to I-95S: NOA successfully avoids the atypical 141A left exit, but then stays in the left lane where it is impossible to make the 141B exit, which is 2 lanes away.
- I-74N at NC exit 77 Spero Rd.: There's no reason I would take this exit, as I continue on another 20 miles. NOA display shows it will continue straight, but without broadcasting prior intent, NOA attempts to exit before I override with steering torque.
Regarding the "failure to overtake" event where NOA slowed down when passing a car in the "slow" lane, is it possible NOA predicted you would be cut off by the faster Mini, and tried to create more separation to prevent an incident? I've not experienced this behavior myself. Perhaps others have?
Yesterday I drove I-95 from Savannah to Jacksonville & back. I realize now that my usual commute route really only evaluates NOA in areas with 2 lanes in each direction of interstate highway. SAV to JAX has 3 lanes in each direction, which highlights an auto lane change deficiency. It seems the logic may work like this:
So, in areas with wider freeways, NOA currently seems to need regular assistance to behave as one would expect.
- 2 lanes: pass in left lane (#1), return to right slow lane (#2) when complete
- 3 lanes: pass in the next lane to the left (#2 or #1), but only return to #2 after passing in #1.
So, in areas with wider freeways, NOA currently seems to need regular assistance to behave as one would expect.
Summer road trips have kept me from my usual "commute", but here's an update on 2019.20.4.4. Note that I've maintained minority firmware status lately until receiving the notification for 2019.24.4 today. Although it's possible that NOA trends have improved slightly in the 6 software updates since I started recording data, present results are also within the bounds of observed variation. I'm looking forward to Musk's promised "improved highway Autopilot" in V10 next month.
Greetings. Today is my first 2019.24.4 update. NOA lane change performance is solid, but not perfect. When I first started this exercise, NOA was successful in about 8 of 9 attempts. That ratio has improved beyond 9 of 10, and results from the last 4 trips averaged 93%. Elon has talked about the march of 9's (99%, 99.9%, 99.99%, etc.), and although NOA performance hasn't reached the second 9 yet, it does appear to be trending in that direction.
Independent of this data gathering exercise, I've noticed two discrete improvements in 2019.24.4:
1. Tendency to dive for some exits (or turning lanes) is reduced, but not eliminated.
2. When the lead car slows or stops to make a left turn (undivided road, and yes, I know, not "authorized"), after the car is clear, AP pulls away more smoothly some of the time, instead of accelerating briefly, letting off, then accelerating up to the set speed.
The first chart shows the full breakdown:
The second chart gathers each type of result into a single weighed score (i.e. safety violations are worse than insufficient courtesy or passenger discomfort). This is where the gradual improvement trend starts to stand out.
Independent of this data gathering exercise, I've noticed two discrete improvements in 2019.24.4:
1. Tendency to dive for some exits (or turning lanes) is reduced, but not eliminated.
2. When the lead car slows or stops to make a left turn (undivided road, and yes, I know, not "authorized"), after the car is clear, AP pulls away more smoothly some of the time, instead of accelerating briefly, letting off, then accelerating up to the set speed.
The first chart shows the full breakdown:
The second chart gathers each type of result into a single weighed score (i.e. safety violations are worse than insufficient courtesy or passenger discomfort). This is where the gradual improvement trend starts to stand out.
Greetings. This week's update includes 2 more round trips with 2019.24.4...
Cumulative performance with this firmware:
I have a working theory about a couple of NOA's quirks. After watching Model3man's video on the topic, and combining that with my own observations, I think that NOA lane change decisions are significantly based on observed evidence of lane usage by other vehicles.
I'm curious if anyone has thoughts on this. I think in normal traffic, there are enough vehicles passing in various lanes to make this logic transparent. But in light traffic, it exposes slightly odd behavior. On one hand, this observed-evidence characteristic is what makes it possible to make an abnormal pass on the right (slow lane) when warranted. But it seems like it may also play a larger role in NOA's lane choice.
Cumulative performance with this firmware:
Allow me to walk back an observation from my prior post. Although there is improvement in the two noted features, it's only better some of the time. The old behavior still shows itself occasionally.
I have a working theory about a couple of NOA's quirks. After watching Model3man's video on the topic, and combining that with my own observations, I think that NOA lane change decisions are significantly based on observed evidence of lane usage by other vehicles.
- For example, sometimes the car will hang out behind a slower vehicle without attempting to pass. Then someone else comes along and passes you both. When the cameras observe that car passing the slow car ahead of you, NOA seems to realize "Hey, that guy just showed us we can pass in that lane. We've been stuck here, but now we know the way. Let's do it."
- In some light traffic situations, after passing a slower car via the fast lane, NOA doesn't exit the passing lane. I had an instance of this yesterday, and just as I was wondering if NOA would ever switch out of the passing lane, another car some distance behind me passed that same slow car. It was at that moment that NOA changed back to the slower lane. It seemed that NOA would have stayed in the fast lane until it saw another car overtaking the slow car (and potentially eventually overtaking us). Other than this distant passing action to the rear, I could identify no other reason for NOA to initiate the lane change at that particular time.
I'm curious if anyone has thoughts on this. I think in normal traffic, there are enough vehicles passing in various lanes to make this logic transparent. But in light traffic, it exposes slightly odd behavior. On one hand, this observed-evidence characteristic is what makes it possible to make an abnormal pass on the right (slow lane) when warranted. But it seems like it may also play a larger role in NOA's lane choice.
I just downloaded 2019.28.3.1, so this marks the final update for 2019.24.4. Performance remains relatively solid (93.8% success overall for 2019.24.4). The most frequent failure mode is "driver abort", due to NOA acting to cut off faster drivers approaching in the passing lane from the rear.
Attachments
The data doesn't show any major NOA improvements in 2019.28.3.1. I'll note that 2 of the 3 "uncomfortable successes" on drive 26 were freeway onramps, where NOA overshot when merging into the slow lane, making unnecessary contact with the far lane line.
One event today that doesn't really fit any of my categories is a single lane that divides into two. AP seems to indecisively hunt before making a random selection. IIRC, as soon as AP settled into the left lane, NOA initiated a lane change to the right to get out of the passing lane. Forethought? Not so much. Sitting int he car, this feels a lot more "drunk driver" than it looks from the TeslaCam video. This isn't new behavior, it's just the first time I remembered to save a clip.
One event today that doesn't really fit any of my categories is a single lane that divides into two. AP seems to indecisively hunt before making a random selection. IIRC, as soon as AP settled into the left lane, NOA initiated a lane change to the right to get out of the passing lane. Forethought? Not so much. Sitting int he car, this feels a lot more "drunk driver" than it looks from the TeslaCam video. This isn't new behavior, it's just the first time I remembered to save a clip.
Dithermaster
Member
I've had that happen as well, and now I know to take over before the lane splits (until I read that AP can handle it). Your description is accurate -- indecisive.
Arguably this strategy is better than hugging the wrong lane line which turns out to not be a lane line but the wrong side of gore zone border line.
I had something similar happen to me recently on I-15 southbound, between Mesquite, Nevada and Las Vegas. I was currently in the left lane (passing a big rig), when the availability of a "climbing lane" split the left lane into two lanes, resulting in a total of three lanes. AP started swerving so badly that the oscillations eventually overcame the ability of the AP to control the car, and the AP disconnected entirely. This must have looked hilarious from behind. Not only did the AP disconnect, but I landed in "Autopilot Jail." The car said that I had exceeded the AP's speed limit, and that AP would not be available for the rest of the drive. I had to take an unplanned exit just to shut the car down so I could get AP back. This was firmware 28.3.1, btw.
SomeJoe7777
Marginally-Known Member
A couple more drives on 2019.28.3.1 remind me that the scatter band in this kind of data is noteworthy.
- Drive #27 included 2 driver aborts and 4 failures to switch out of the passing lane.
- Drive #28 had the fewest lane change events to date: only 46 attempts (average 68). I was driving towards an active hurricane, when most sensible people were driving away from it.
This week's update:
Weighted scores:
Cumulative results with my current firmware:
- Drive #29 was another hurricane-related oddity: there weren't many other drivers following Dorian up the coast with me on 9/5... just utility trucks. Due to the rain, NOA was unavailable for a good portion of the drive. Thus, another record low 27 NOA events (average 67) with only one "uncomfortable", resulting in the highest weighted score to date, 0.981.
- Drive #30 was unremarkable, except for the realization that every time a single lane splits into two, AP acts like a crazy person. I don't think I've found any examples where it simply takes stab at the best lane choice and just sticks with it. Always the swervy indecision at the last second.
Weighted scores:
Cumulative results with my current firmware:
Similar threads
