Tracking local maximum to FSD improvements

[Mardak]

It seems pretty clear there won't be much improvement to the current pre-rewrite Autopilot, so that gives an opportunity to see the limits of the local maximum. Hopefully people can share examples of where Autopilot requires interventions and ideally when Autopilot is able to consistently pass with the FSD rewrite.

I've got plenty of examples, so I wanted to have a reference of failures as once Autopilot is able to pass some of these, it'll be harder to differentiate from "of course Autopilot could handle this." I've tried to keep these to be "static" cases of just staying in the current lane without dynamic actors.

Curve with clear lines on both sides taken too fast/wide

Angled intersection where Autopilot wanted to switch from left to right lane when crossing

Crested intersection where Autopilot panicked because it couldn't infer lines

Large intersection under the interstate where Autopilot starts to follow lines into the oncoming turn lane

Shifted lanes across a large intersection where Autopilot wanted to switch from the right lane to left

Single lane splitting 3 ways where Autopilot wanted to take the onramp instead of staying straight

A temporary gap in the lines and a dip then rise where Autopilot panics after incorrectly following the bike lane

Roundabouts just aren't handled yet - this one is pretty simple with one lane

 

[Ludalicious]

Ahhhh have you actually read the release notes. No mention of autopilot working on city streets but looks like most scenarios are in the city.

 

[Mardak]

No mention of autopilot working on city streets but looks like most scenarios are in the city.

Yes, I realize these are all unofficial use cases, but Autopilot does actually work pretty well on local roads and staying in the current lane. I would estimate for each of the examples I shared so far, Autopilot went through 20 intersections just fine (and not just the simple 2-road crossing at 90°). Then again, with stop behavior for traffic lights and stop signs, one would expect some city street ability as that's where one would find those traffic controls.

 

[Mardak]

Do people expect all or most of the above examples to be addressed by the initial FSD rewrite/beta? I suppose they might be considered some of the easier aspects of FSD as I purposely chose examples that don't involve planning for other vehicles' movements. So in some sense, these examples are trivial for those that rely on HD maps as that information should be correct. Karpathy did show off intersection predictions with lanes and traffic flow directions, but unclear if this is intended for the initial release.

 

[thewishmaster]

Most of your examples expose the rather naive algorithms currently used for lane following; they seem to check for some level of confidence in left/right lane markings and then bias the system to pick whichever left lane line is latitudinally closest to the car as the new left lane line when moving across intersections (or something like that). In the onramp case it probably is attracted to the sharply defined left curb on that on-ramp and the slight curve in the pavement markings that nudge it to take the curve instead of going straight.

I expect lane-following over ambiguous lane lines to improve. I'd be surprised if this isn't addressed as part of the rewrite because of how critical it is for city driving, although I half expect it to work "intelligently" only if you set a destination. Wouldn't be surprised if basic AP (or even FSD cars using simple lane-following without a destination) continues behaving the way it does today.

Not sure how much longitudinal control will improve for taking sharp curves, but I'd expect it to get better since the car will need to take 90+ degree turns.

 

[Mardak]

Most of your examples expose the rather naive algorithms currently used for lane following

Yeah, most of the examples I've posted so far are of the same family of problems of knowing what lane you're supposed to stay in. I'm not sure if Autopilot has an explicit algorithm or is just relying on neural network outputs, e.g., the thicker orange line in green's videos

I guess if people think lane following will be fully addressed with the initial FSD rewrite, are there other types of interventions I should share? E.g., parked cars with/without lane lines, behaviors around pedestrians/cyclists/animals. Once the rewrite is available, there might be some types of turning through intersection and yielding to other vehicles situations that the initial version fails and would be interesting to track.

Wouldn't be surprised if basic AP (or even FSD cars using simple lane-following without a destination) continues behaving the way it does today.

As in the current software runs for basic AP even after the rewrite? Or the rewrite could be smarter or at least behave differently when there's no destination -- just not guaranteed to "stay straight?"

 

[thewishmaster]

I guess if people think lane following will be fully addressed with the initial FSD rewrite, are there other types of interventions I should share? E.g., parked cars with/without lane lines, behaviors around pedestrians/cyclists/animals. Once the rewrite is available, there might be some types of turning through intersection and yielding to other vehicles situations that the initial version fails and would be interesting to track.

Those are probably all would be good to note, although parked cars to me falls under the same "lane finding algorithm failures" category. It would also be interesting to see if there was any specific behavior for navigating around double-parked cars or cars that intrude into your lane; right now with the old algorithms you'd stop behind the obstruction (assuming it was even detected, as in the case of when only a small part of a car is sticking out into your path).

Generally reacting to behavior of other agents on the road like pedestrians, cars that make left turns in front of you, etc are probably good to observe as well. I've noticed some extra-panicky behavior recently if there are pedestrians near your lane, even if they're not even moving towards you.

As in the current software runs for basic AP even after the rewrite? Or the rewrite could be smarter or at least behave differently when there's no destination -- just not guaranteed to "stay straight?"

The former would be my guess, as I'd expect performance/reliability reasons for the rewrite not scaling down to HW2.5 well, which they would need to support for running basic AP in the older cars. Of course there's always the chance that they'd keep the feature set the same but fork the code for HW2.5 cars vs. HW3...

 

[Mardak]

It would also be interesting to see if there was any specific behavior for navigating around double-parked cars or cars that intrude into your lane; right now with the old algorithms you'd stop behind the obstruction

From watching FSD beta videos, looks like all of my shared examples and more including intruding parked vehicles are handled just fine. Perhaps we set very low expectations for what the initial early access release would be capable of.

When I get the FSD software probably as part of general wide release, I'll definitely retest my examples and more that I have saved but haven't shared yet. But I somewhat expect Autopilot to handle all of these cases of "stay in lane" now.

I wonder if this indicates Autopilot team has indeed been collecting disengagements and tested that FSD rewrite would pass them. So many of the cases that FSD would have trouble are when Autopilot hasn't been allowed to be enabled, e.g., turning left or right, so there might be issues with predicting destination lanes.