How Tesla could potentially solve “feature complete” FSD decision-making with imitation learning

[strangecosmos2]

@DrDabbles, do you understand the difference between control (i.e. low-level actuator commands) and planning (i.e. making decisions and determining a trajectory) in an autonomous car context?

Do you understand that HW2+ Autopilot currently uses a neural network trained via imitation learning for at least one planning task, i.e., path prediction (but not for control)?

Do you understand that the deep learning technique Tesla used to train path prediction — imitation learning — is the same technique DeepMind used to train AlphaStar Supervised?

 

[im.thatoneguy]

EDIT: i don't believe that's simply a view of a traj of a car (the first minute). that looks like a full HD map if i ever seen one.

I've never seen a full HD map with massive gaps in the data. Worst FSD-HD map ever. There is also noise in the splines. It's being solved in realtime. But that's impressive that their line detection and prediction neural network is so good that you think it's a human-annotated HD map. Note that the turn lane dashes aren't detected as lane lines. It makes no sense if they were HD map annotations have random gaps like that. Makes perfect sense in that they don't look like lane lines from that perspective so the neural net isn't predicting lane lines there.

Note that current public releases of AP similarly predict 'virtual' lane-lines where there are no markings as you drive through intersections. I drove through a strange interchage today and noticed that there was a full gap in a double line for cross traffic and the lane line was accurately double, and then turned into a "single" lane line through the cross-traffic gap in all lane markings. And when cars pass you on either side the car infers virtual lane lines behind the cars as well. Those aren't HD maps... if they were, they would be way better.

It's just the urban application of their road curve prediction model but presumably trained on annotated city streets and intersections.

To quote Karpathy in the video

Path prediction actually is live in the fleet today by the way so if you're driving clover leafs if you're in a clover leaf on the highway until maybe five months ago or so your car would not be able to do clover leaf now it can that's a prediction running live on your cars we've shipped this a while ago and today you are going to get to experience this for traversing intersections a large component of how we go through intersections in your drives today is all sourced from a prediction from automatic labels.
​

I think Tesla is wrong about maps. I think they should use HD Maps. I'm a huge fan of HD maps. And I think Tesla lies through omissions constantly and over promises and hypes future tech. But in this instance, they would have to be flat out committing fraud and lying to investors. What's more likely? Tesla's legal department reviewed and approved official slides for investors that are patently false or their glitchy data is real glitchy data? If it was an HD Map, it should be rock solid without any noise and perfect. There should also be alignment issues because I doubt Tesla's localization without LIDAR could achieve cm level accuracy.

​

 

[strangecosmos2]

I’ve been on the Internet long enough to learn that when people say stuff like this...

You're making up pretty much everything in here based on ignorance and complete speculation. ... There's really no reason to debate facts against fabrication.

...they’re often wrong.

I think the reason for this correlation is simple. Epistemic humility begets learning. The more unsure you are about the correctness of your own views or hunches and, conversely, the more unsure you are about the incorrectness of other people’s views when they happen to contradict your own, the more open you are to absorbing new ideas, knowledge, and evidence.

Hyperconfidence and berating people who think differently is a red flag.

Epistemic humility isn’t just polite; it’s a crucial part of learning and evolving your opinions based on new evidence.

 

[dgatwood]

There should also be alignment issues because I doubt Tesla's localization without LIDAR could achieve cm level accuracy.

To be fair, humans are lucky to achieve foot-level accuracy, and if you get close enough to something that a centimeter matters, you’ve probably already hit it.

So I, too, am unconvinced that HD maps are necessary, though I wouldn’t mind seeing Tesla use them as a way of detecting missing stop signs and nonfunctional traffic lights that should be working, as an added layer of safety.

 

[HappyFunBall]

The more unsure you are about the correctness of your own views or hunches and, conversely, the more unsure you are about the incorrectness of other people’s views when they happen to contradict your own, the more open you are to absorbing new ideas, knowledge, and evidence.

Hyperconfidence and berating people who think differently is a red flag.

I agree with you Trent. Humility is important, and it is associated with the ability to absorb new knowledge — especially when supported by strong evidence.

It seems to me that you don’t ever feel this way about yourself, though. You don’t practice what you preach.

You’ve written hundreds of articles, and you routinely present yourself as an expert. Your presentation certainly doesn’t suggest humility or openness to new ideas.

You’ve also been saying exactly the same things for years: that Tesla has a huge data / fleet advantage, and that they are either currently ahead of everyone else, or they will be soon.

Yet your SeekingAlpha articles routinely get literally hundreds of comments pointing out your misconceptions, and you just ignore them. You only respond to the people who compliment you.

You’ve also posted your Medium articles all over internet, and you’ve had Oliver Cameron, a leading mind in the field, literally tell you, “I appreciate the thoughtfulness, but this is a misguided and inaccurate post.”

But whenever you have a disagreement with someone, even people who clearly know more than you, you find a way block them, get them banned, or go on a holier-than-thou diatribe about how they’re being rude or not epistemically humble. You always find a way to declare yourself the winner.

Turn your lens around and look at your own behavior. Happy new year, my friend. I hope 2020 is a good one for you. We are all in need of personal growth, and that includes you.

 

[strangecosmos2]

So I, too, am unconvinced that HD maps are necessary, though I wouldn’t mind seeing Tesla use them as a way of detecting missing stop signs and nonfunctional traffic lights that should be working, as an added layer of safety.

Isn't Tesla doing something along these lines with stop signs? I recall Elon mentioning this as a possibility and I also recall verygreen saying maps were involved in stop sign detection.

u/brandonlive on r/SelfDrivingCars suggested Tesla could use maps to detect false negatives. If the map says there's a stop sign and the vision NN doesn't detect it, upload a snapshot to HQ and get the annotators to look over it. So, use the map to assist in training in this instance rather than in inference. Sounds like a good idea to me.

 

[ohmman]

Moderator note: Please keep discussions on the topic and off of individuals. 9 posts were moved away to snippiness.

 

[im.thatoneguy]

I wouldn’t mind seeing Tesla use [HD Maps] as a way of detecting missing stop signs and nonfunctional traffic lights that should be working, as an added layer of safety.

That's not HD maps though. That's just maps. And Tesla is using them. Navigate on Autopilot uses mapped lane counts/positions for exits. So it uses vision for cm level accuracy and then standard maps (not HD maps) as backups for lane position requirements and double-checking traffic signals.

 

[Bladerskb]

I've never seen a full HD map with massive gaps in the data. Worst FSD-HD map ever. There is also noise in the splines. It's being solved in realtime. But that's impressive that their line detection and prediction neural network is so good that you think it's a human-annotated HD map. Note that the turn lane dashes aren't detected as lane lines. It makes no sense if they were HD map annotations have random gaps like that. Makes perfect sense in that they don't look like lane lines from that perspective so the neural net isn't predicting lane lines there.

Note that current public releases of AP similarly predict 'virtual' lane-lines where there are no markings as you drive through intersections. I drove through a strange interchage today and noticed that there was a full gap in a double line for cross traffic and the lane line was accurately double, and then turned into a "single" lane line through the cross-traffic gap in all lane markings. And when cars pass you on either side the car infers virtual lane lines behind the cars as well. Those aren't HD maps... if they were, they would be way better.
View attachment 494708

It's just the urban application of their road curve prediction model but presumably trained on annotated city streets and intersections.
View attachment 494711

To quote Karpathy in the video

Path prediction actually is live in the fleet today by the way so if you're driving clover leafs if you're in a clover leaf on the highway until maybe five months ago or so your car would not be able to do clover leaf now it can that's a prediction running live on your cars we've shipped this a while ago and today you are going to get to experience this for traversing intersections a large component of how we go through intersections in your drives today is all sourced from a prediction from automatic labels.
​

I think Tesla is wrong about maps. I think they should use HD Maps. I'm a huge fan of HD maps. And I think Tesla lies through omissions constantly and over promises and hypes future tech. But in this instance, they would have to be flat out committing fraud and lying to investors. What's more likely? Tesla's legal department reviewed and approved official slides for investors that are patently false or their glitchy data is real glitchy data? If it was an HD Map, it should be rock solid without any noise and perfect. There should also be alignment issues because I doubt Tesla's localization without LIDAR could achieve cm level accuracy.

​

No dude its aggregated and post processed data, this is why there is no instability of any kind. Dude look at the Image, ITS A MAP.
It even have parking lots mapped and have areas not in the vicinity of the car showing up in the image.

I'm not talking about the second part of the video with the path prediction NN (your second pic). That one is a NN and you can tell from its micro jumps and instability. Remember that NN are doing per frame detection which is why you can always see the instability no matter how micro and insignificant it is but maps never move no matter what no matter if you are 1 meter away or 1,000 meter its 100% stable.

But what you see in the first image is a HD map with lanes in the hd map. The blue line is a aggregated/post processed trajectory of what cars took. The path prediction network is trained using raw images and that trajectory.

Unless you are trying to say that a NN can see through a house from 500+ meters away and outline the exact lane lines down to 1cm with no jitter of any kind all from 500+ meters away while its being occluded by a house.

Dude that's a hd map. You don't need a full set of recognizable features. You can have a hd map with just lane lines and driving trajectory of each lane.

Here's an early version of REM being demo on Aptiv's car using just lane line and drivable paths and the car was able to localize.

 

[im.thatoneguy]

No dude its aggregated and post processed data, this is why there is no instability of any kind. Dude look at the Image, ITS A MAP.

Timecode and screen shot to defend your point (not a random screenshot from actual HD maps from another company). I see a standard map and jittery unstable predicted lane lines. The map doesn't have multiple lanes, it's just road segments like Google or OSM.

 

[Bladerskb]

Timecode and screen shot to defend your point (not a random screenshot from actual HD maps from another company). I see a standard map and jittery unstable predicted lane lines. The map doesn't have multiple lanes, it's just road segments like Google or OSM.

I did, I pointed to the first and second pictures you posted. Those are two different things. The first pic is an obvious hd map and the second is the NN path prediction model that is in production today.

This is Tesla's actual lane detection model and their NN path prediction model.
Notice the obvious jitters. Yeah. That's called per frame detection. Also notice you never see lanes detected through a house from like 500 meters away.

Those red lines are actual actually individual lane lines from x00 meters away. They are just showing up as a cluster. Notice how its seeing through the houses and as the car gets closer there are no jitters or stuff appearing and being detected. The lanes are just there. that's not a NN per frame detection. Its so obvious.

The cyan/green line also is just the trajectory of the car and isn't a NN path prediction compared to the yellow line in the videos. This is equivalent as the cyan/green line in the Mobileye REM picture.

 

[strangecosmos2]

I know there was some confusion in this thread about the different subsystems of the self-driving car software stack, so I wanted to share a helpful diagram from Waymo:

As you can see, there are four main subsystems:

1. Perception (including computer vision, radar-based perception, and sensor fusion).

2. Behaviour prediction (mainly predicting the trajectories of vehicles, pedestrians, and cyclists).

3. Planning (also sometimes called decision-making, behaviour generation, or driving policy).

4. Controls (i.e. what translates between the planner and the vehicle's actuators).

This thread's OP and any analogy between Tesla's autonomy software and AlphaStar pertains only to (3), the planning or decision-making component of the software stack. The planner is where mid-to-mid imitation learning (such as Waymo's ChauffeurNet) is applied. Not the computer vision subsystem or the controls software.