Drago Anguelov – Machine Learning for Autonomous Driving at Scale

[shrineofchance]

Waymo's Head of Research on:

• simultaneous detection, tracking, and prediction

• representing the built environment and road users as polylines in vector space

• deep structure from motion (deep SfM)

• visual LIDAR (VIDAR)

• using generative adversarial networks (GANs) to generate synthetic data for driving simulations

 

[diplomat33]

Waymo's Head of Research on:

• simultaneous detection, tracking, and prediction

• representing the built environment and road users as polylines in vector space

• deep structure from motion (deep SfM)

• visual LIDAR (VIDAR)

• using generative adversarial networks (GANs) to generate synthetic data for driving simulations

This is great stuff. Waymo is doing real cutting edge work in autonomous driving. This is one reason why they are a leader in autonomous driving. And this video is actually "old". It's from June 2020. I am sure Waymo has made a lot of progress since this video.

 

[mikes_fsd]

I am sure Waymo has made a lot of progress since this video.

Oh, yeah? Where?

 

[shrineofchance]

The parts on deep SfM and VIDAR are fascinating because from a purely technical perspective you could interchange those slides from Anguelov’s talk with slides on visual depth perception from a Karpathy talk and you wouldn’t spot the difference.

It’s a reminder that cars with lidar vs. cars without lidar are not a difference in kind, but a difference in degree. Namely, the degree of perceptual redundancy.

The difference between the “lidar is unnecessary” camp (primarily Tesla but also others) and the “lidar is necessary camp” is a difference in opinion about whether a vehicle without lidar will have redundant enough perception.

Moreover, what is redundant enough is not a purely technical opinion, but also an ethical, political, sociological, and business opinion. What is redundant enough depends on what is safe enough, and what is safe enough is a divisive issue.

For instance, Elon has said he considers a 10x lower rate of fatalities, injuries, and collisions than the human average to be safe enough for an autonomous vehicle. By contrast, Amnon Shashua has explicitly argued that autonomous vehicles must be 1,000x safer than the human average or else, he believes, there will be so much public outcry when crashed happen that autonomous vehicles will be rejected by society.

Waymo’s leadership has evaded an explicit answer to the “safe enough” question, but I would guess from multiple kinds of indications that the company wants to err on the side of extreme caution.

If Waymo wants lidar to provide redundancy for cameras and not replace cameras, then Waymo has to solve the same computer vision problems as Tesla, including visual depth perception. That’s why Anguelov’s and Karpathy’s technical talks have so much overlap on the topics of VIDAR and deep SfM.

 

[diplomat33]

Oh, yeah? Where?

If you don't see the progress, you clearly have not been paying attention.

 

[shrineofchance]

Observers of this space often interpret CEO/CTO’s remarks as expressing a hard and fast technical opinion, but I think what they’re actually doing is explaining, justifying, or defending the company’s current technical strategy, which can change.

Both Tesla and Mobileye, for instance, have changed their technical strategies: Tesla is developing visual driver monitoring after Elon previously dismissed the idea and Mobileye embraced lidar after initially taking a stance against it.

Their technical strategies could very well change (again) with regard to lidar. For example:

 

[Bladerskb]

Observers of this space often interpret CEO/CTO’s remarks as expressing a hard and fast technical opinion, but I think what they’re actually doing is explaining, justifying, or defending the company’s current technical strategy, which can change.

Both Tesla and Mobileye, for instance, have changed their technical strategies: Tesla is developing visual driver monitoring after Elon previously dismissed the idea and Mobileye embraced lidar after initially taking a stance against it.

Their technical strategies could very well change (again) with regard to lidar. For example:

Mobileye were never anti-LiDAR. You can easily see from their presentations well back from 2015.

 

[diplomat33]

Observers of this space often interpret CEO/CTO’s remarks as expressing a hard and fast technical opinion, but I think what they’re actually doing is explaining, justifying, or defending the company’s current technical strategy, which can change.

Both Tesla and Mobileye, for instance, have changed their technical strategies: Tesla is developing visual driver monitoring after Elon previously dismissed the idea and Mobileye embraced lidar after initially taking a stance against it.

Their technical strategies could very well change (again) with regard to lidar. For example:

Maybe you already know this but Mobileye's strategy is to develop 2 separate FSD systems that can independently operate the car. They are developing a camera-only FSD system and a separate lidar/radar FSD system. The car can fully self-drive on just cameras OR fully self-drive on just lidar and radar. The idea is to have the 2 FSD systems in the car, like a pilot and a co-pilot in a plane, so that if one fails, the other can take over. Mobileye calls it "true redundancy". With this system, the chance of total failure would be significantly less since it would require both systems failing at the same time. Mobileye believes that this approach will be several orders of magnitude safer.

So I don't think Mobileye would ever remove lidar even if camera-only were "safe enough" since they already believe camera-only can do pretty safe FSD but still want the extra redundancy. Their entire approach relies on having both FSD systems in the car.

Now, Tesla is in a very different situation. Elon is "all in" on not using lidar, betting that Tesla can get to "safe enough" without lidar. If Tesla did need to add lidar, I think it would be a significant problem because they would need to redesign their cars to add lidar in them, adding cost and time, and they would also need to do a serious rewrite of their entire software, to train the lidar vision and incorporate the lidar vision into the perception stack.

My guess is that even if lidar is needed, Tesla will still refuse to add lidar in order to avoid the problems above. Instead, they will either release FSD without driver supervision at the best safety level they are able to achieve and argue that it is "safe enough" (for example: "2x safer than humans is good enough, deal with it"). Or, they can release FSD and just keep driver supervision, relying on the driver to intervene if something goes wrong.

 

[shrineofchance]

So I don't think Mobileye would ever remove lidar even if camera-only were "safe enough" since they already believe camera-only can do safe FSD but still want the extra redundancy. Their entire approach relies on having both FSD systems in the car.

They might remove lidar from their production AV kit if it meant they could deploy FSD to millions of production cars in a short period of time, rather than sitting around waiting for the cost of lidar to come down.

All that would need to change is their belief about whether lidar is required is to get to “safe enough”. For example, if another company deployed FSD without lidar and the social revolt that Amnon predicted didn’t happen.

Now, Tesla is in a very different situation. Elon is "all in" on not using lidar, betting that Tesla can get to "safe enough" without lidar. If Tesla did need to add lidar, I think it would be a significant problem because they would need to redesign their cars to add lidar in them, adding cost and time, and they would also need to do a serious rewrite of their entire software, to train the lidar vision and incorporate the lidar vision into the perception stack.

I think adding lidar would be a much lesser undertaking in terms of both hardware design and software development than the hardware and software changes that Tesla has already made or is currently in the process of making.

A separate but related issue is that Tesla has an obligation to every Tesla owner who has bought FSD so far.

If adding lidar to vehicles is enough for Tesla to make them fully autonomous, then the revenue from full autonomy will be enough to compensate people who own Teslas without lidar and who had purchased FSD. (For instance, refunding the cost of FSD and even refunding a portion of the purchase price of the car, to compensate customers who bought Teslas partially based on the company’s claims of what the hardware would ultimately be capable of.)

If adding lidar to vehicles is not enough for Tesla to make them fully autonomous, then Tesla isn’t in a different situation then it’s in today.

 

[powertoold]

I think the solution for FSD is simple, albeit difficult to achieve. FSD will not be a reality until human level vision (however you want to define it) is achieved. Tesla happens to be in the best position to not only achieve this but also capitalize on it (as they already have the fleet with sensors out in the wild).

Elon talks about 4D vision and whatnot, but that just means surround video and time. I hope 4D also solves the cone misplacements in the current visualizations. Although it does seem the cone placements are better in the fsd beta because the cones can be placed in the context of the birds eye view. If cone placement isn't almost "perfect", I don't see how it would be considered human level vision or better.

 

[Bladerskb]

I think the solution for FSD is simple, albeit difficult to achieve. FSD will not be a reality until human level vision (however you want to define it) is achieved. Tesla happens to be in the best position to not only achieve this but also capitalize on it (as they already have the fleet with sensors out in the wild).

Do you understand that Lidar handles every single thing that camera does for autonomous other than seeing the color of traffic lights? I suppose by human level vision you are referring to cameras?


You do understand you still have to solve prediction and planning which are harder than perception?

 

[powertoold]

Do you understand that Lidar handles every single thing that camera does for autonomous other than seeing the color of traffic lights? I suppose by human level vision you are referring to cameras?


You do understand you still have to solve prediction and planning which are harder than perception?

I think what a lot of people fail to consider is that once you can achieve human level vision or better in a neural network, not only do you achieve it for perception but also for all sorts of predictions from real world environments.

As for your comment about lidar, I disagree.

 

[Bladerskb]

I think what a lot of people fail to consider is that once you can achieve human level vision or better in a neural network, not only do you achieve it for perception but also for all sorts of predictions from real world environments.

This is simply not true and completely made up. I don't understand why don't you get your info from academic papers and technical talks. Why resort to making stuff up? I for example have watched and read over 1000 SOTA academic papers, AV tech presentation. Of-course you think all those people are stupid and Tesla AP engineers are the only smart people in the world. Even though tesla uses what is industry standard in AV and in academia 2-5 years after.

While having a perfect or near perfect perception system helps you create prediction networks by leveraging your perception system in a supervised learning auto labeling/training format. It DOESN'T solve prediction. You still have to develop new NN architectures. Prediction is still an open problem. Perception in good weather however is already solved. This is why Waymo is able to run their fleet in Phoenix.


Below in the Waymo video, you see clear improvement with each network based on independent verifiable benchmarks. Not some "quantum leap", "exponential", PR marketing nonsense.

Waymo Latest Prediction Network called TNT. In addition Drago talks about their "4D Deep Temporal Scene Understanding, ViDar & 3D Auto Labeling"

Cruise Prediction System (Continuous Learning Machine)

As for your comment about lidar, I disagree.

There's no such thing as agree/disagree. Its like saying i don't agree with a captcha.

List everything below that vision from camera does and then list everything that vision from lidar does and lets check.

For example, what below can lidar not do?

 

[powertoold]

For example, what below can lidar not do?

Not sure if you're trolling, but lidar can't do the majority in that picture.

 

[Bladerskb]

Not sure if you're trolling, but lidar can't do the majority in that picture.

can you list it out? Just want to make sure we are on the same page

 

[powertoold]

can you list it out? Just want to make sure we are on the same page

Well, first of all, what do you mean by "can lidar not do"? In particular, what does "do" mean?

 

[Bladerskb]

Well, first of all, what do you mean by "can lidar not do"? In particular, what does "do" mean?

Its quite simple. Whatever capabilities a camera based perception system can do, a lidar based perception system can do also besides color (mainly status of traffic lights).

Your response is "majority " of that picture. I wanted to give you a second chance so i asked you to list out everything on that picture or any other category that pertains to AV. So there's no "I actually meant this". When i go through each one. Clearly you believe a Lidar perception system can't do almost none of that because you think its a troll question.

 

[powertoold]

Its quite simple. Whatever capabilities a camera based perception system can do, a lidar based perception system can do also besides color (mainly status of traffic lights).

When you say "do", do you mean by virtue of the sensor alone or within the context of an HD map for example?

 

[Bladerskb]

When you say "do", do you mean by virtue of the sensor alone or within the context of an HD map for example?

The same exact way a camera-based perception system would do it.

 

[powertoold]

The same exact way a camera-based perception system would do it.

And when you say "do", do you mean it's been deployed for actual human use or just as an academic exercise? Because in academia, anything's possible but often not actually accurate enough for real world deployment.