Karpathy talk today at CVPR 2021

[diplomat33]

Karpathy just finished a new presentation moments ago at CVPR 2021.

Here is livestream.

Karpathy presentation starts around 7:53:00:

He mentions FSD Beta can do zero intervention drives in sparse conditions but struggles with dense, high adversity conditions (like busy traffic in San Franscisco). He also talks about auto labelling. He discusses how Tesla is taking a vision-only approach in contrast with Waymo that uses a combination of camera, radar, lidar and HD maps. He also gives some examples of why Tesla dropped radar. Basically, Tesla's radar and sensor fusion sucks so Tesla decided to just drop it rather than try to fix it. And he mentions the data that Tesla has collected so far (it's a lot!) and hints at Dojo and more...

 

[Bladerskb]

Karpathy just finished a new presentation moments ago at CVPR 2021.

Here is livestream.

Karpathy presentation starts around 7:53:00:

He mentions FSD Beta can do zero intervention drives in sparse conditions but struggles with dense, high adversity conditions (like busy traffic in San Franscisco). He also talks about auto labelling. He discusses how Tesla is taking a vision-only approach in contrast with Waymo that uses a combination of camera, radar, lidar and HD maps. He also gives some examples of why Tesla dropped radar. Basically, Tesla's radar and sensor fusion sucks so Tesla decided to just drop it rather than try to fix it. And he mentions the data that Tesla has collected so far (it's a lot!) and hints at Dojo and more...

Who could have known that fusing with a 2021 and 2014 low resolution 2d radar would be hard compared to a 2021 4d high resolution imaging radar that others are using.

 

[diplomat33]

Who could have known that fusing with a 2021 and 2014 low resolution 2d radar would be hard compared to a 2021 4d high resolution imaging radar that others are using.

Yeah. Elon and Karpathy talk as if radar and sensor fusion don't work, in order to justify their vision-only approach. But the problem is not sensor fusion itself, the problem is Tesla was trying to do sensor fusion with low quality sensors. Everybody else is using high quality sensors and doing great with sensor fusion.

At this point, Tesla does not really have a choice IMO. They committed early to putting cameras on every car and claiming the hardware was capable of FSD. And they've sold hundreds of thousands of cars now with the hardware. They can't afford to upgrade cars to newer radar or put newer radar in future cars. I think Tesla is basically forced now to somehow try to make "vision-only" work as best they can.

 

[Knightshade]

How much does better radar cost?

30 second google search suggests like $100 per car for automotive radar (unclear if this is the "good" kind though).

At 1 million cars that'd cost Tesla 100 million dollars. Let's say $100 for labor too. So 200 million dollars.

Even if these estimates are off again by double you're talking 400 million bucks.


Tesla has what like 17 billion in cash sitting around?

The idea they couldn't afford it doesn't seem to hold up to math.

 

[BitJam]

The big question is: how far will Tesla get with vision-only using the current cameras?

This could be a treacherous path for them.

 

[thewishmaster]

Damn, I bookmarked to watch this tonight but looks like they made the whole livestream private. Hopefully they post the talks somewhere as that whole workshop sounds pretty interesting.

 

[RobDickinson]

The big question is: how far will Tesla get with vision-only using the current cameras?

This could be a treacherous path for them.

Humans do fine with single point stereoscopic vision, its not an impossible problem to solve.

Is it harder or easier to do than with mixed sensor data from radar etc also?

Will it add more time and cost more than retrofitting better radar?

 

[powertoold]

If what Karpathy is saying is true, then V9 is going to blow our minds, in probably two weeks.

 

[diplomat33]

Damn, I bookmarked to watch this tonight but looks like they made the whole livestream private. Hopefully they post the talks somewhere as that whole workshop sounds pretty interesting.

No, you should be able to click on the link to view in youtube and watch it there. It won't play embedded in the forum but it will play in youtube. I can still view it just fine in youtube.

 

[diplomat33]

If what Karpathy is saying is true, then V9 is going to blow our minds, in probably two weeks.

IMO, "blow our minds" is becoming a tired cliche at this point. We are told every release will "blow our minds".

The truth is that "blow our minds" is subjective. What might impress one person, might not impress another person the same way.

Also, "blow our minds" tends to go away when the novelty of the feature wears off. I will admit that smart summon, nav on AP, traffic light control, blew my mind the first time I used the features. But that was only because they were brand new and I'd never seen anything like it before. Once I used the features for awhile and started seeing some of the shortfalls, the features did not "blow my mind" anymore. I think the same thing will happen with V9. Yes, it may blow our minds the first time we use it, only because it is new and something we've never experienced before. But once the novelty of it wears off and especially when we experience safety interventions, it won't blow our minds anymore, or at least not as much.

 

[msanghavi]

IMO, "blow our minds" is becoming a tired cliche at this point. We are told every release will "blow our minds".

The truth is that "blow our minds" is subjective. What might impress one person, might not impress another person the same way.

Also, "blow our minds" tends to go away when the novelty of the feature wears off. I will admit that smart summon, nav on AP, traffic light control, blew my mind the first time I used the features. But that was only because they were brand new and I'd never seen anything like it before. Once I used the features for awhile and started seeing some of the shortfalls, the features did not "blow my mind" anymore. I think the same thing will happen with V9. Yes, it may blow our minds the first time we use it, only because it is new and something we've never experienced before. But once the novelty of it wears off and especially when we experience safety interventions, it won't blow our minds anymore, or at least not as much.

At the end, answering the question about low light, Karpathy says that humans can get away with vision only, so cars should also. He didn't answer the part about thermal cameras. My understanding is that Tesla cameras may have low megapixels (which might have trouble reading far away signs but that's solved as the car gets closer), but the cameras have 115 db (dynamic range). Which I think is high dynamic range? So the car should be able to function in low light and bright light? That could help in low visibility scenarios like rain, fog, dust.

 

[aronth5]

Karpathy talk, very informative

 

[thewishmaster]

No, you should be able to click on the link to view in youtube and watch it there. It won't play embedded in the forum but it will play in youtube. I can still view it just fine in youtube.

Yeah, looks like they made it public again. Whew!

 

[Bladerskb]

How much does better radar cost?

30 second google search suggests like $100 per car for automotive radar (unclear if this is the "good" kind though).

At 1 million cars that'd cost Tesla 100 million dollars. Let's say $100 for labor too. So 200 million dollars.

Even if these estimates are off again by double you're talking 400 million bucks.


Tesla has what like 17 billion in cash sitting around?

The idea they couldn't afford it doesn't seem to hold up to math.

Definitely not $100 for the kind of radar used for autonomous driving. There’s a vast difference between ACC and autonomous driving.

You are talking about a company who didn’t put a rain sensor on their car even though it cost less than $1 each.

The 2011 and 2014 ACC radar that Tesla uses are around $60-70 dollars.

 

[Knightshade]

Definitely not $100 for the kind of radar used for autonomous driving. There’s a vast difference between ACC and autonomous driving.

You are talking about a company who didn’t put a rain sensor on their car even though it cost less than $1 each.

The 2011 and 2014 ACC radar that Tesla uses are around $60-70 dollars.

Ok, then how much is "good" radar?

They didn't put a rain sensor because they felt they could solve with vision. Allegedly the newest code (in 18.3 I think) is quite good (it now is using multiple cameras when it previously was not).

 

[BitJam]

Humans do fine with single point stereoscopic vision, its not an impossible problem to solve.

Humans are able to move their heads and they have access to other senses too. Maybe someone's done tests of people driving with their heads in a fixed position and their ears muffled but I would not volunteer to participate because it seems dangerous. Also, I thought the point of FSD is to do much better than humans. If the goal gets reduced to doing as well as humans then I'd feel disappointed. ISTM getting a vision system that is at least on par with humans combined with radar (and even lidar) that can see when humans can't would be ideal.

Historically, Tesla has been wildly over-optimistic about FSD. On YouTube some people are gushing over the Karpathy talk. TBH, I'm not impressed. What would impress me is if they fixed the vision-only cars they're selling now. For examples: increase 75 MPH speed limit on auto-steer; reduce the increased minimum distance to the car ahead of you; restore seeing the car ahead of the car ahead of you; fix or improve auto-highbeams and auto-wipers or no longer require them to be on for AP and FSD.

In theory there's no difference between theory and practice but in practice there is.

 

[Knightshade]

A vision system that can see 360 degrees at the same time, with a reaction time far faster than humans, seems a lot better than "on par"

(for a system with fully solved vision anyway)

 

[BitJam]

(for a system with fully solved vision anyway)

Sure. Totally agree. But installing 8 cameras and a computer is the easy part. Solving FSD (vision-only or not) is the hard part.

 

[Knightshade]

Sure. Totally agree. But installing 8 cameras and a computer is the easy part. Solving FSD (vision-only or not) is the hard part.

Yup- and Karpathy appears to be saying it's a vastly better use of engineering resources to solve that than to try and get useful integration of low quality, noisy, radar data, especially when you STILL need to solve vision with that too.

 

[linux-works]

Basically, Tesla's radar and sensor fusion sucks so Tesla decided to just drop it rather than try to fix it.

I just said the same thing, just a few hours ago, and got tons of downvotes.

people dont want to hear this and I understand why, but facts are facts. dumbing down your sensor array because making sense of multi inputs is 'hard' - that's just another way of saying 'we give up'.

its fine to change strategy if what you have isn't working, but I think reducing sensor count is going 100% the wrong way. I suspect key individuals at tesla realize this, in their heart, as well.

time will tell.