Karpathy talk today at CVPR 2021

[mark95476]

Key parts of Karpathy's presentation--

"No longer being held back by a third party" and "Determine your own destiny"

Imagine paying a company for their product ($) and then having to spend your engineering resources ($$$) to fix or patch it for them. Even worse, waiting for them to fix it.

Yup- and Karpathy appears to be saying it's a vastly better use of engineering resources to solve that than to try and get useful integration of low quality, noisy, radar data, especially when you STILL need to solve vision with that too.

 

[linux-works]

Key parts of Karpathy's presentation--

"No longer being held back by a third party" and "Determine your own destiny"

Imagine paying a company for their product ($) and then having to spend your engineering resources ($$$) to fix or patch it for them. Even worse, waiting for them to fix it.

oh, if I could only tell some of my war stories. suffice to say, it happens all the time. 'we' clean up for vendors all the time.

I'll just drop this one word here and those who know, will grin.

"watchdog"

 

[Bladerskb]

Key parts of Karpathy's presentation--

"No longer being held back by a third party" and "Determine your own destiny"

Imagine paying a company for their product ($) and then having to spend your engineering resources ($$$) to fix or patch it for them. Even worse, waiting for them to fix it.

Imaging selling your 2011 and 2014 old radar that is only to be used for simple ACC to some company who has been trying to use it for a L5 driverless system.
Even worse, their fans are now saying your radar is broken and that the company was helping you by spending their engineering resources to help you fix or patch it.

 

[BitJam]

Karpathy appears to be saying [...]

They've been saying FSD is just around the corner for years now. They said going to video-only would improve performance of the system but it got demonstrably worse. They said/implied these issues would be fixed in a few weeks. They were not.

There is a disconnect between what they say and what they deliver. It also appears they were disingenuous about why they removed radar from North American Ys and 3s. You are, of course, free to believe whatever you want but for me, personally, I no longer trust what they say re FSD/AP and instead rely on what they deliver.

It appears they haven't made (deliverable) headway on bringing their vision-only (radar-free) cars back to the level of cars they were selling six months ago despite their assurances otherwise.

 

[Knightshade]

They've been saying FSD is just around the corner for years now.

Who is "they"?

Outside of Elons broad statements (which have been optimistic about everything for years longer than FSD existed) I'm not seeing much about real promised deadlines.

They said going to video-only would improve performance of the system but it got demonstrably worse

Unclear what you are talking about here- do you mean the BEV change a while back? Because it got better then.... Or do you mean the vision-only one, becuase that JUST happened a few weeks ago.

They said/implied these issues would be fixed in a few weeks. They were not.

Which issues? Again it's only BEEN weeks for the vision-only cars.

It appears they haven't made (deliverable) headway on bringing their vision-only (radar-free) cars back to the level of cars they were selling six months ago despite their assurances otherwise.

Again this claim is puzzling. They were selling radar cars not much more than six weeks ago, not months.

Beginning with deliveries in May 2021, Model 3 and Model Y vehicles built for the North American market will no longer be equipped with radar.

That they might take a little longer than they think to reach parity wouldn't be surprising... indeed we've already seen that with the original AP1->AP2 change.

But they did eventually reach, and then eventually far exceed, system parity.

Here, there's a lot less loss of features initially, so I would expect them to fill that gap far quicker

 

[jabloomf1230]

The AK presentation is worth watching although he doesn't reveal anything new that hasn't been discussed here before. That said, Tesla's pure vision approach will either sink the company or dominate the vehicle AI market. Only Tesla has an immense fleet of vehicles, millions of miles of videos and a trained NN supercomputer.

If the approach can't be made to work, Tesla has no fallback option like LIDAR and HD maps. In the case of failure, other vehicle AI approaches will eventually dominate the market. However if the pure vision approach does work, Tesla will end up licensing it to other manufacturers.

As to the radar issue, there seems to be an undercurrent of thought here that the more sensors, the better the AI will be. Radar in Teslas was basically used for training the vision system. As soon as the NN was trained sufficiently, it no longer needed radar to make decisions. Even though the future FSD code may not use radar to control the car, radar will probably continue to be used for training the NN.

 

[emmz0r]

Very cool with their efforts, yet I am disappointed that they still are working on elementary things like velocity and distance.
"This new stack IS the way" - I think I've heard that 2-3 times already.
Hoped for "yeah we are just finetuning now."

 

[RobDickinson]

Humans are able to move their heads and they have access to other senses too. Maybe someone's done tests of people driving with their heads in a fixed position and their ears muffled but I would not volunteer to participate because it seems dangerous.

Historically, Tesla has been wildly over-optimistic about FSD.

Deaf people drive. The only sense (legally) required in most places for driving is vision, and we cant look everywhere at once. sure we can move our heads but thats only looking in 1 direction from within a fixed location (within a few inches).

And yes Elon has been wildly optimistic, its somewhat a strategy to push progress. Any general self driving system available to the public before 2024-25 has done incredibly well IMO.

 

[Daniel in SD]

The AK presentation is worth watching although he doesn't reveal anything new that hasn't been discussed here before. That said, Tesla's pure vision approach will either sink the company or dominate the vehicle AI market. Only Tesla has an immense fleet of vehicles, millions of miles of videos and a trained NN supercomputer.

If the approach can't be made to work, Tesla has no fallback option like LIDAR and HD maps. In the case of failure, other vehicle AI approaches will eventually dominate the market. However if the pure vision approach does work, Tesla will end up licensing it to other manufacturers.

As to the radar issue, there seems to be an undercurrent of thought here that the more sensors, the better the AI will be. Radar in Teslas was basically used for training the vision system. As soon as the NN was trained sufficiently, it no longer needed radar to make decisions. Even though the future FSD code may not use radar to control the car, radar will probably continue to be used for training the NN.

With high resolution RADAR or LIDAR you can directly "see" the side of a semi truck trailer so you don't need AI to tell you it's there. It doesn't make AI better, it makes you less dependent on it.
Tesla's vision only approach will work fine for ADAS and will not sink the company (The new Honda system also dumps RADAR and is single camera. I assume they're using Mobileye so we'll probably see more manufacturers going vision only too). Not achieving FSD with the current hardware suite will also not sink the company, no one else is selling cars that will achieve FSD either.

 

[linux-works]

If the approach can't be made to work, Tesla has no fallback option like LIDAR and HD maps. In the case of failure, other vehicle AI approaches will eventually dominate the market. However if the pure vision approach does work, Tesla will end up licensing it to other manufacturers.

if tesla can show that pure vision can be made to work and that extra sensors don't justify themselves in complexity or cost, then it would be an industry POC (proof of concept). that would be useful and great to have. so far, there is no concensus on what the solution is to level5. lots are trying, but no one really knows yet since we are not even close to 'there' yet.

but one thing I'm pretty sure of, no one will want to pay for tesla's code base and the only way it would work is to fully blackbox it (like mobileye does), and that means integrating so much of tesla's tech into YOUR car. I just dont see that ever happening. and everyone else uses other internal tech (autosar, for many, which I think tesla does not follow) and so others taking on tesla tech is not really worth the effort.

 

[Knightshade]

but one thing I'm pretty sure of, no one will want to pay for tesla's code base and the only way it would work is to fully blackbox it (like mobileye does), and that means integrating so much of tesla's tech into YOUR car. I just dont see that ever happening. and everyone else uses other internal tech (autosar, for many, which I think tesla does not follow) and so others taking on tesla tech is not really worth the effort.

What's the alternative though?

In a world where Tesla has legit L5 driving on their cars- why is anyone buying a dodge that doesn't?

At that point dodge can either license Teslas tech... or spend their own 5+ years of nobody wanting their cars trying to reverse engineer what Tesla did and eventually building enough of their own fleet data to do it.


(and you can replace Tesla with basically anybody who gets to L5 in a way not easily/cheaply copied and same applies)

 

[daktari]

Karpathy says they believe in incremental development. Many dedicated NNs that can be adjusted independantly.
They had a few petabytes of data and had to train the NN for velocity and distance 7 times (7 cycles in shadow mode) on the 5th biggest supercomputer in the world to get here. They also need this big fleet and they need auto labelling. Took them apprx 5 months. (but one need training data from a whole calendar year probably). Certainly an expensive approach.

What was not told was how many NN branches (the right term?) do their stack run? How many must be retrained to get to FSD? How many new ones do they need? How many are complete? 5 months each x 10 new NNs = FSD finished?

I guess we also know now why they run test with Lidars - he said they used radar to benchmark during this process.

 

[RobDickinson]

AFIK the cars run multiple NN on both cpu's and for many things they use a consensus method between 3+ different NN on the same issue

 

[ZeApelido]

IMO, "blow our minds" is becoming a tired cliche at this point. We are told every release will "blow our minds".

The truth is that "blow our minds" is subjective. What might impress one person, might not impress another person the same way.

Also, "blow our minds" tends to go away when the novelty of the feature wears off. I will admit that smart summon, nav on AP, traffic light control, blew my mind the first time I used the features. But that was only because they were brand new and I'd never seen anything like it before. Once I used the features for awhile and started seeing some of the shortfalls, the features did not "blow my mind" anymore. I think the same thing will happen with V9. Yes, it may blow our minds the first time we use it, only because it is new and something we've never experienced before. But once the novelty of it wears off and especially when we experience safety interventions, it won't blow our minds anymore, or at least not as much.

Fancy simulations blow @diplomat33 and @Bladerskb minds...

 

[RobDickinson]

TBH a steady increase in capabilities that dont blow our minds is the best way to get to where we want to be.

 

[diplomat33]

Fancy simulations blow @diplomat33 and @Bladerskb minds...

Simulations don't blow my mind. Read world driverless blows my mind.

 

[scottf200]

He also gives some examples of why Tesla dropped radar. Basically, Tesla's radar and sensor fusion sucks so Tesla decided to just drop it rather than try to fix it.

He said that vision was 100 times more accurate than the radar sensor. That's huge. (~8:17:00 had 3 examples comparing vision only vs fusion of both -- see graphs) And pointed out examples of where radar would give noise (bridge, other overhead, signs, ?) --- I interpreted this to be the phantom braking problems. Keep in mind that they had this issue on TWO different radar components (Bosch and Continental).

Below via: Undocumented – TeslaTap

Radar Transponder – Tesla uses a 77 GHz radar transceiver (from an article by Ron Freund in Jan-2015 Current EVents). The first radars likely used the NXP’s MR2001 chipset or similar and the chip consumes a negligible 2.5W when operating. The Radar system is present in all cars built after late-September 2014 and until Fall-2017 were sourced from Bosch.

Vehicles with AP2.5/3.0 hardware, including all Model 3 and Model Y cars, now use a radar assembly from Continental, the ARS410 CV, similar to the ARS404-21 or ARS408-21. It consumes under 7W typical, with a peak of 12W.

Stats:

 

[diplomat33]

He said that vision was 100 times more accurate than the radar sensor. That's huge. (~8:17:00 had 3 examples) And pointed out examples of where radar would give noise (bridge, other overhead, signs, ?) --- I interpreted this to be the phantom braking problems. Keep in mind that they had this issue on TWO different radar components (Bosch and Continental).

Yes, I believe those numbers. But keep in mind that Tesla is comparing their vision to really poor radar. Tesla is not comparing their vision to the best radar.

The fact is that plenty of AV companies use radar and do sensor fusion, with a lot more sensors than Tesla, and it works great, and they don't have the radar issues that Tesla was having. So the problem is not sensor fusion itself, the problem has to be with how Tesla was doing sensor fusion.

 

[scottf200]

Yes, I believe those numbers. But keep in mind that Tesla is comparing their vision to really poor radar. Tesla is not comparing their vision to the best radar.

The fact is that plenty of AV companies use radar and do sensor fusion, with a lot more sensors than Tesla, and it works great, and they don't have the radar issues that Tesla was having. So the problem is not sensor fusion itself, the problem has to be with how Tesla was doing sensor fusion.

Re: Best radar --- That seems confusing to me. I pointed out they have used two different radar components from well-known companies. I get the feeling you are implying 'cheap' outdated radar.

Re: how Tesla was doing radar --- That is puzzling too. Given all they've done with their software and level of tech folks they have it is very hard to believe that if it was reasonable to do that they would have figured it out. I'm not buying that they just couldn't figure it out well enough. I think that other companies may be doing it with more specific environments and more limited use cases ... Tesla's goal is that you can pick the car up and let 'FSD' work 'whereever' you drop the car. In other words, a more generic approach (plus at higher speeds).

I wholely admit I wish they would keep radar for weather reasons. I've driven through some harsh rains and even in the past two weeks in the 'mountains' of VA that made me glad to have radar seeing farther ahead than my eyes (and thus the Tesla cameras could see -- even with wipers going max speed).

 

[drtimhill]

Also, "blow our minds" tends to go away when the novelty of the feature wears off. I will admit that smart summon, nav on AP, traffic light control, blew my mind the first time I used the features. But that was only because they were brand new and I'd never seen anything like it before. Once I used the features for awhile and started seeing some of the shortfalls, the features did not "blow my mind" anymore. I think the same thing will happen with V9. Yes, it may blow our minds the first time we use it, only because it is new and something we've never experienced before. But once the novelty of it wears off and especially when we experience safety interventions, it won't blow our minds anymore, or at least not as much.

True, but also true of any novelty, especially a technology one. Smart phones. Flat-screen TVs. HDTV. Heck, even the good old CD in its day. The true test of any technology is: do people keep using it once the novelty wears off?