Karpathy leaves - what's next for AP/FSD?

[diplomat33]

So much disinformation in this thread and in the links here. He was not the brain behind Tesla Autopilot, he joined to replace Lattner who was a great lead for Software 1.0, but Elon saw that the projected needed a Software 2.0 approach to succeed. Karpathy‘s main contribution was improving how to balance data, how they clean the data, how to manage labelers, how to tune weight function when you run multiple tasks in a neural network and how to work as a big team training one large neural network. Basically he helped setting up the software 2.0 stack correctly.

Yes, but you seem to downplaying Karpathy's role. Everything you mention, "setting up software 2.0", was absolutely essential work. Without Karpathy, Tesla would probably not have FSD beta today. Remember that before Karpathy came in, Tesla was struggling to even do good lane keeping. Autopilot would ping pong a lot in the lane. Karpathy solved the ping pong problem. And it was Karpathy's "software 2.0" approach that put Autopilot on the path to where it is today with FSD Beta. Karpathy built the foundation that FSD Beta is built on.

 

[heltok]

Yes, but you seem to downplaying Karpathy's role. Everything you mention, "setting up software 2.0", was absolutely essential work. Without Karpathy, Tesla would probably not have FSD beta today. Remember that before Karpathy came in, Tesla was struggling to even do good lane keeping. Autopilot would ping pong a lot in the lane. Karpathy solved the ping pong problem. And it was Karpathy's "software 2.0" approach that put Autopilot and FSD on the path to where it is today with FSD Beta. Karpathy built the foundation that FSD Beta is built on.

Your hypothetical where Tesla failed to recruit Karpathy and Tesla would not have FSD today is total speculation. Tesla had many other very talented people in DL. He was great, but he was not the hero that single handedly saved FSD. Maybe he speed it up a few months, that would be a monumental achievement in itself, but saying there would be no FSD is imo too much speculation.

Solved the ping pong? I assume you refer to the “extensive rewrite”:

https://twitter.com/x/status/974704734073274368

I assume this was an extensive rewrite, with updates to the architecture, with added memory, with a better cost function for the neural network and probably also improving the control as this sounds like a control problem. Don’t think we can attribute this to Karpathy alone.

Imo Elon Musk‘s part of the development of FSD is underrated. He was the mastermind behind it, he was the one who made all cars be connected and have autopilot hardware, he was the one who made the decision to cut radar etc. Karpathy seems to admire Musk‘s intuition if you listen to Karpathy’s Robot brain or Lex interview(don’t remember which one).

I admire Karpathy, but he did not do everything himself. He was a part of a great team, did some awesome contributions, but the program would have progressed even without him.

 

[diplomat33]

Your hypothetical where Tesla failed to recruit Karpathy and Tesla would not have FSD today is total speculation. Tesla had many other very talented people in DL. He was great, but he was not the hero that single handedly saved FSD. Maybe he speed it up a few months, that would be a monumental achievement in itself, but saying there would be no FSD is imo too much speculation.

Solved the ping pong? I assume you refer to the “extensive rewrite”:

https://twitter.com/x/status/974704734073274368

I assume this was an extensive rewrite, with updates to the architecture, with added memory, with a better cost function for the neural network and probably also improving the control as this sounds like a control problem. Don’t think we can attribute this to Karpathy alone.

Imo Elon Musk‘s part of the development of FSD is underrated. He was the mastermind behind it, he was the one who made all cars be connected and have autopilot hardware, he was the one who made the decision to cut radar etc. Karpathy seems to admire Musk‘s intuition if you listen to Karpathy’s Robot brain or Lex interview(don’t remember which one).

I admire Karpathy, but he did not do everything himself. He was a part of a great team, did some awesome contributions, but the program would have progressed even without him.

This is just more downplaying of Karpathy's role, "he was not the hero that single handedly saved FSD", "he did not do everything himself", "he was part of a team", "the program would have progressed even without him". Of course, Tesla had a great team of talented engineers but Karpathy was the leader. And yes I am referring the 2018 rewrite. It was the first major update to AP to include Karpathy's software 2.0 rewrite and it showed a significant improvement.

Sure I am speculating a bit about what might have happened had Karpathy never joined Tesla but I think it is based on some logic. Before Karpathy, AP struggled, after Karpathy, AP made quick progress. Karpathy was the turning point. And if you are correct that Tesla would have been able to do AP just fine without Karpathy, why didn't Tesla do software 2.0 sooner? Why did AP2 struggle for over a year an a half to even reach parity with AP1? The fact is that Tesla was struggling with AP before Karpathy. The evidence certainly points to Tesla continuing to struggle if they had continued with a software 1.0 approach. And maybe Tesla would have pivoted to software 2.0 without Karpathy but Karpathy brought a lot of personal talent. I think it would have taken Tesla a lot longer to pivot to software 2.0 without Karpathy.

 

[BrerBear]

he joined to replace Lattner who was a great lead for Software 1.0, but Elon saw that the projected needed a Software 2.0 approach to succeed.

Latter was famously only at Tesla for a few months, compared to the 12 years he spent making his name at Apple.

Musk has a knack for burning through talent.

 

[cruiserlarry]

Maybe Karpathy felt his contribution to the job was done. Most brilliant folks I watch have a narrow focus of interest. Setting up a functional FSD architecture may have been his. Having lead the development of the 2.0 software rewrite, it may be that most of the work from here forward will now be handled by data processing with Dojo, or lower level engineers using the structure Karpathy helped to put in place. His role might have become more supervisory in nature, and that might have lead to his leaving the program. The march to FSD will continue IMO, as I agree that Tesla will continue to attract brilliant out-of-the-box talent, all who will advance the state of the program until they, too, "burn out".

 

[emmz0r]

Maybe Karpathy felt his contribution to the job was done. Most brilliant folks I watch have a narrow focus of interest. Setting up a functional FSD architecture may have been his. Having lead the development of the 2.0 software rewrite, it may be that most of the work from here forward will now be handled by data processing with Dojo, or lower level engineers using the structure Karpathy helped to put in place. His role might have become more supervisory in nature, and that might have lead to his leaving the program. The march to FSD will continue IMO, as I agree that Tesla will continue to attract brilliant out-of-the-box talent, all who will advance the state of the program until they, too, "burn out".

It's not done, it has barely started.
Either he burned out, or he got sacked.

 

[JulienW]

It's not done, it has barely started.
Either he burned out, or he got sacked.

I say more likely burned out. He is more of an academia person and living under Musk's insufferable demands on time and for unrealistic results was too much.

 

[diplomat33]

Interesting trivia. In 2012, Andrej Karpathy almost interned for Drago Anguelov, who is now the Head of Research at Waymo. At the time, Anguelov was at Google. He had a choice of interns between Andrej Karpathy to work on deep learning or Xiangxin Zhu to work on deformable models. In the end, he picked Xiangxin.

 

[DanCar]

I say more likely burned out. He is more of an academia person and living under Musk's insufferable demands on time and for unrealistic results was too much.

My guess is he couldn't meet Musk's demands so he stepped away. Not burned out, just smart and walked.

..., he joined to replace Lattner who was a great lead for Software 1.0, but Elon saw that the projected needed a Software 2.0 approach to succeed. Karpathy‘s main contribution was improving how to balance data, how they clean the data, how to manage labelers, how to tune weight function when you run multiple tasks in a neural network and how to work as a big team training one large neural network. Basically he helped setting up the software 2.0 stack correctly.
...

You seem to be well informed. Do you have insider info?

 

[enemji]

I think he left because he knows FSD advancement has hit a wall with current hardware and tech, so he got out before Elon’s false statements tarnish his reputation.

Not quite. Development is a relay racing not 100m racing.

 

[jebinc]

Not quite. Development is a relay racing not 100m racing.

Perhaps, but the relay baton is laying on the ground…. No one took the handoff from AK. He left, and no one came in prior to AK’s departure to receive the baton handoff. Can you hear it hit the ground?!

 

[DrChaos]

He'd been on leave since like March, and a lot of people (including me) kinda knew that was an exit.

Tesla burns people out, including good ones. But it also attracts almost unlimited new talent, so they get away with it. Hopefully the next guy will also be well respected and skilled.

People like Karpathy (he is much more than talent, he is deep wisdom) aren't widely available, and they definitely do not want to work for a grinder kind of company. Unfortunately after a certain amount of time, that sort of place will attract undesirable personality types who thrive on aggression, conflict and sucking up to the boss instead of nice nerds who think long and deep. Ballmer's Microsoft is a great example, and that's when Microsoft lost over and over.

 

[DrChaos]

"one stack to rule them all" is one of the biggest scam to showcase how gullible tesla fans.
Now they go around saying: "no one else has one stack".
Not understanding that others have had one stack from the very beginning.
That they didn't have to create and then nurse adas systems with lane keeping, acc and other features and maintain it to keep thousands of customers happy.
That their focus from the start has always been making fully autonomous system from day one.

MobileEye?

I think continued contact with real world drivers & deployed customers, despite all of that noise, is going to help deliver useful technology. And the value of the data so collected is substantial.

 

[enemji]

People like Karpathy (he is much more than talent, he is deep wisdom) aren't widely available, and they definitely do not want to work for a grinder kind of company. Unfortunately after a certain amount of time, that sort of place will attract undesirable personality types who thrive on aggression, conflict and sucking up to the boss instead of nice nerds who think long and deep. Ballmer's Microsoft is a great example, and that's when Microsoft lost over and over.

The problem with Ballmer waa that he was an idiot to begin with. His strategies were meaningless and as noted drove MS down till Satya took over with a convincing vision.

 

[Ramphex]

Imagine the next person comes in and says, yea... we need radar/lidar.

 

[diplomat33]

Imagine the next person comes in and says, yea... we need radar/lidar.

Elon would never hire them. He will only hire "vision-only" true believers.

 

[Krash]

Elon would never hire them. He will only hire "vision-only" true believers.

So why the new FCC radar filing?

 

[Ramphex]

So why the new FCC radar filing?

wonder how long they’ll milk the current state FSD beta… to phase out the vision only cars… to rewrite it for new cars that might so happen come equipped with radar lol

 

[diplomat33]

So why the new FCC radar filing?

wonder how long they’ll milk the current state FSD beta… to phase out the vision only cars… to rewrite it for new cars that might so happen come equipped with radar lol

The FCC radar filing could be Tesla preparing to use radar in the future. All the AV companies use HD radar for redundancy to help with seeing through rain and fog. So, I think it would make sense for Tesla to add HD radar. Interestingly, in 2016, Elon was super pro-radar per this tweet but then as we know, Tesla ditched the radar and went pure vision. Now, Elon seems very adamant about vision-only, so I think it would look like a huge flip flop if he added radar back, not to mention all the owners with vision-only cars who were told their cars would be robotaxis. LOL. But Elon probably would not care about flip flopping. My guess is that next year, he will claim that vision-only has achieved safety greater than humans just as he promised and Tesla is adding radar to achieve even greater safety for true driverless. And he knows his loyal base will go along with whatever he says.

 

[Ramphex]

The FCC radar filing could be Tesla preparing to use radar in the future. All the AV companies use HD radar for redundancy to help with seeing through rain and fog. So, I think it would make sense for Tesla to add HD radar. Interestingly, in 2016, Elon was super pro-radar per this tweet but then as we know, Tesla ditched the radar and went pure vision. Now, Elon seems very adamant about vision-only, so I think it would look like a huge flip flop if he added radar back, not to mention all the owners with vision-only cars who were told their cars would be robotaxis. LOL. But Elon probably would not care about flip flopping. My guess is that next year, he will claim that vision-only has achieved safety greater than humans just as he promised and Tesla is adding radar to achieve even greater safety for true driverless. And he knows his loyal base will go along with whatever he says.

And they sure will go along and listen to every word as gospel