I am also thankful today for
this 30-something: LLMs FTW.
[1hr Talk] Intro to Large Language Models |
Andrej Karpathy (17 hrs ago)
Cheers to the Pilgrims (and the Pioneers!)
TL;DR of long post:
It is ~100% certain that Robotaxi FSD can be solved with Tesla's V12 approach. This fact alone should make you shake the couches.
Look at Karpathy's talk above and pay special attention to the slide at about 25:45 (see below set at proper time).
He presents the LLM scaling law which says performance of an LLM is a smooth, well-behaved, predictable function of
N, the number of parameters and
D, the amount of training data. He also mentions that
there is no known limit to this scaling law.
I see no reason the same law would not apply to Tesla's FSD models. This is something that has been talked about a lot.
Conclusion #1: There is no reason to believe that FSD V12 will reach a local maximum. V12 is the kind of model that Tesla wanted all along. It will just keep getting better and better without any new approach or technology breakthrough. (Or as Karpathy puts it, algorithmic progress is a bonus, but not necessary).
Conclusion #2: It is ~100% certain that Robotaxi FSD can be solved with Tesla's V12 approach.
Conclusion #3: It is extremely likely that Tesla will be the first to market with a Robotaxi FSD. Tesla is currently the only company that can satisfy the "D" variable. Tesla is the only one with enough quality training data. (Yes, Kathy Wood has been right about this)
Conclusion #4: It is possible that Hardware 3 will not be able to handle a model with a large enough "N", number of parameters, to do Robotaxi FSD. Hardware 4 has a better shot at it. But I think we can safely say that even Hardware 3 will eventually get a version of FSD that is very, very good at driving. This is why Elon is saying that it looks like Hardware 3 will be better than a human. Hardware 4 will be even better and Hardware 5 will be better still. This is all about the ability to deal with a larger "N".
If you have been following the progress of FSD closely, none of the above should be new. But it is nice to see how the narrative in the Tesla community comports perfectly with the LLM scaling law Karpathy is discussing.
According to the scaling law, it's not a matter of if Robotaxi FSD will get here. It's when. "When" is anyone's guess. But if you can grok what I've said above, you should gain a lot of confidence that FSD V12 is key because it promises almost infinite improvement over time. It is the most important FSD release ever. The system is now in place to solve this. It will happen.
So shake those couches folks. It's very likely that FSD is fixin' to get really, really good.
(Disclaimer: While I do have a degree in computer science, I am not an AI expert. I'd love to get feedback from those who know this stuff better than I do. Am I right?)
