-
Want to remove ads? Register an account and login to see fewer ads, and become a Supporting Member to remove almost all ads.
Dojo discussion
- Thread starter Discoducky
- Start date
Buckminster
Well-Known Member
Martin Viecha
Thank you. The next question is, is Dojo on track for summer 2022? And what challenges, if any, are you working through? Is Dojo necessary for FSD to operate better in cities like New York City? Or on a separate note, where should we expect the first implementation of Tesla Bots? In your factories?
Elon Musk
Okay. There's a few questions on there. Like 6 questions. Yes, Dojo appears to be on track for doing something useful in the summer this year. I think the threshold that really matters is at which point when does it become more competitive than a GPU cluster for training? And obviously, the GPU cluster is getting better. So, it's a moving target. But that's the goal I've set for the team is the FSD team running our GPU supercluster needs to tell me that they want to use Dojo instead that. That's where -- that's the obvious sort of threshold.
And I don't know when that will. I wouldn't say like success is 100% certain here. I think, we just generally want to overestimate meeting options to underestimate ourselves. But it does seem as though we might pass that threshold next year with Dojo if we execute well. Dojo is not needed for Full Self-Driving but it is a cost optimization on creating vast amounts of video data.
Cost optimization also, a rate of improvement. So, if you can train models faster, have a shorter iteration interval, then you can make progress faster. So, not everything can be distributed to deep GPUs. There's some elements of serialization there, so. And then, if Dojo is competitive, then it does seem like the kind of thing where we would offer it to other companies that want to do neural net training. Those are very much a neural net training optimized system.
But in theory, it should be better than a generalize computing platform or say, GPUs, which were not really intended for the pixel trader. [ph] It is not directly intended for optimizing training of neural networks. They just happen to work better than CPUs in most cases. So, Dojos like a giant ASIC optimized for neural net training, especially video, or video like things. But as -- like said, we're not saying, for sure, Dojo would succeed. We think it will. We would encourage those who think this is an interesting problem to join Tesla, and -- yes.
Thank you. The next question is, is Dojo on track for summer 2022? And what challenges, if any, are you working through? Is Dojo necessary for FSD to operate better in cities like New York City? Or on a separate note, where should we expect the first implementation of Tesla Bots? In your factories?
Elon Musk
Okay. There's a few questions on there. Like 6 questions. Yes, Dojo appears to be on track for doing something useful in the summer this year. I think the threshold that really matters is at which point when does it become more competitive than a GPU cluster for training? And obviously, the GPU cluster is getting better. So, it's a moving target. But that's the goal I've set for the team is the FSD team running our GPU supercluster needs to tell me that they want to use Dojo instead that. That's where -- that's the obvious sort of threshold.
And I don't know when that will. I wouldn't say like success is 100% certain here. I think, we just generally want to overestimate meeting options to underestimate ourselves. But it does seem as though we might pass that threshold next year with Dojo if we execute well. Dojo is not needed for Full Self-Driving but it is a cost optimization on creating vast amounts of video data.
Cost optimization also, a rate of improvement. So, if you can train models faster, have a shorter iteration interval, then you can make progress faster. So, not everything can be distributed to deep GPUs. There's some elements of serialization there, so. And then, if Dojo is competitive, then it does seem like the kind of thing where we would offer it to other companies that want to do neural net training. Those are very much a neural net training optimized system.
But in theory, it should be better than a generalize computing platform or say, GPUs, which were not really intended for the pixel trader. [ph] It is not directly intended for optimizing training of neural networks. They just happen to work better than CPUs in most cases. So, Dojos like a giant ASIC optimized for neural net training, especially video, or video like things. But as -- like said, we're not saying, for sure, Dojo would succeed. We think it will. We would encourage those who think this is an interesting problem to join Tesla, and -- yes.
Buckminster
Well-Known Member
Buckminster
Well-Known Member
Buckminster
Well-Known Member
Elon advised that Dojo is close to parity with GPU cluster. Looking forward to getting an update in 6 weeks.
Buckminster
Well-Known Member
Buckminster
Well-Known Member
Buckminster
Well-Known Member
bkp_duke
Well-Known Member
Anyone else notice how DOJO will squarely take the #2 spot, and be extremely close to #1, on the supercomputer rankings?
mspohr
Well-Known Member
Buckminster
Well-Known Member
Buckminster
Well-Known Member
Buckminster
Well-Known Member
Buckminster
Well-Known Member
Imo Tesla will go deep into LLMs. I was thinking about the latest GPT4 paper. At page 9 there is this image:
View attachment 926647
Then it hit me that this is not too far from what Optimus will be doing. Input the images -> compress them into some vector and use as input for the model.
Replace the user prompt with
User: Grab me that can of coke, open it and pour it into a glass
Replace the GPT4 output with:
At XYZ1 there is a can of coke that the user is pointing at
XYZ2 there is the shelf with glasses
Execute list of tasks:
1. Move closer to shelf
2. Select glass
3. Grab glass
...
That's it.
And also in order to interact with the user they need a good speech recognition. OpenAI has released Whisper which runs pretty fast:
Introducing Whisper
We’ve trained and are open-sourcing a neural net called Whisper that approaches human level robustness and accuracy on English speech recognition.openai.com
You don't need crazy offline compute for this, you can run some of these LLMs on modern laptops:
So what does this mean? I think Tesla has made upgrades to Dojo to better handle LLMs in the future:
And they will be needing these to train their models. Both their massive offline models and their destilled online models. And process so many user interactions and quickly iterate on these huge billions/trillions parameter models. And I think HW5 will have some more LLM specific architecture, not just vision batch size of 1.
Imo Tesla needs to get onto this soon or OpenAI will do with them what they did to Google with ChatGPT:
Ilya commented on this a while ago:
Basically it's all about being willing to bet big and get to scale. Before they were not ready and digital was easier, but now they are getting ready. Elon understands this and is crazy enough to try. And as Elon said, if anyone should get to AGI it's probably best if it's Tesla as they are a public company and he doesn't trust the rest:
Buckminster
Well-Known Member
I have some nvda. I was thinking of selling in the next 6 months. Maybe I will keep a little longer as they sound like they are going in the right direction.
However, it sounded like 1 chip per card opposed to 25 per tile on dojo. They are then reliant on a lot of optical fibre. Tesla are like aws here, they can lose money on dojo where this will be nvidias bread and butter soon. They won’t be able to maintain margins.
Buckminster
Well-Known Member
Similar threads
