I am a bit skeptical, but then it is a lot about semantics. Elon said Dojo was FPGA on his tweet.
Dojo will have two components that both are very important.
1. Generate a huge well annotated dataset
2. Train a neural network on the annotated dataset
1. Will require vasts amount of video processing and use neural networks for inference. The advantage of the 4D labelling is that you can use future data to label current timestep, for example use the side camera 4s into the future to label where the lane marking 100m in front of the car is in 3D space. You can also predict where an occluded car is based on where the radar said it was 1s ago and where it will be in 1s according to the cameras. Here some form of Kalman filter will be useful in combination with the neural network predictions. Also some for of Human labelling and likely some form of AI to predict how a human would label to help the labellers. I think designing this system is the 140+ IQ problem that I look forward to hearing the answer to in a few years and this is where the competition will struggle because there are not enough Karpathys and machine learning jedi engineers out there...
Thus 1 will require a mix of compute, a lot will be very GPU friendly and some will be pure HW3-like inference friendly, but on the server side and no need for real time batch size of one. The good news is that this is very parallelizable and it would be pretty easy to just let AWS do it for you, but maybe they would save money by using their own neural engine chips for it at some point.
2. Will require very large memory or some form of distribution, see for example this video at 6.14:
Imo Dojo is all about reducing the cost to run a build of any new code/dataset changes. Maybe they will reduce build time a bit also, but that could probably also have been achieved by throwing more money into the cloud.
I like the last video by GeoHotz where he says that using the cloud is a crutch, it’s better to build your own system as you learn to be more efficient:
Around 1.30.00 into the video he talks about their cluster
Last edited: