But from everything I've been hearing from Tesla over the last year, compute power is the bottleneck right now. I haven't heard Tesla saying they need more clever ideas. They say the need to iterate faster. And that requires raw computation.
"iteration" usually is "tweak some parameters & recalculate". That is a problem you can throw compute at, IF you have good people tuning things. Kaparthy said, that most issues were with what you put into the mix in what percentage. I.e. tesla once had too few tunnel-images for their rain-sensor-NN.. the wipers went mad inside tunnels. Same for things like "yellow light", "cone moved in your lane by wind".
But that "raw compute" only gets you so far, as all it does is getting things nearer to the local optimum - and usually an order of magnitude more compute/tries per percent improvement.
I'm hoping that the recent layoff of human labelers is a sign that the auto-labeling effort is going well. From my understanding, auto-labeling would be a compute-intensive task? Sounds like you would know better than I.
That is not compute-intensive at all. At my last employer i wrote such an "in-the-loop"-system to get more trainings-data. The way way most expensive part is the human labor of annotating. And given you GOT such a team, then the big question is "what" to label. This ties into the previous point: What data is worth labeling to improve & make the massive compute you throw at each iteration worth it.
I was constrained by having only 2-4 "Labelers" doing this for 1h/day on 2-4 days/week. It was extracting structured information from job-ads. They managed to get ~100 Ads done per hour per labeler. To decide what was worth i devised a metric i called "confusion" that strongly corrolated with wrong answers from the net - without having to know the true answer. It would have been impossible for them to label all ~10k ads that were processed every day
Think of it like a driving-scene.. You label everything in 3d-space with 99% accuracy for the top-class, 40% for the second-best. Then you have one object there that has "only" 98% in the top-class, but 80% in the second-best class. Then you have a "confused" state, where 2 classes could be true. Real world sometimes is ambiguous. But imagine one thing being in "drivable" class (like painting of a stop-sign on the road), one being "non drivable" like a stop-sign painted on a brick wall.
These are exactly the examples you want to "discover" and "bring up" in your huge pile of data. Or - in the case of tesla - data that is not temporarily coherent. Like at the beginning of the clip you say "80% tree" but 3 seconds later "90% thing painted on van". This also is a discrepancy you want "fixed" in your data.
And with each iteration of the network it will be confused about other things. 2 steps forward, 1 step back.
Also after some time i started logging WHICH annotator did what. Because people are inconsistent. Sometimes they label the same thing differently when presented with the same data on different days (a problem that came up regarding data in my bachelors thesis -.-). Sometimes they think they understood what they should do - but don't. Then you have to throw away whole batches of data from that annotator. And go through the past data (or mark them for relabeling through others) and correct them as well.
In real-world-ML-applications you lose 90% of your hair over the data, annotation and preprocessing... not the computation, algorithms etc.. that is only the other 10%
tl;dr: Tesla laying off labelers but still not binnig FSD can be thought of as bullish, because each iteration needs less annotation to "fix errors".
Even if you have infinite compute the limiting factor then will still be annotators because humans are slow (and need to be supervised (random samples) for this -,-).. and programmers that tweak & fix things and i.e. mark stuff for annotation or select new data (like sending a request to the fleet to catch "wheel flying through the air" and similar).
To the extent that they need breakthrough ideas from Karpathy, Tesla might be more likely to get what they need with him working in an academic environment.
fully agree.