There have been self-supervised training approaches that input and output the same modality, e.g., partial images -> full image prediction, that can train on a lot more unlabeled data, and then later the self-supervised representation can be used to train a relatively simple linear classifier to output a different modality, e.g., image -> text labels.
Phil Duan did say quality is important, but it sounds like diversity is also important especially for the world model:
Presumably not all combinations require comprehensive coverage in the training data if the networks are able to generalize from enough situation pairs. For example, driving on a straight road at night or curved road with inclement weather or turning at an intersection with heavy traffic might have many more examples than making a turn at night with snow, but ideally the relatively small number of good examples of slower-than-usual approach for the snowy turn could be enough to learn about increased caution from general wet/snowy/icy conditions.
Sim helps a lot. You can simulate the same scene in artificially created weather.
Yeah, there is a reason Waymo is avoiding places with black ice (but apparently they're working on it).
Openpilot does something like this for E2E navigation, they draw a map view with the intended driving path for the model, like this...
Yes, that's exactly what I think: success at perception, even self supervised, still leaves a large gap to the next stage of control.
I get this, and I also get that Mr. Musk is a visionary who trusts his people and what they say. What is totally cool is that he takes the heat for slipped schedules, engineering misses, and still supports his design and engineering team. That's what made Tesla, SpaceX and Starlink the successes they are. Send people to the moon scoreboard - Apollo 6 and Artemis 0. Musk is a hero to me even though I spent $10000 for "FSD" in early 2020. Bravo. Pioneers usually have a lot of arrows in their backs. Too bad Tesla bet on California for their leading edge tech manufacturing. Compare to California High Speed Rail ha ha ha ha ha ha ha ha ha. Even Newsom killed it at one point...(moderator edit)
If mods don't let us mention Tr* .... noone should be allowed to mention other politicians either.
That's an oversimplification. Elon have in the past not listened to his engineering team, for example for the radar removal (which from rumors, the engineers opposed removing, but Elon insisted).
He is different now, and worse than he used to be, when he counteracted major flaws with strengths. The flaws have amplified and strengths no longer as valuable.
Why? The manufacturing which made them where they are? Where the factory was close to the engineering design? Where they had Palo Alto engineers from 2010 on and not Detroit or Munich engineers? The ones who could write car software right? The ones who got the battery design right long before others did? (GM is still ultimaing away with pouch cells in modules)
Sure, rail costs way too much to build but that's everywhere in USA.
Notice 'auto labeling pipeline' which is not the pure vision transformer or pure end to end training from control signal exclusively back to cameras. Auto labeling pipeline is more like the current Karpathy solution.
They will get diversity in video for perception---the problem is as always the control problem.
Yes, for control, quality is more important than diversity.
This is why I think they will still need the 300K lines of policy code somewhere. If not on board, it is executing somewhere on the training simulation to produce simulated training data with an appropriately safe policy that meets external requirements.
You qualify the video -> video prediction task as a mainly helping perception and not control policy, but would you agree that for video predictions to be accurate, it probably has at least some general internalization of control related concepts? For example, if you provided video leading up to a red light vs green light, the predicted subsequent video frames could reflect slowing down vs maintaining speed even though it was not explicitly trained to output controls. Expand that to many other video prediction situations where there's a lead vehicle or not, stop signs, crossing traffic, etc. where it needs to predict video frames reflecting speed control.
I don't think you get what they're doing. At some point, in the not to distant future (or maybe right now) the program will understand what's happening the same way (or better) than a human. Sounds like a fantasy - like so many other modern marvels - but that's what's happening. The bad news is that they need to solve for AGI to get autonomy working. The good news is that they're solving AGI.
Yeah, I noticed that too and wondered if it's used for another fine-tuned output head adjacent to control. Explicit labeled training targets for lanes, objects, signals, etc. could potentially boost world model internal weights for these concepts and speed up learning, and these stronger signals could then be more useful for other downstream tasks like control policy.
