FSD v12.x (end to end AI)

[DrChaos]

It’s a lot easier to replace a project manager or software QA with an AI agent

If developers think administrative scutwork like filling out all the JIRA tickets to someone else's satisfaction is bad now, meeting all the data input requirements of the "AI project manager" is going to be an even deeper circle of hell.

And people will ignore the AI project manager even more than a human one. So no I dont think they can be effectively replaced as it is in reality adding more off-task labor to the people doing the project rather than taking off labor and removing roadblocks.

 

[rentierparasit]

I don't see it being a stretch a general purpose robot replacing those specialized ones,

Hi, StopCrazypp --

This is at the core of the debate. If you've got a burger flipping robot (machine?) and fry-cooking robot (machine?) what do you need the general purpose robot for? Is the theory that broader adoption will lead to increased production leading to economies of scale that allow a GPR to be produced for less than SPR? Maybe! But you'd have to look at that on a case-by-case basis. And note there's a good bit of circularity here; what are the tasks that require a GPR that are so numerous to lead to economies of scale sufficient to drive out SPRs?

Honestly, I think what drives a lot of the interest in humanoid GPRs is the idea of having a personal servant. It's obviously impractical to have an omelette making robot, and a dishwashing robot, and a putting-away-the-groceries robot, there just isn't enough room in the kitchen, but easy to imagine a humanoid GPR that was capable of all them. The difficulty you run into there is the AI. The most recent version of the Turing Test I've heard is, "Go into a strange apartment and make a pot of coffee." Thanks to abundant promiscuity, I've done that, but sometimes it's not easy!

Yours,
RP

 

[spacecoin]

If developers think administrative scutwork like filling out all the JIRA tickets to someone else's satisfaction is bad now, meeting all the data input requirements of the "AI project manager" is going to be an even deeper circle of hell.

And people will ignore the AI project manager even more than a human one. So no I dont think they can be effectively replaced as it is in reality adding more off-task labor to the people doing the project rather than taking off labor and removing roadblocks.

I agree that PMing is a complex job, but it was merely an example. My point is that intelligent software agents are likely coming way before humanoid robots. Everyone thinks someone else’s job will go first though. If AGI happens, white collars will likely be the first to go. Moravec’s paradox.

I don’t think there will be single general purpose (humanoid) robot doing the equivalent complex* work of a human in the coming 15 years.

*) that you couldn’t automate today using a special purpose robot.

 

[spacecoin]

Really for single task? For example the burger flipping as the example was given up thread? What about cooking fries? There are specialized robots that already doing that today:
AI robots are making burgers and fries at this new restaurant

It's a $30,000 unit that was reported on a few years ago (it actually can do many different items all in one unit):
Flippy, the $30,000 automated robot fast-food cook, is now for sale with 'demand through the roof' — see how it grills burgers and fries onion rings

I don't see it being a stretch a general purpose robot replacing those specialized ones, as long as the UI is made easy for training it (current virtual reality controllers or even just holding its hand may make it possible to train it easier than having to manually program it, like the specialized robot likely is).

Given that you need millions of examples to train a system for a single task using today’s technology I don’t think training of a physical robot is feasible to happen on site. That’s not practical and will take years or decades using a single robot. I don’t think world models will change this meaningfully.

A generic robot is also likely to be a lot more expensive to make, train, and maintain than a specialized one. What’s the point of those legs and 12 DoF when you‘re “making burgers and fries”?

A software agent can be trained and be deployed cheaply at scale to get the RLHF feedback loop going. Not so much for robotics: Robots are expensive to make, need onsite service and hw gets obsolete quickly.

 

[sleepydoc]

And think about how much you’d save on hair nets net alone food with no hair!

Ewww! Gross! There’s hydraulic fluid in my fries!

 

[JB47394]

Is the theory that broader adoption will lead to increased production leading to economies of scale that allow a GPR to be produced for less than SPR?

Not just in the GPRs, but in the ecosystem that would spring up around them. Our lives would standardize around GPRs even more thoroughly than they have around smart phones. Instead of having an information device with us at all times, we'd have an action device.

 

[Mardak]

The answer is that controlling a vehicle is harder because one has to consider that the rest of the drivers on the road are likely to do stupid things.

I was actually asking more narrowly of what a vehicle vs humanoid robot needs to specifically control to even function correctly with end-to-end, but multiple people answered based on the larger longer-term question of actually deploying each in the real world with considerations of safety and scope. Conceptually, there can be various "levels" of capability for humanoid robots similar to driver assist vs robotaxi, but it's interesting how it seems natural to focus humanoid on specific "simpler" tasks with less real-time or safety concerns whereas it seems more expected that a vehicle can "do everything."

Perhaps Tesla working on both types of robots found some synergies in exploring end-to-end control for humanoid even for specific tasks and applied learnings to the more general driving task?

 

[spacecoin]

Perhaps Tesla working on both types of robots found some synergies in exploring end-to-end control for humanoid even for specific tasks and applied learnings to the more general driving task?

More likely than not, the main purpose of Optimus is to pump up the stock.

Read this is you missed it: FSD V12 (end to end AI)

 

[DanCar]

Given that you need millions of examples to train a system for a single task using today’s technology

No you don't.

I don’t think training of a physical robot is feasible to happen on site.

Training using simulation environment is practical and done today.

That’s not practical and will take years or decades using a single robot.

There will be much more than one robot.

 

[Mardak]

Not working in heavy rain is going to catch up with them almost anywhere

Do people think Tesla has trained end-to-end so far on clear weather data or more that heavy rain situations require significantly more variety of data to behave correctly? I experienced 11.4.9 neural networks failing for Moving objects but were fine for Lanes with what showed up as a relatively small blurry patch on the main camera:

But that got me realizing end-to-end could learn when to rely more on the wide camera vs main camera vs recent predictions if temporarily occluded by a smeared wiper pass. My defrost example is a potentially relatively simple case of a common melting pattern of which parts of the camera view are blurred to learn when to rely on other camera views, but a more general case of rain drops or patches of water moving across the views has many more possible combinations of occlusions.

Tesla talked quite a bit about dark, foggy, occluded, rainy scenarios at AI Day 2022, so they're well aware of the problem, and maybe that experience of getting it working with 11.x gives them some expectation of how much harder it will be to get working with 12.x?

 

[spacecoin]

No you don't.

Yes you do. See for example this article:
https://lamarr-institute.org/blog/reinforcement-learning-and-robotics/

"RL typically needs millions of interactions with the environment in order to converge, which is impractical for the real world"

Training using simulation environment is practical and done today.

For sure. How do you do that "on site", which was what I questioned the viability of. Also Sim2Real gap need to be handled.

But the answer according to most people in this thread is of course simple; "end to end" will solve all problems.

 

[DanCar]

Yes you do. See for example this article:
https://lamarr-institute.org/blog/reinforcement-learning-and-robotics/

The article doesn't say need millions of examples for a single task. It does say with the environment, so this encompasses more than one task.
See here for few shot learning:

https://scholar.google.com/scholar?q=robot+learning+few+shot&hl=en&as_sdt=0&as_vis=1&oi=scholart

"RL typically needs millions of interactions with the environment in order to converge, which is impractical for the real world"


For sure. How do you do that "on site", which was what I questioned the viability of.

 

[willow_hiller]

More likely than not, the main purpose of Optimus is to pump up the stock.

If you said "impress his friends" or "stroke his ego," those would be plausible explanations. But I think all of the insight into Elon's thinking we have would refute pumping the stock.

He very explicitly does not care about TSLA:

https://twitter.com/x/status/1256239815256797184

And the last time the media was consistently accusing him of being greedy, he overreacted to prove a point by selling all of his property:

https://twitter.com/x/status/1256239554148724737

 

[spacecoin]

The article doesn't say need millions of examples for a single task. It does say with the environment, so this encompasses more than one task.

There is no upper bound really, but your claim is just wrong. Few shot learning doesn't get to a deployable system at this point afaik, even for object classification.
it takes a day of training (2M games) to get AlphaZero to play a 6x6 board Othello at a decent level, and that's a simple sandbox.

Robotics is never that simple unfortunately.

 

[spacecoin]

If you said "impress his friends" or "stroke his ego," those would be plausible explanations. But I think all of the insight into Elon's thinking we have would refute pumping the stock.

I don't know what the main objective is, but probably not "more money for Elon". Retain expensive talent perhaps? That humanoid isn't likely doing anything useful that motivates its form factor in 10-15 years regardless.

 

[DanCar]

There is no upper bound really, but your claim is just wrong. Few shot learning doesn't get to a deployable system at this point afaik, even for object classification.
it takes a day of training (2M games) to get AlphaZero to play a 6x6 board Othello at a decent level, and that's a simple sandbox.

Robotics is never that simple unfortunately.

You made a claim of millions of examples for a single task. You posted a reference that doesn't back up your claim. Your claim is just wrong.
> Few shot learning doesn't get to a deployable system at this point afaik
If you are suggesting I made such a claim, then you are just wrong a second time.
The link for few shot learning is just to show that there are examples of learning a single task with a few shots. Yes , there is often millions of training done before that on other tasks.
Tesla previously stated that autopilot / FSD training is applicable to the Bot. If that is true, the bot already has millions of training examples.

 

[Bladerskb]

He very explicitly does not care about TSLA:

This is ridiculously and factually not true and only something fervent fans actually believe.

 

[willow_hiller]

This is ridiculously and factually not true and only something fervent fans actually believe.

Care to explain or back up that claim?

 

[DrChaos]

Perhaps Tesla working on both types of robots found some synergies in exploring end-to-end control for humanoid even for specific tasks and applied learnings to the more general driving task?

yes they already said that the general occupancy networks and visual prediction foundation models share architecture & theory.

the problem is not perception, it's policy (what to do and when) and that's even harder for most general robotics tasks than driving, which is narrow and stereotyped and yet when you get into the weeds even that is really highly diverse.

The best application I can see for next gen robotics is in Amazon's picking and packing warehouses but they've had a robotics program for decades. And they still hire people (and treat them like droid slaves).

For that matter, what about a robotic UPS carrier? Can you get robots to pick up packages and deliver them to doorsteps effectively? Even solving that might be quite hard for a very limited zero decision zero interaction job.

 

[DrChaos]

If you said "impress his friends" or "stroke his ego," those would be plausible explanations. But I think all of the insight into Elon's thinking we have would refute pumping the stock.

He very explicitly does not care about TSLA:

https://twitter.com/x/status/1256239815256797184

The end of quarter push to get numbers and earnings, a practice spanning many years, belie inferences from one tweet.

And the last time the media was consistently accusing him of being greedy, he overreacted to prove a point by selling all of his property:

https://twitter.com/x/status/1256239554148724737

Cmon, that was to disestablish tax residency in California. When does he sell his jets?