diplomat33
Average guy who loves autonomous vehicles
One problem with using MMLLMs for autonomous driving is that the entire model is too large to work on the computer in the car. So Ghost is proposing a hybrid approach where a smaller specially trained models would be in the car for driving and the MMLLM would be hosted in the cloud. The car could prompt the MMLLM in the cloud if it needs help.
Source: Harnessing the Power of Multi-Modal LLMs for Autonomy | Ghost
I feel like this approach risks having the same issues that Cruise had since the car needs an internet connection. So the car could stall if it needs help and can't connect to the cloud for assistance. This approach seems to just replace human remote assistance with AI remote assistance.
Implementing MLLMs for driving requires a new architecture, as today’s MLLMs are significantly too large to run on embedded in-car processors. A hybrid architecture is required, where large-scale MLLMs running in the cloud collaborate with specially trained models running in-car, splitting the autonomy task and long-term vs. short-term planning between car and cloud.
Source: Harnessing the Power of Multi-Modal LLMs for Autonomy | Ghost
I feel like this approach risks having the same issues that Cruise had since the car needs an internet connection. So the car could stall if it needs help and can't connect to the cloud for assistance. This approach seems to just replace human remote assistance with AI remote assistance.