Welcome to Tesla Motors Club
Discuss Tesla's Model S, Model 3, Model X, Model Y, Cybertruck, Roadster and More.
Register

Uber's Self-driving AI Predicts the Trajectories of Pedestrians, Vehicles, and Cyclists

This site may earn commission on affiliate links.
Read an article on slashdot and wondering if Anyone knows if Tesla is working on this?

It’s super annoying when a car crosses my lane with plenty of room and my X slams on the brakes thinking it a stationary object for a moment.

“In a preprint paper, Uber researchers describe MultiNet, a system that detects and predicts the motions of obstacles from autonomous vehicle lidar data. From a report:They say that unlike existing models, MultiNet reasons about the uncertainty of the behavior and movement of cars, pedestrians, and cyclists using a model that infers detections and predictions and then refines those to generate potential trajectories. Anticipating the future states of obstacles is a challenging task, but it's key to preventing accidents on the road. Within the context of a self-driving vehicle, a perception system has to capture a range of trajectories other actors might take rather than a single likely trajectory. For example, an opposing vehicle approaching an intersection might continue driving straight or turn in front of an autonomous vehicle; in order to ensure safety, the self-driving vehicle needs to reason about these possibilities and adjust its behavior accordingly.

MultiNet takes as input lidar sensor data and high-definition maps of streets and jointly learns obstacle trajectories and trajectory uncertainties. For vehicles (but not pedestrians or cyclists), it then refines these by discarding the first-stage trajectory predictions and taking the inferred center of objects and objects' headings before normalizing them and feeding them through an algorithm to make final future trajectory and uncertainty predictions. To test MultiNet's performance, the researchers trained the system for a day on ATG4D, a data set containing sensor readings from 5,500 scenarios collected by Uber's autonomous vehicles across cities in North America using a roof-mounted lidar sensor. They report that MultiNet outperformed several baselines by a significant margin on all three obstacle types (vehicles, pedestrians, and cyclists) in terms of prediction accuracies. Concretely, modeling uncertainty led to improvements of 9% to 13%, and it allowed for reasoning about the inherent noise of future traffic movement.”
Slashdot
 
  • Informative
Reactions: Watts_Up
Read an article on slashdot and wondering if Anyone knows if Tesla is working on this?

It’s super annoying when a car crosses my lane with plenty of room and my X slams on the brakes thinking it a stationary object for a moment.

“In a preprint paper, Uber researchers describe MultiNet, a system that detects and predicts the motions of obstacles from autonomous vehicle lidar data. From a report:They say that unlike existing models, MultiNet reasons about the uncertainty of the behavior and movement of cars, pedestrians, and cyclists using a model that infers detections and predictions and then refines those to generate potential trajectories. Anticipating the future states of obstacles is a challenging task, but it's key to preventing accidents on the road. Within the context of a self-driving vehicle, a perception system has to capture a range of trajectories other actors might take rather than a single likely trajectory. For example, an opposing vehicle approaching an intersection might continue driving straight or turn in front of an autonomous vehicle; in order to ensure safety, the self-driving vehicle needs to reason about these possibilities and adjust its behavior accordingly.

MultiNet takes as input lidar sensor data and high-definition maps of streets and jointly learns obstacle trajectories and trajectory uncertainties. For vehicles (but not pedestrians or cyclists), it then refines these by discarding the first-stage trajectory predictions and taking the inferred center of objects and objects' headings before normalizing them and feeding them through an algorithm to make final future trajectory and uncertainty predictions. To test MultiNet's performance, the researchers trained the system for a day on ATG4D, a data set containing sensor readings from 5,500 scenarios collected by Uber's autonomous vehicles across cities in North America using a roof-mounted lidar sensor. They report that MultiNet outperformed several baselines by a significant margin on all three obstacle types (vehicles, pedestrians, and cyclists) in terms of prediction accuracies. Concretely, modeling uncertainty led to improvements of 9% to 13%, and it allowed for reasoning about the inherent noise of future traffic movement.”
Slashdot

Well, Multinet uses lidar data to make its calculations. Tesla does not use lidar so they could not use this exact method. But being able to make these sorts of path predictions for vehicles, pedestrians and cyclists is an absolute necessity for any autonomous car, no matter what. So if Tesla is not doing it now, they will need to do it eventually. The only difference is that Tesla will need to use camera data to make the predictions, instead of using lidar data.
 
Well, Multinet uses lidar data to make its calculations. Tesla does not use lidar so they could not use this exact method. But being able to make these sorts of path predictions for vehicles, pedestrians and cyclists is an absolute necessity for any autonomous car, no matter what. So if Tesla is not doing it now, they will need to do it eventually. The only difference is that Tesla will need to use camera data to make the predictions, instead of using lidar data.

I can't recall. Does Waymo's Lidar (or any automotive Lidar) use Doppler shift to detection motion, like Radar? Sure, it's only toward or away, but that info could help with motion prediction. Unfortunately, something Pseudo-Lidar / Vidar can't do, but I suppose could be compensated by the high refresh rate of video (compared to the low hertz/rpm of conventional Lidar).
 
  • Like
Reactions: diplomat33
Well, Multinet uses lidar data to make its calculations. Tesla does not use lidar so they could not use this exact method.

Reading through the paper, it looks like Uber uses lidar returns to paint in voxels in a 3D volume, which they represent with a top down view. We've seen in the February Karpathy video use of Pseud-Lidar and what appears to be conventional image recognition to generate a top down view (I believe red is camera based on jiggling). If that's the case, it's possible Tesla could also do a similar vehicle prediction system like Uber's MultiNet. MultiNet does use Lidar + HD maps, so that would be a challenge, as the Tesla would need to do very accurate camera based SLAM, to localize other vehicles in their lanes using the ego car as a reference point.

As an update to my earlier post, I didn't see anything in the paper about using Doppler shift to determine movement direction.
 
Reading through the paper, it looks like Uber uses lidar returns to paint in voxels in a 3D volume, which they represent with a top down view. We've seen in the February Karpathy video use of Pseud-Lidar and what appears to be conventional image recognition to generate a top down view (I believe red is camera based on jiggling). If that's the case, it's possible Tesla could also do a similar vehicle prediction system like Uber's MultiNet. MultiNet does use Lidar + HD maps, so that would be a challenge, as the Tesla would need to do very accurate camera based SLAM, to localize other vehicles in their lanes using the ego car as a reference point.

As an update to my earlier post, I didn't see anything in the paper about using Doppler shift to determine movement direction.

If you can detect the precise position of an object at regular time intervals, then you can extrapolate a velocity vector and a path. So I don't think you necessarily need to use doppler shift. What doppler shift does is give you the instantaneous velocity in a certain direction and at a given point in time.

But definitely, Tesla will need super accurate camera based SLAM to even make it work. And by accurate, I mean probably accurate within a few centimeters. That's the accuracy of the lidar approach.
 
Read an article on slashdot and wondering if Anyone knows if Tesla is working on this?

It’s super annoying when a car crosses my lane with plenty of room and my X slams on the brakes thinking it a stationary object for a moment.

“In a preprint paper, Uber researchers describe MultiNet, a system that detects and predicts the motions of obstacles from autonomous vehicle lidar data. From a report:They say that unlike existing models, MultiNet reasons about the uncertainty of the behavior and movement of cars, pedestrians, and cyclists using a model that infers detections and predictions and then refines those to generate potential trajectories. Anticipating the future states of obstacles is a challenging task, but it's key to preventing accidents on the road. Within the context of a self-driving vehicle, a perception system has to capture a range of trajectories other actors might take rather than a single likely trajectory. For example, an opposing vehicle approaching an intersection might continue driving straight or turn in front of an autonomous vehicle; in order to ensure safety, the self-driving vehicle needs to reason about these possibilities and adjust its behavior accordingly.

MultiNet takes as input lidar sensor data and high-definition maps of streets and jointly learns obstacle trajectories and trajectory uncertainties. For vehicles (but not pedestrians or cyclists), it then refines these by discarding the first-stage trajectory predictions and taking the inferred center of objects and objects' headings before normalizing them and feeding them through an algorithm to make final future trajectory and uncertainty predictions. To test MultiNet's performance, the researchers trained the system for a day on ATG4D, a data set containing sensor readings from 5,500 scenarios collected by Uber's autonomous vehicles across cities in North America using a roof-mounted lidar sensor. They report that MultiNet outperformed several baselines by a significant margin on all three obstacle types (vehicles, pedestrians, and cyclists) in terms of prediction accuracies. Concretely, modeling uncertainty led to improvements of 9% to 13%, and it allowed for reasoning about the inherent noise of future traffic movement.”
Slashdot

Tesla has been moving to Machine Reasoning for a while now although they do not have anything in production. This is really Tesla's strength as they have access to a lot more data than few Uber vehicles, for example, can provide.