Cruise conducts 200,000 hours of simulation autonomous driving per day

[diplomat33]

"Cruise’s highly accurate simulated environments enable engineers to develop and test more efficiently. Every day we run millions of tests, totaling over 200,000 hours of “driving” a day. That’s the equivalent of one car driving nonstop for 22 years!"

It shows the power of computer simulations now. It does not replace real world testing of course but it allows you to do far more testing than any real fleet could do.

 

[CyberGus]

While impressive, the simulated environment is always going to be missing that element of unpredictable human behavior.

Google's self-driving cars are really confused by 'hipster bicyclists' and their fixies

 

[willow_hiller]

"Cruise’s highly accurate simulated environments enable engineers to develop and test more efficiently. Every day we run millions of tests, totaling over 200,000 hours of “driving” a day. That’s the equivalent of one car driving nonstop for 22 years!"

It shows the power of computer simulations now. It does not replace real world testing of course but it allows you to do far more testing than any real fleet could do.

Very interesting. I came across this article while doing some research for another thread, and I thought it highlighted how simulation learning can supplement imitation learning: Tesla’s Deep Learning at Scale: Using Billions of Miles to Train Neural Networks

"DeepMind used examples from a database of millions of human-played games of StarCraft to train a neural network to play like a human. The network learned the correlations between the game state and human players’ actions, and thereby learned to predict what a human would do when presented with a game state. Using only this training, AlphaStar reached a level of ability that DeepMind estimates would put it roughly in the middle of StarCraft’s competitive rankings. (Afterward, AlphaStar was augmented using reinforcement learning, which is what allowed it to ascend to pro-level ability. A similar augmentation may or may not be possible with self-driving cars – that’s another topic.)"

So imitation learning alone brought their StarCraft playing AI to the mid-tier, and it was only after they let the network play on simulations that it exceeded that level of play.

I think this kind of training could be most useful for teaching AI planning and driving policy behavior, after the perception system is trained. Obviously training perception on a simulated object won't be as useful as training perception on real life, but after an object is perceived, a simulation of a person running across the road should be just as useful training data for the planning network as the real thing.

 

[diplomat33]

While impressive, the simulated environment is always going to be missing that element of unpredictable human behavior.

Google's self-driving cars are really confused by 'hipster bicyclists' and their fixies

Yeah. Simulation driving cannot replace real world driving but I do think it has a lot of value nonetheless in testing your software in a lot of possible scenarios. You can run a million different scenarios and see how your FSD software would handle it which would take ages to duplicate in real world testing. Case in point, you would need to drive a car 22 years non stop just to duplicate the same amount of testing that the simulation can do in 1 day. So that is a lot of testing that simulations can provide. But real world testing is still needed eventually in order to test for those unpredictable situations.

 

[diplomat33]

I think this kind of training could be most useful for teaching AI planning and driving policy behavior, after the perception system is trained. Obviously training perception on a simulated object won't be as useful as training perception on real life, but after an object is perceived, a simulation of a person running across the rode should be just as useful training data for the planning network as the real thing.

I agree. I think simulations are more designed for solving planning and driving policy than solving perception. You need to have solved perception first. But as the Cruise video showed, you can put your car in lots of scenarios like a weird construction zone or pedestrians blocking the way, in order to improve your AI planning and driving policy.

 

[CyberGus]

Yeah. Simulation driving cannot replace real world driving but I do think it has a lot of value nonetheless in testing your software in a lot of possible scenarios...

Agreed, simulators are great at testing the software you have, not showing you the software you still need to write.

 

[Sertis]

I agree. I think simulations are more designed for solving planning and driving policy than solving perception. You need to have solved perception first. But as the Cruise video showed, you can put your car in lots of scenarios like a weird construction zone or pedestrians blocking the way, in order to improve your AI planning and driving policy.

Just train it on GTA V which has plenty of unpredictable behavior

 

[willow_hiller]

Just train it on GTA V which has plenty of unpredictable behavior

Honestly, even better if they do it online. Humans are the ultimate generator of real randomness.

 

[Sertis]

Honestly, even better if they do it online. Humans are the ultimate generator of real randomness.

Soon self driving developers will pay to embed random vehicles (horses, carriages) appearing in our MMOs, and occasionally plowing into players. Win/Win!

 

[Saghost]

The value of simulated driving will vary widely based on how well they include random and dangerous unexpected events in the simulation.

Tesla already has cars handling normal events just fine. The only reason you have to watch on AP is those unpredictable parts...

 

[linux-works]

Humans are the ultimate generator of real randomness.

the singularity (big bang) disagrees.

 

[linux-works]

Tesla already has cars handling normal events just fine. The only reason you have to watch on AP is those unpredictable parts...

so, phantom braking is now solved?

(they still have a long way to go.)

 

[FreshPrince]

I wonder how Cruise's virtual LIDAR works in real world.

 

[Saghost]

so, phantom braking is now solved?

(they still have a long way to go.)

Phantom braking is the car thinking it is encountering one of those events and trying not to get you killed.

 

[willow_hiller]

Phantom braking is the car thinking it is encountering one of those events and trying not to get you killed.

Honestly, I think what most people describe as "phantom braking" these days is actually a problem with speed limits. The car thinks you're a lane or two to the right for a second, picks up the off-ramp speed limit of 45 MPH while you're going 70 MPH and taps on the brakes.

Hopefully this particular problem will get better with actual speed-limit sign reading instead of using a speed limit map.

 

[diplomat33]

I used to have bad phantom braking with just shadows on the road or low hanging branches over the road. They are essentially gone now.

 

[Watts_Up]

"Cruise’s highly accurate simulated environments enable engineers to develop and test more efficiently.
Every day we run millions of tests, totaling over 200,000 hours of “driving” a day.
That’s the equivalent of one car driving nonstop for 22 years!"
It shows the power of computer simulations now. It does not replace real world testing of course
but it allows you to do far more testing than any real fleet could do.

Why Waymo Will Fail​

Data is the new oil. Waymo has collected ~30M real-world miles, versus 3B+ for Tesla's Autopilot.

 

[willow_hiller]

There is presumably some value in simulated training in addition to real world training; Tesla does simulation too: Autopilot Simulation, Tools Engineer | Tesla

"The foundation on which we build these elements is our simulation environment. We develop photorealistic worlds for our virtual car to drive in, enabling our developers to iterate faster and rely less on real-world testing. We strive for perfect correlation to real-world vehicle behavior and work with Autopilot software engineers to improve both Autopilot and the simulator over time."

But I do wonder how hard it is to avoid overfitting your networks. If your simulated environment has 100 different ways in which a pedestrian can run across the street, and the network learns so well to avoid a collision in those 100 different scenarios, how can you be sure that the network is actually reacting to novel events rather than just effectively memorizing the possible scenarios?

 

[diplomat33]

Why Waymo Will Fail​

Data is the new oil. Waymo has collected ~30M real-world miles, versus 3B+ for Tesla's Autopilot.

If Tesla has such a huge data advantage, how come they have not delivered "feature complete" FSD yet? With a whopping 3B miles, how come AP is still just L2? And if Waymo is so hopeless on data, how come they have L4 autonomous driving?

 

[diplomat33]

But I do wonder how hard it is to avoid overfitting your networks. If your simulated environment has 100 different ways in which a pedestrian can run across the street, and the network learns so well to avoid a collision in those 100 different scenarios, how can you be sure that the network is actually reacting to novel events rather than just effectively memorizing the possible scenarios?

Well, the video I posted from Cruise says they can run millions of tests per day. So I would guess that they do more than just 100 different pedestrian scenarios. With simulations, they can easily run thousands or tens of thousands of pedestrian scenarios.

But to try to answer your question, I think real world driving is probably how they make sure that the car can react to novel events. Basically, they can run 1M simulations, iterate on the software, and then do real world driving to see if there is anything in the real world that the car can't handle. If yes, they can go back and iterate on the software again until that case is solved.

What simulations does, is save time on this process. Because if you could only test for cases when your car encounters them in the real world, then you've have to wait awhile or maybe have a large fleet like Tesla and do billions of miles until statistically you encounter the same number of cases. But like the OP says, with simulations, you can compress the equivalent of 22 years of driving into 1 day of simulations. So with simulations, you can quickly test for the same amount of cases that it would take a large fleet years to encounter by chance.