Cruise conducts 200,000 hours of simulation autonomous driving per day

[MrBadger]

When simulating, what components are being simulated and how?

At simplistic level, you could put a car in front of screen, a project the simulation. But what about radar, ultra sonics, lidar, accelerated learning - thats quite some simulation farm (8000+ parallel simulations) to fit 200,000 hours into a day if doing it in real-time? Presumably they must be simulating the sensors themselves? But then, what about rogue sensor data that affects everyday driving - rain, snow, other climatic conditions, non reflective surfaces, non obvious radar reflections etc etc etc? I can see simulation being of benefit in many clear cut cases, but its the cases where uncommon sensor behaviour that may not be able to be easily simulated where a wide variety may be needed - which may well explain some of the weird behaviour in otherwise normal looking scenarios. imho, its not the 99.9% times when things go right that is the issue, its the 0.1% when random things seem to occur that is the problem, such as phantom braking, not seeing the edge of the road or lane etc etc.

 

[malcolm]

Excellent news for simulation.

The Porn Industry will be delighted.

 

[WarpedOne]

You can run a million different scenarios

And still don't have a clue how relevant those scenarios are to the 'real world' situations.
Testing in real world is testing relevant scenarios, the real world is providing them 'for free'.

In pure simulation those scenarios are provided by whom? A committee? Random number generator? How do you know how important or relevant they are?
How do you know you are not missing a very important scenarios?

 

[Bladerskb]

While impressive, the simulated environment is always going to be missing that element of unpredictable human behavior.

Google's self-driving cars are really confused by 'hipster bicyclists' and their fixies

That's an article from 5 years ago...which is like an eternity in this field

 

[Bladerskb]

There is presumably some value in simulated training in addition to real world training; Tesla does simulation too: Autopilot Simulation, Tools Engineer | Tesla

"The foundation on which we build these elements is our simulation environment. We develop photorealistic worlds for our virtual car to drive in, enabling our developers to iterate faster and rely less on real-world testing. We strive for perfect correlation to real-world vehicle behavior and work with Autopilot software engineers to improve both Autopilot and the simulator over time."

But I do wonder how hard it is to avoid overfitting your networks. If your simulated environment has 100 different ways in which a pedestrian can run across the street, and the network learns so well to avoid a collision in those 100 different scenarios, how can you be sure that the network is actually reacting to novel events rather than just effectively memorizing the possible scenarios?

So if Tesla tells you its good, it good. If they tell you its bad its bad? No objective deduction?

But I do wonder how hard it is to avoid overfitting your networks. If your simulated environment has 100 different ways in which a pedestrian can run across the street, and the network learns so well to avoid a collision in those 100 different scenarios, how can you be sure that the network is actually reacting to novel events rather than just effectively memorizing the possible scenarios?

Your driving policy network doesn't know the difference between the real world vs simulation. So the same problem applies there. Overfitting/underfitting is always a problem in any model you are trying to create.

 

[mongo]

"Cruise’s highly accurate simulated environments enable engineers to develop and test more efficiently. Every day we run millions of tests, totaling over 200,000 hours of “driving” a day. That’s the equivalent of one car driving nonstop for 22 years!"

It shows the power of computer simulations now. It does not replace real world testing of course but it allows you to do far more testing than any real fleet could do.

So you are saying the fleet of Teslas cannot do 200,000 hours of driving a day???
With an average daily drive time of 20 minutes, that only requires 600,000 cars. Tesla made more than that just in 2018 and 2019. At a million cars on the road, they need only operate 12 minutes each to hit that 200k hour figure.

 

[diplomat33]

And still don't have a clue how relevant those scenarios are to the 'real world' situations.
Testing in real world is testing relevant scenarios, the real world is providing them 'for free'.

You can easily create simulations based on real world scenarios. Do a simulation with a pedestrian jay walking. Do a simulation with a pet that runs loose in a residential neighborhood. Do a simulation with a delivery truck that is double parked. It's not hard to come up with situations that happen in the real world.

The problem with real world testing is that it will be hard to test for a specific situation on demand. What if I want to test if my autonomous driving can handle a broken traffic light or a pet running onto the street? Just driving around aimlessly waiting for those scenarios to happen on their own could take a very long time. That would not be an efficient use of testing time. Or I can set up a test track but that also takes time and you can't test a lot of scenarios quickly. Simulations allow you to test exactly the scenario you want when you want and do it very rapidly. You can test hundreds of scenarios in a short period of time, much quicker than trying to test those scenarios in the real world.

Real world testing is important because sooner or later, you do need to make sure your software can actually handle the real world since that is where it will operate. Plus, the real world may introduce a new scenario or some randomness that your simulations did not expect. So real world testing is important but simulations are incredible valuable as well. I don't think you can develop autonomous driving in a timely manner without simulations.

In pure simulation those scenarios are provided by whom? A committee? Random number generator? How do you know how important or relevant they are?

I am sure the engineers come up with scenarios for the simulations. And you can program the computer to generate variations automatically as well.

How do you know you are not missing a very important scenarios?

You still do lots of real world testing to check for any situations that might come up that your simulation scenarios did not anticipate. That's why companies like Cruise or Waymo drive around a city for millions of miles to validate that their FSD can handle all the real world situations that might come up. And this gives them a big enough sample that they can be more confident that their FSD is reliable.

But you still need simulations to speed up the process. Because simulations can give you scenarios much quicker than real world testing.

 

[diplomat33]

So you are saying the fleet of Teslas cannot do 200,000 hours of driving a day???
With an average daily drive time of 20 minutes, that only requires 600,000 cars. Tesla made more than that just in 2018 and 2019. At a million cars on the road, they need only operate 12 minutes each to hit that 200k hour figure.

No, I never said that. My post never mentioned Tesla. My post merely presented the fact that Cruise can do 200,000 hours of driving per day. Of course the Tesla fleet can do 200,000 hours of driving per day. And we know that Tesla is able to collect specific data that they need from the fleet data. I am not bashing Tesla's fleet. It provides Tesla with invaluable data and the ability to test their software quickly in the real world to further optimize it. We've seen the progress of AP thanks to the fleet.

But I am saying that the argument that if you don't have a large fleet, you are doomed, is nonsense. There are other ways to get the same data. Case in point, with simulations, Cruise can get pretty much the same data as Tesla gets, without a large fleet. In other words, simulations allow you to get much the same data as a large fleet, without needing a large fleet.

The advantages that I do think that simulations offer over real world testing is the speed of testing and the ability to target a specific scenario on demand. Tesla has to rely on the fact that statistically, a scenario is likely to come up eventually the bigger the fleet is. So with a big enough fleet, which Tesla has, they have a high probability of getting the right data. But they are not really in control of what kind of data they get. It's whatever the fleet happens to collect based on driving preferences. And as I have said before, a lot of miles will be repeat miles that may not have any useful new information. But with simulations, I can target the specific data I need much quicker.

 

[Watts_Up]

Cruise conducts 200,000 hours of simulation autonomous driving per day

What was your point making this 'clickbait' title? Do all the other Self Driving companies, except 'Cruise', are 'retard'?

Autopilot Simulation, Tools Engineer

Autopilot is of critical importance to Tesla's mission.

It is safer, makes driving more enjoyable, and will ultimately deliver on the promise of self-driving cars.

As a member of Tesla's Autopilot Simulation team, you will be in a unique position
to accelerate the pace at which Autopilot improves over time.

The main ways in which the simulation team realizes this include:

• building tools that enable Autopilot software developers to perform virtual test drives instead of real ones
• testing all Autopilot software releases for regressive behavior
• generating synthetic data sets for neural network training

The foundation on which we build these elements is our simulation environment.

We develop photorealistic worlds for our virtual car to drive in, enabling our developers
to iterate faster and rely less on real-world testing.

We strive for perfect correlation to real-world vehicle behavior and work with Autopilot software engineers
to improve both Autopilot and the simulator over time.

Our group is a cross-disciplinary team with people from various backgrounds and fields of expertise,
like CS generalists, mechanical engineers, game developers, and computer graphics artists.

As a member on this team, you will be challenged to learn about all these different disciplines
and use your own expertise to further improve the scope and usability of our tools.​

 

[linux-works]

that sounds like a fun job. not lots of details, but you know what you are doing is going right into the driving experience of so many people.

 

[Soda Popinski]

Each has their advantages. Real-world testing can come up with edge case scenarios simulated testing cannot. Simulated testing provides precisely labeled data, for example vehicle distance to the cm, which can let a NN become more accurate with distance estimation.

I'd expect it's easier for Tesla to catch up on simulation time than Waymo to catch up on real-world miles.

 

[outdoors]

What was your point making this 'clickbait' title? Do all the other Self Driving companies, except 'Cruise', are 'retard'?

We can be better than use this word right?

Spread the word to end the word | Special Olympics Northern California​

 

[diplomat33]

What was your point making this 'clickbait' title? Do all the other Self Driving companies, except 'Cruise', are 'retard'?

It is not clickbait. I think it pretty accurately reflects the content of my post and it is factually correct. Cruise does indeed conduct 200,000 hours of simulation driving per day according to Cruise's own announcement.

My point was simply to share some news that Cruise announced. Why are you using words like "retard"? There was nothing in my post to bash other companies. I never mention other companies at all in my OP. Again, I was merely presenting news about Cruise. It seems like you are taking it personally for some reason.

 

[Bladerskb]

Why Waymo Will Fail​

Data is the new oil. Waymo has collected ~30M real-world miles, versus 3B+ for Tesla's Autopilot.

I can't take you seriously if you are listening to what that idiot has to say. who doesn't even know how many camera/radars that Tesla uses. Who wouldn't last 30 secs in a debate because he's absolutely clueless. Really? You get your info from a superfan?