Game AI approach to help autonomous cars to avoid incidents

[kurdakov]

Found and decided to share one interesting approach to increase AI effectiveness which is also potentially applicable to autonomous cars

Next Big Future: Artificial Intelligence Agent outplays human and the in Game AI in Doom Video Game
( the paper is here [1609.05521] Playing FPS Games with Deep Reinforcement Learning )

on autonomous cars authors say:

"the deep reinforcement learning techniques they used to teach their AI agent to play a virtual game might someday help self-driving cars operate safely on real-world streets and train robots to do a wide variety of tasks to help people"

indeed, this approach learns only from 'what it observes' in 3D and still outperforms human players. So
if there are several trained networks (as proposed ) which allow both navigation and taking actions using neural network memory, and those networks make better decisions than human driver can do, then safety of autonomous cars might increase and become much better than we could expect even from experienced driver.

 

[kurdakov]

btw while fleet learning helps Tesla autopilot a lot, it might be a good idea, having huge amount of data ( what vehicles 'see' on the road ), to create a vehicle simulator which one one hand - simulates a tesla car with the use of real data and decisions being made by autopilot and reproduce those strange AP decisions which users report here on forum and fix them

Then in would be also possible to create new dangerous situations in addition to what was viewed by real cars.

Then Tesla AP might progress much faster training neural networks in this semi game like simulator.

as for creating a 3D world which is close to what vehicle sees via radar then some ideas could be taken from such projects as OSM-3D Globe. Public elevation data could be merged with street maps ( openstreetmaps is used in a link ) and then a virtual Tesla with all other data can be placed there. Another example is here osgEarth — OSGeo-Live 10.0 Documentation
Non free elevation data can be as good as 2m resolution AW3D World 3D Topographic Data - Global Digital Elevation Model

 

[Zybd1201]

btw while fleet learning helps Tesla autopilot a lot, it might be a good idea, having huge amount of data ( what vehicles 'see' on the road ), to create a vehicle simulator which one one hand - simulates a tesla car with the use of real data and decisions being made by autopilot and reproduce those strange AP decisions which users report here on forum and fix them

Then in would be also possible to create new dangerous situations in addition to what was viewed by real cars.

Then Tesla AP might progress much faster training neural networks in this semi game like simulator.

as for creating a 3D world which is close to what vehicle sees via radar then some ideas could be taken from such projects as OSM-3D Globe. Public elevation data could be merged with street maps ( openstreetmaps is used in a link ) and then a virtual Tesla with all other data can be placed there. Another example is here osgEarth — OSGeo-Live 10.0 Documentation
Non free elevation data can be as good as 2m resolution AW3D World 3D Topographic Data - Global Digital Elevation Model

This is already being done by tons of people including tesla.

 

[kurdakov]

This is already being done by tons of people including tesla.

and how do you know?

those silly errors ( like AP does not slow down when someone cuts from another lane and only few inches are left just in front of Teslas ) or strange behavior near big trucks could be learnt from simulations and eliminated - but still those errors persist

more: that approach to use depth buffer like in article with VizDoom is quite innovative and was not used much earlier ( at least in game agents training )

so I'm sure many companies use simulators for engineering ( I made some simulations for VW back in early 2000s working online for small german research firm ) but still - it does not look that those possibilities are used for full extent because somehow those errors which users report could be caught in well thought in prepared simulations - but they are not caught and eliminated

 

[kurdakov]

This is already being done by tons of people including tesla.

another take on this.

up to version 8.0 radar was a secondary detection device which did not produce '3D' info - rather a 3D cloud. Temporal smoothing which provides something like 'object contours restoration' was announced just in this summer.

Thus - there were no reasons to make simulations with 3D information like depth buffer - that was not applicable to what Tesla 'seen' via camera and radar.

And just a visual simulation to get some parameters tuned - is quite different - it could not be such helpful as it is possible now: merging radar 'smoothed 3D world representation' and nearly identical representation from depth buffer of 3D rendering with detailed elevation data and merged streets and houses from vector maps. Now - it is really possible to tune actual radar AI with game engine.

While with previous approach ( third party camera with proprietary detection algorithm - which just output detection now saying how and radar which provided just secondary data about surrounding world ) that would be almost useless to tune actual AI onboard of real Teslas with computer renderings and simulations - that could provide some design hints, but not actual tuning of real device.

 

[kurdakov]

just've read Electrek article Tesla comings and goings: video game programmers, an Israeli tank designer, automation experts and more

apparently, now Tesla develops simulator.

 

[JeffK]

just've read Electrek article Tesla comings and goings: video game programmers, an Israeli tank designer, automation experts and more

apparently, now Tesla develops simulator.

They've always done it that way...even for AP1.

For reference you can check out any talk give by Tesla or MobilEye on the subject. How else do you think AP1 got better with time?

Google employs the same types of learning and simulation, but Tesla has far more real world data. Elon actually described the process they go through for validation a short while ago during a call if I remember correctly.

There are various machine learning techniques used in Autopilot.

 

[kurdakov]

For reference you can check out any talk give by Tesla or MobilEye on the subject.

ok, would be glad you to point to any talk, where game simulation is mentioned.
Note - "various machine learning techniques" are not necessary used via game-like simulations. It is quite straightforward to use machine learning and not using additional game-like simulations - ex just by gathering data and training neural nets with data

 

[JeffK]

ok, would be glad you to point to any talk, where game simulation is mentioned.
Note - "various machine learning techniques" are not necessary used via game-like simulations. It is quite straightforward to use machine learning and not using additional game-like simulations - ex just by gathering data and training neural nets with data

Deep-Q is machine learning ... did you have another source for non machine learning game-like simulations?

You need to be more specific about "game-like simulation"...
Do they have and use simulation? yes, and so does Google, MobilEye, and everyone else.

 

[kurdakov]

Deep-Q is machine learning ... .

Deep Q learning is subtype of Q-learning and is a machine learning algorithm

Q-learning - Wikipedia

so what is needed here - is data - 'event' 'action' - and fleet learning ( in Teslas ) uses such types of algorithms, no question

But - you say, that what I proposed and now is developing in Tesla ( after new hiring ) - specifically training not via using real Teslas, but using virtual simulated Teslas in game like environments - like - rendering cities (see links above ) with simulated Teslas and fed with some gathered real data was done before.

So I ask you to provide with hints that 3D game simulations were extensively used before
for training Tesla autopilots.

I'm sure that learning algorithms, some offline post processing with some visualizations were used before.
But for me it looks like much more extensive use of simulations with 3D starts only now, in a way I proposed in above posts and links.

I think it is important step, which will make autopilot development much faster,

so not sure why you try to insist that 3D gaming simulations were used extensively before and nothing interesting to see in those new hiring

 

[JeffK]

Deep Q learning is subtype of Q-learning and is a machine learning algorithm

Q-learning - Wikipedia

so what is needed here - is data - 'event' 'action' - and fleet learning ( in Teslas ) uses such types of algorithms, no question

But - you say, that what I proposed and now is developing in Tesla ( after new hiring ) - specifically training not via using real Teslas, but using virtual simulated Teslas in game like environments - like - rendering cities (see links above ) with simulated Teslas and fed with some gathered real data was done before.

So I ask you to provide with hints that 3D game simulations were extensively used before
for training Tesla autopilots.

I'm sure that learning algorithms, some offline post processing with some visualizations were used before.
But for me it looks like much more extensive use of simulations with 3D starts only now, in a way I proposed in above posts and links.

I think it is important step, which will make autopilot development much faster,

so not sure why you try to insist that 3D gaming simulations were used extensively before and nothing interesting to see in those new hiring.

You are supposing that there MUST be graphical representation to have a 3D simulation... this is 100% false. The graphical representation is for humans, not for learning.

The game developers are probably there to develop a user interface like nvidia is demoing here:

 

[kurdakov]

You are supposing that there MUST be graphical representation to have a 3D simulation... this is 100% false. The graphical representation is for humans, not for learning.

ok I see, that you do not see any value in links in my top posts - the trick there is that without 3D representation the described type of learning is not possible, but with 3D rendering it is possible and gives huge benefits for faster and better development of autonomous navigation algorithms.

so you missed that 3D rendering might be very beneficial for learning and quite possibly those huge benefits will be reaped now

 

[JeffK]

ok I see, that you do not see any value in links in my top posts - the trick there is that without 3D representation the described type of learning is not possible, but with 3D rendering it is possible and gives huge benefits for faster and better development of autonomous navigation algorithms.

so you missed that 3D rendering might be very beneficial for learning and quite possibly those huge benefits will be reaped now

The paper is a joke. Google already did this in Feb 2016 just with a custom doom-like game for ethical reasons. They do a paper months later using the same techniques... not original work.

imagine 3D rendering but without the monitor.

Watch any MobilEye presentation or the Chris Urmson TED talk. The data sets are always mapped to 3D space. When this is done it doesn't matter if the data is real or simulated. It also doesn't have to be pretty. Here's one from google.

Another from Nvidia:

These can be from real data or simulated, again it doesn't matter. These are visualizations for humans.

Playing a video game is very different because you don't necessarily have a 360 degree field of view like a self driving car has. The neural networks use the same data used to render those images, but it doesn't look like that. It's a multidimensional dataset.

You can render it in a modern game engine if you want... but you'd be stripping all the data (real or simulated).

 

[kurdakov]

These can be from real data or simulated, again it doesn't matter.

in Google case (an image with gathered data from Lidars ) they have really good real world data, which is mapped to 3D.

now with Tesla case - they have a 2D radar which is software proceeded to give some 3D data (but they still fight to distinguish overpasses - as we seen from reports from HW2 owners ), but they also have cameras and also ultrasonic sensing - which does not reconstruct 3D - just can sense nearby objects.

here a rendered depth data can really help to overlay real data to make faster simulations and learning - because in case of Tesla - they just don't have that data which Google has.

Chris Urmson is also from Google - so he describes what lidar sees. And that is ok. but neither of current Tesla sensor has true 3D capabilities

 

[JeffK]

in Google case (an image with gathered data from Lidars ) they have really good real world data, which is mapped to 3D.

now with Tesla case - they have a 2D radar which is software proceeded to give some 3D data (but they still fight to distinguish overpasses - as we seen from reports from HW2 owners ), but they also have cameras and also ultrasonic sensing - which does not reconstruct 3D - just can sense nearby objects.

here a rendered depth data can really help to overlay real data to make faster simulations and learning - because in case of Tesla - they just don't have that data which Google has.

Chris Urmson is also from Google - so he describes what lidar sees. And that is ok. but neither of current Tesla sensor has true 3D capabilities

Radar, ultrasonics, and even cameras using optical flow and a known speed can give you depth data.

 

[brkaus]

I've assumed the game folks were there for the UI.

Not sure about the car cutting in and AP not catching it quickly as an example for a game simulator benefit. This has to be one of the easiest scenarios to recreate on a track or test environment.

 

[kurdakov]

I've assumed the game folks were there for the UI.

OK we will see if all those people which designed game mechanic internals will produce that new UI which you expect

 

[JeffK]

OK we will see if all those people which designed game mechanic internals will produce that new UI which you expect

You aren't reading the materials... check their Linkedin profiles:

Christophe Thibault worked on rendering
Jaewon Jung worked on rendering
Robert Lin created UIs

All signs point to their involvement in either customer facing UIs or UIs for internal development work.

 

[kurdakov]

Checked profiles, yes Robert Lin is a specialist in UI. and I also now think he will be working on UI

as for

All signs point to their involvement in either customer facing UIs or UIs for internal development work.

I would gladly ask to give your theory on how it follows from

"
- Created in-game render debug view modes so that developers can quickly check why rendering doesn't work as expected or which rendering stage is becoming a bottleneck
- Technically led a project creating a new major map of the game
- Implemented swaying grass for more interactive environment in the game
- Improved Fog of War rendering for the new map
- Spearheaded C++11 transition
- Worked on Local Server Manager, which is a back-end process for LoL that manages game server instances running on each game server host
- Developed a game data pipeline for a new front-end client for LoL
- Joined team working on chat system, which uses Erlang / Ejabberd / Riak, and worked on chat federation"

and
"
● An out-of-process rendering system that allows Blizzard games to display and control rich content such as web browser frames
● A video decoding/encoding pipeline used by Overwatch in order to record and play back games
● Client SDKs for Blizzard's internal telemetry system that allows all the Blizzard properties to use the same telemetry bus for logging, error reporting and data analysis
● A WebSocket library used by core Battle.net services
● Overhauled the entire World of Warcraft's in-game support pages technology with a faster and more secure browser system
"


as what I see - that both was working on saving debug rendering ( where rendering has additional information to what people view - such that is used in a very first link - where besides 3D also depth maps are generated and then grabbed for learning )

and the there is also indicated experience for logging information from 3D simulation ( telemetry logging ) which will be very beneficial for 3D technical simulations ( I had similar system when developed engineering 3D simulation system for German automakers )

though considering how few of them, there could be other possibilities like they will be working on those debug 3D visualizations we seen in promotional video ( overlays over roads, quads over vehicles, etc )
but still hope that engineering simulations which I like to develop will also appear in Tesla development pipeline someday

 

[JeffK]

Checked profiles, yes Robert Lin is a specialist in UI. and I also now think he will be working on UI
I would gladly ask to give your theory on how it follows from

"
- Created in-game render debug view modes so that developers can quickly check why rendering doesn't work as expected or which rendering stage is becoming a bottleneck
- Technically led a project creating a new major map of the game
- Implemented swaying grass for more interactive environment in the game
- Improved Fog of War rendering for the new map
- Spearheaded C++11 transition
- Worked on Local Server Manager, which is a back-end process for LoL that manages game server instances running on each game server host
- Developed a game data pipeline for a new front-end client for LoL
- Joined team working on chat system, which uses Erlang / Ejabberd / Riak, and worked on chat federation"

and
"
● An out-of-process rendering system that allows Blizzard games to display and control rich content such as web browser frames
● A video decoding/encoding pipeline used by Overwatch in order to record and play back games
● Client SDKs for Blizzard's internal telemetry system that allows all the Blizzard properties to use the same telemetry bus for logging, error reporting and data analysis
● A WebSocket library used by core Battle.net services
● Overhauled the entire World of Warcraft's in-game support pages technology with a faster and more secure browser system
"

Things they've done are human facing. I've selectively bolded items to make it more clear.

I'm not quite sure that implementing swaying grass translates to training autonomous vehicles but those skills might make an awesome UI where vehicle data coming in from sensors could be rendered for the Driver and passengers. (similar to the NVIDIA demo)