Neural Networks

[diplomat33]

Basically I'm trying to figure out in terms of FC, where is Tesla, what we can expect by end of the year, what kind of quality can we expect esp. vis-à-vis what we see with Waymo.

If Tesla can get to where Waymo is today in Phoenix by end of the year - but in most cities of US, that would be amazing.

The only real evidence that we have of where Tesla might be right now is the NOA on city streets "hacked" video, the FSD demo in April and Elon's testimony about his commute on the dev software. Elon even admitted that the software is not ready for the public yet. Based on that evidence, I would describe the current state of NOA on city streets as functional but flawed. Yes, it works in general but it still has some big bugs (such as the hacked video where the car oversteered in a left turn and would have hit a divider).

Now, there is 6 months before the end of the year so Tesla has time to work on the feature. We should remember that it was only 8 months between the initial release of NOA with confirmation and the release of NOA without confirmation. I think it is entirely possible that Tesla will iron out the issues in their dev software and be able to release NOA on city streets by year's end or early next year. So I do expect that by Dec-Jan, that we will get NOA on city streets that works well in most US cities, assuming good weather too. Having said, I don't think it will be as good as Waymo. But I only say that because Waymo has such a big lead in perfecting their software in the Phoenix area. I am skeptical that Tesla can go from the current work-in-progress system to Waymo's near perfect city driving in just 6 months. Of course, we should keep in mind that Tesla's machine learning is continuously improving so NOA on city streets will get better and better over time. So we should not lose hope if NOA on city streets in December is not as good as we had hoped.

One reason, I am looking forward to NOA on city streets is because it will give us our first real experience of Tesla's full "A to B" self-driving. We will have one system, NOA, that works uninterrupted on both highways and city streets. NOA on city streets AKA "Automatic City Driving" is really the last remaining Major Feature that Tesla has left to do in FSD. Don't get me wrong, that won't mean that Tesla is finished with FSD. Far from it. Tesla will still have a lot of work to do before FSD is hands-off. But at least, all the major systems will be in place to self-drive us from A to B.

This is a false dichotomy. There is not a single autonomous vehicle out there today that does not use computer vision, and they are all more or less based on a neural net technique that was first developed early this decade as part of the DARPA challenge. Lidar does not do things like recognizing lane markers, traffic lights and signs, turn signals or emergency flashers etc. which are necessary for every successful AV.

Good point. I guess I should clarify. Whereas Waymo uses a combination of LIDAR + vision + machine learning, it seems Tesla is aiming to do everything with just vision + machine learning. Yes, there are things like traffic lights where you must use vision instead of LIDAR. But there are other things like say detecting pedestrians crossing the street or detecting a stopped vehicle where Waymo can use LIDAR but Tesla uses vision. So for those parts, Waymo and Tesla are using different tools but they will still end up at the same end goal of having to figure out the right "driving policy". For example, say a pedestrian is crossing the street. Waymo uses LIDAR to see that pedestrian. Tesla uses vision+radar to see the pedestrian. But regardless of what tool they use to see the pedestrian, they both end up in the same place of needing a "driving policy" to make sure the car does not hit the pedestrian.

 

[Matias]

Yes that's what they already have now but with HW3 and new software 2.0 neural net not installed and/or activated in cars in the field yet. Demo cars in the investor event do have those but they are still in the final testing and development stage. It's real not theoretical although it's not yet released to the general public. Be a little patient I don't think the wait is going to be very long now.

You seem to have insider information about Tesla’s FSD. Aren’t you violating your NDA?

 

[mongo]

You seem to have insider information about Tesla’s FSD. Aren’t you violating your NDA?

That was all publicly available information.

 

[Eno Deb]

Good point. I guess I should clarify. Whereas Waymo uses a combination of LIDAR + vision + machine learning, it seems Tesla is aiming to do everything with just vision + machine learning. Yes, there are things like traffic lights where you must use vision instead of LIDAR. But there are other things like say detecting pedestrians crossing the street or detecting a stopped vehicle where Waymo can use LIDAR but Tesla uses vision.

Tesla does not rely on vision only either. They use vision + Radar. In fact, the capabilities of Lidar are more comparable to Radar than to computer vision, just with higher resolution.

So for those parts, Waymo and Tesla are using different tools but they will still end up at the same end goal of having to figure out the right "driving policy". For example, say a pedestrian is crossing the street. Waymo uses LIDAR to see that pedestrian. Tesla uses vision+radar to see the pedestrian.

How do you know Waymo doesn't use vision + Lidar?

 

[diplomat33]

Tesla does not rely on vision only either. They use vision + Radar. In fact, the capabilities of Lidar are more comparable to Radar than to computer vision, just with higher resolution.

True but my understanding is that Tesla uses radar differently than lidar. Tesla does not use 360 degree radar. Instead, Tesla just uses a front radar to measure the distance of a leading car. Waymo uses 360 degree LIDAR and can use it to get a real-time map of all objects in the car's environment. So the functions are different. And Tesla uses the 360 degree coverage of cameras to get a map of nearby objects. So Tesla uses the cameras to perform a similar function to how Waymo uses LIDAR.

How do you know Waymo doesn't use vision + Lidar?

I guess I don't. Maybe Waymo is using both vision and LIDAR to track an object like a pedestrian. But Waymo's LIDAR is good enough to do that without camera vision. In other words, Waymo can track a car or a pedestrian with just LIDAR.

 

[Eno Deb]

True but my understanding is that Tesla uses radar differently than lidar. Tesla does not use 360 degree radar. Instead, Tesla just uses a front radar to measure the distance of a leading car. Waymo uses 360 degree LIDAR and can use it to get a real-time map of all objects in the car's environment.

Yes, but there is no law that says Lidar has to cover 360 degress. For example, the Audi A8 has a front-only Lidar. Conversely, some cars have side- and/or rear-facing Radar.

So the functions are different. And Tesla uses the 360 degree coverage of cameras to get a map of nearby objects. So Tesla uses the cameras to perform a similar function to how Waymo uses LIDAR.

From what they have shown publicly Tesla does not currently have the capability to create an accurate 3D map of the car's surroundings in real time just using the cameras. To the best of my knowledge nobody does. They can only recognize certain objects that the vision system has been trained to recognize.

I guess I don't. Maybe Waymo is using both vision and LIDAR to track an object like a pedestrian. But Waymo's LIDAR is good enough to do that without camera vision. In other words, Waymo can track a car or a pedestrian with just LIDAR.

They all use sensor fusion. For example, Waymo can recognize if other vehicles have their turn signal set, which is only possible if they use some combination of vision and Lidar to detect the vehicles.

 

[Daniel in SD]

I guess I don't. Maybe Waymo is using both vision and LIDAR to track an object like a pedestrian. But Waymo's LIDAR is good enough to do that without camera vision. In other words, Waymo can track a car or a pedestrian with just LIDAR.

You can't use LIDAR to track pedestrians in many cases. Look at this example of the man walking next to the construction area. When you have someone walking very close to other object it's very difficult to "see" them using LIDAR data, you have to use cameras and neural nets.

The presenter (who is a Cruise's computer vision lead) talks about it at 30:00.

 

[diplomat33]

You can't use LIDAR to track pedestrians in many cases. Look at this example of the man walking next to the construction area. When you have someone walking very close to other object it's very difficult to "see" them using LIDAR data, you have to use cameras and neural nets.
View attachment 421342
The presenter (who is a Cruise's computer vision lead) talks about it at 30:00.

Thanks!! I was not aware that LIDAR had issues in those instances. That probably explains why Tesla is all in on vision + machine learning.

 

[electronblue]

@diplomat33

You do realize a Waymo has around 18 cameras in 9 camera modules and does far more visual recognition that a Tesla.

They can for example recognize the intent of a pedestrian (movement-wise) from their limbs. Their cameras are also far superior to Tesla’s.

All this talk about Lidar vs. vision is just nonsense. Absolutely nobody is doing Lidar only. They are doing sensor fusion.

 

[electronblue]

"Driving policy" is a term used in the non-machine learning world. You don't set "driving policies", at least not all of them, with deep learning. Notice Fridman said deep learning is not "explainable"? You train the neural net with huge amount of data then it will do its own things to improve. You will know if it works or not but not how and why. That's the reason why Elon and few others said AI poses a threat to humanity.

What you get right is the theoretical promise of ”Software 2.0”.

Where you are completely mistaken is your belief that Autopilot/FSD is being developed this way. It is totally a combination of lots of algorithms (or ”software 1.0”) and a set of separate neural networks (the deep learning or ”software 2.0” part) and looks to remain that way for a long time.

 

[diplomat33]

@diplomat33

You do realize a Waymo has around 18 cameras in 9 camera modules and does far more visual recognition that a Tesla.

They can for example recognize the intent of a pedestrian (movement-wise) from their limbs. Their cameras are also far superior to Tesla’s.

All this talk about Lidar vs. vision is just nonsense. Absolutely nobody is doing Lidar only. They are doing sensor fusion.

I get that. But I don't think the LIDAR controversy is really LIDAR or Vision. It's not one or the other. Clearly, vision is requires for things like traffic signs and such. The LIDAR controversy is more about whether LIDAR should be included in the sensor fusion package. Everybody does sensor fusion, even Tesla. But many companies include LIDAR as part of their sensor fusion package whereas Tesla does not. So the question is whether LIDAR should be included or not.

 

[electronblue]

@diplomat33 I think the controversy is in the claim that having Lidar in the mix somehow is ”doomed” instead of one approach amongst many possibilities. I beleive Lidar in no way limits your possibilities because all it does is give a baseline of surrounding surfaces and their distances that is complementary to vision and radar with certain unique benefits. It is a tool and many mixes of tools can get you to the goal.

 

[Saghost]

From what they have shown publicly Tesla does not currently have the capability to create an accurate 3D map of the car's surroundings in real time just using the cameras. To the best of my knowledge nobody does. They can only recognize certain objects that the vision system has been trained to recognize.

So what is it they are showing and discussing here? Edit: TMC ate the time code and I can't seem to fix it. 2:17:00 in the linked video

At 2:19:40 while they're spinning the 3D constructed map from their 6 seconds of camera only data, he explicitly says they're applying the same techniques on a slightly more sparse version in the car.

And here, with the vector space debug tool they "routinely use" on the data the car provides: Edit: Lost this time code, too. 2:53:10 or so.

 

[EVNow]

From what they have shown publicly Tesla does not currently have the capability to create an accurate 3D map of the car's surroundings in real time just using the cameras.

Karpathy talked and showed that at autonomy day. If you see his Twitter feed there are some others doing that as well, including Google.

Tesla, TSLA & the Investment World: the 2019 Investors' Roundtable

 

[Eno Deb]

So what is it they are showing and discussing here? Edit: TMC ate the time code and I can't seem to fix it. 2:17:00 in the linked video

At 2:19:40 while they're spinning the 3D constructed map from their 6 seconds of camera only data, he explicitly says they're applying the same techniques on a slightly more sparse version in the car.

He actually said "similar" and "approximate", which is telling. Also, the demo showed a still scene. Obviously you can recover a scene using offline computation (i.e. in non-realtime). One of the most well-known examples of this are Apple's "flyover" maps which are recovered from 2D aerial photography. The trick is to do this in realtime in a comparable accuracy and resolution as Lidar, which I have never seen demonstrated (and I follow this field quite closely). If Tesla was able to do this, they wouldn't crash into parked Firetrucks.

 

[Inside]

There is not a single autonomous vehicle out there today that does not use computer vision

Not true. While in Las Vegas last month I rode is a self-driving Lyft car (made by Aptiv). It uses only lidar & radar. No cameras, no ultrasonics. I took 6 different autonomous rides, in 6 different cars, during my week out there. It drove perfectly, 100 times better than any Tesla. It handled jay walkers, merging, red lights, left turns, being cut off, etc. I suggest everyone who is interested in this sort of thing should try it next time you are in Vegas. Just use your normal Lyft app.

p.s. They can get away with no cameras because Las Vegas is a "smart city" and the traffic light status (red light, green light) is computerized and the car can poll the status remotely. I seem to recall being told that most signal lights out there also have a radio transmitter that broadcasts the signal's ID# as well as its current status. The cars use high-def map data and know where all the limit lines are and where all the lane lines are, and which lanes are turn-only lanes, etc.

 

[Eno Deb]

Not true. While in Las Vegas last month I rode is a self-driving Lyft car (made by Aptiv). It uses only lidar & radar. No cameras, no ultrasonics.

The Aptiv cars actually have a trifocal camera.

 

[Daniel in SD]

So what is it they are showing and discussing here? Edit: TMC ate the time code and I can't seem to fix it. 2:17:00 in the linked video

The question is if it's an "accurate 3D" map. We know that what's in the released software cannot create an accurate 3D map. It needs to be extremely accurate in order to be suitable for an autonomous vehicle.
I also haven't been able to figure out how stop TMC from stripping time codes.

 

[Saghost]

The question is if it's an "accurate 3D" map. We know that what's in the released software cannot create an accurate 3D map. It needs to be extremely accurate in order to be suitable for an autonomous vehicle.

How accurate does it need to be? Obviously, it needs to have all the correct objects in it, and they need to be correctly identified and at least approximately located correctly in 3d space.

But how precise do you really need the geometry and scaling to be?
Why do you believe you need that particular level of precision - no more and no less?

Off the top of my head, it seems like knowing there's a stopped car ahead is important, and knowing it's half in your lane is vital, but knowing it's 312.3 feet in front of you instead of ~300-350 feet probably isn't important.

Relations between the elements seem critical, but exact placements and scales probably aren't except things right near you.

Also, how do we know the current software cannot create an accurate map? You seem quite confident of that, but I haven't seen anything one way or the other that I know of.

 

[Daniel in SD]

How accurate does it need to be?
Also, how do we know the current software cannot create an accurate map? You seem quite confident of that, but I haven't seen anything one way or the other that I know of.

My guess is it needs to be accurate to within about a foot to avoid hitting curbs and jersey barriers.
There are plenty of videos and accounts of autopilot hitting things that it wouldn't if it had an accurate 3D map. Curbs, jersey barriers, trucks, off the top of my head.