Camera-Based Object Detection - The Limiting Factor to FSD

[ZeApelido]

Trying to get out of these threads complaining about missed deadlines, as I wanted to discuss more specifically the limiting factors to achieving some level of FSD. We know Tesla has chosen to focus on camera-based vision. They believe vision will have to be solved even if using LIDAR, and if it is solved, it will work well enough on its own.

Competitors have chosen LIDAR as they believe camera-based vision alone will not be reliable enough.

These stands are not necessarily total opposites. There is still no guarantee that camera-based object detection can have sufficient reliability given current hardware & software specs.

Obviously Tesla has the major advantage of amount of training data. But is it enough? Competitors cite the computational complexity of a human vision system as something far more complex and intensive than HW3.

Although I work in machine / deep learning, it is hard to get a sense of where exactly the failure modes are appearing and what it will take to overcome them. For instance, detecting stationary objects in the middle of the road. The reliability needs to be really high. If a current model shows too high of false positive / false negatives rates, Tesla will of course try to train on even more useful data. But what if the failure rates are still too high?

Build a bigger model, of course. But bigger models are harder to train. And after that, of course they have to fit onto the hardware, so can't be too big! Tesla already determined that HW 2.5 was not sufficient enough. But how do they know that 3.0 will be sufficient?

 

[powertoold]

Tesla is building brand new ML development stack for FSD based on their data advantage vs. competitors. It's difficult to assess their progress.

It's possible that once they "activate" their unit-testing and unsupervised learning methods (eluded to by Elon's "Dojo" comment), Tesla will be far ahead of the game in terms of vision-based FSD.

Possibilities are endless with Tesla's data advantage.

 

[Bladerskb]

Tesla is building brand new ML development stack for FSD based on their data advantage vs. competitors. It's difficult to assess their progress.

It's possible that once they "activate" their unit-testing and unsupervised learning methods (eluded to by Elon's "Dojo" comment), Tesla will be far ahead of the game in terms of vision-based FSD.

Possibilities are endless with Tesla's data advantage.

For you to claim that there's an advantage you actually have to demonstrate that advantage.
Its been almost 4 years and AP is still behind in Computer Vision therefore there can't be an advantage.
AP still fails in the same way it failed 3 years ago therefore there is no exponential increase in reliability or performance which Elon claims time and time again.

 

[powertoold]

I agree, Elon loves to say exponential. It doesn't apply to AP.

However, my other points are valid. I'm not sure what you mean "AP is still behind in CV." AP, in its current form, is vastly superior to any consumer deployed CV based driving system.

 

[diplomat33]

Tesla is building brand new ML development stack for FSD based on their data advantage vs. competitors. It's difficult to assess their progress.

It's possible that once they "activate" their unit-testing and unsupervised learning methods (eluded to by Elon's "Dojo" comment), Tesla will be far ahead of the game in terms of vision-based FSD.

Possibilities are endless with Tesla's data advantage.

Believe me, I want Tesla to make progress towards FSD. But potential does not automatically translate into success. Having a lot of data, having a new ML stack or unsupervised learning might give Tesla some powerful tools but that does not automatically mean that Tesla will leap ahead. That would be like me saying "just wait until I get my new Nike running shoes, then I will be the fastest man on the planet". Good shoes are helpful but they don't automatically make me the best runner. it also takes time and training. Tesla still needs to do the work.

AP still fails in the same way it failed 3 years ago therefore there is no exponential increase in reliability or performance which Elon claims time and time again.

Can you give me an example of a failure that AP still has 3 years later? I can't quantify if the progress is exponential but I've definitely seen significant progress in AP, especially with V10. And I can give you many examples of cases where it used to fail 3 years ago but now, does not fail anymore. For example, AP used slam on the brakes almost too late when approaching stopped cars at a red light but now brakes smoothly in advance. My AP used to have trouble with lane keeping going through a particular intersection that bends to the left but now it keeps the lane perfectly through the intersection. My AP used to veer to the right when the right lane line would temporarily disappear but now it stays dead center. My AP used to be a bit slow in accelerating after a red light turns green but now it accelerates more quickly. I used to get quite a bit of phantom braking from overpasses, or just a shadow on the road but now I get virtually no phantom braking at all. I could go on. Overall, reliability and performance has definitely increased a lot in 3 years.

Again, I am not saying that Tesla is ahead. I know Waymo is way ahead, Mobileye is ahead in camera vision, etc... I am just saying that Tesla has made a lot of progress compared to where they used to be.

 

[powertoold]

CV-based FSD is dependent on tons of real life edge case data. Pretty much fact.

No other company has real life edge case data on tap.

Tesla is still developing the development stack and approach to take advantage of this data. I believe they'll figure out the stack soon.

That's why I believe they'll leap far ahead of the competition.

 

[alsetym]

Yeah, unfortunately it is hard to tell how close they are to some things. For instance, traffic light recognition. It's on, or its off basically. Same with avoiding objects on the road. It has to be nearly 100% confident to slam on the brakes with a car behind you to avoid something. Then one day, it will be that confident and it will stop. Lane change was like that as well. They went from sort of manual lane change to auto lane change and though it was slow and over cautious etc. I haven't heard of any accidents because it misinterpreted a car.

I'm sure that the dev versions of their software have little boxes around everything with a classification, and it probably pretty accurate because that's something that people (not just tesla) has been working on for decades. Heck, they may also be always correct in identifying a human, but not have the rule set developed for what the heck to do with the information. The AAA test though shows they have some work to do between the detection and decision making. You can see smart summon pretty much stops anytime anything moves (probably) because it is not 100% confident in its classification nor predicting movement. You can do that at 5mph, you can't do that at 50.

Anyway, my understanding of lidar is that it is great for telling you something is there, helping define drivable space, but isn't going to tell you what it is and if you should avoid it and what that object is likely to do (boxes act differently than toddlers). So, I think either way you need the camera to tell you whether it is a human (and which way it is facing, if it is looking at you, running into the road etc), a squirrel or whatever. It definitely can help in training your cameras faster since it has very low false positive rate for detecting that an object is there. So if you have less data it may be a necessity. Once your cameras are good enough for telling you if the object is a human, the lidar starts adding less and less value, but hey, I'm sure it doesn't hurt and likely helps while we figure it out. Tesla didn't exclude lidar because it doesn't add value, they just can't afford it (and it was big and ugly) and have access to enough data (and a human safety driver) that I'm sure they are hoping they can develop their training enough to skip the middle lidar step. So there is some chance it will pay off and lidar was not needed, and 0% chance they could have afforded to do it with lidar and they would have failed (at FSD), and in the mean time they have some cool stuff to play with! The choice made itself.

 

[alanbyvale]

I do not have the knowledge to argue any particular point and in fact, as of today, I don't even have my Tesla. But, saying Waymo is far ahead is slightly misleading (I think) as they map a ring fenced area to give the car prior knowledge of fixed objects and road layouts. I assumer that if you put a Waymo car on a UK road (or any other road) it would not have a clue whereas a Tesla would, and does, make a valiant effort to identify objects based on it's data model.

 

[jnads]

For you to claim that there's an advantage you actually have to demonstrate that advantage.
Its been almost 4 years and AP is still behind in Computer Vision therefore there can't be an advantage.
AP still fails in the same way it failed 3 years ago therefore there is no exponential increase in reliability or performance which Elon claims time and time again.

I think it's funny you post a conference / demonstration video and call it perfection.

EVERYONE manicures their data for presentation purposes. Of course they're not going to show their system failing.

Let's not forget Tesla used Mobileye for their early systems and people got killed crashing into cement barriers.


Has it not ever occurred to you Tesla is actually conservative in deploying their technology?

They already demonstrated beyond MobilEye autonomy in the Autonomy Day video. But they know their false positive/negative rate is high so they're collecting data to refine their technology.

 

[Daniel in SD]

Let's not forget Tesla used Mobileye for their early systems and people got killed crashing into cement barriers.

The first crash into a semi truck trailer was HW1 (Mobileye/Tesla), the Model X crash into a concrete barrier was HW2 (Tesla).
The Model 3 crash into the side of semi truck trailer was HW2.5.

They already demonstrated beyond MobilEye autonomy in the Autonomy Day video.

There is no evidence that Tesla's system is more advanced yet. You can find tons of video demos of Mobileye's system, some from third parties. Tesla has only released two and the 2019 video doesn't appear any more advanced than the 2016 video.

 

[Bladerskb]

Let's not forget Tesla used Mobileye for their early systems and people got killed crashing into cement barriers.

And since then dozens have gotten injured and killed crashing into the same cement barriers.
Also the cement barrier crash was all 2.0+ not AP1.

Has it not ever occurred to you Tesla is actually conservative in deploying their technology?

Absolutely not, this is blatantly obvious form their initial release of AP2 and NOA.

They already demonstrated beyond MobilEye autonomy in the Autonomy Day video. But they know their false positive/negative rate is high so they're collecting data to refine their technology.

Not only is mobileye ahead in Neural network development and deployment in production cars today but their demos are far more sophisticated than the basic demo that Tesla did on autonomy day on empty roads.

 

[Bladerskb]

Anyway, my understanding of lidar is that it is great for telling you something is there, helping define drivable space, but isn't going to tell you what it is and if you should avoid it and what that object is likely to do (boxes act differently than toddlers). So, I think either way you need the camera to tell you whether it is a human (and which way it is facing, if it is looking at you, running into the road etc), a squirrel or whatever. It definitely can help in training your cameras faster since it has very low false positive rate for detecting that an object is there. So if you have less data it may be a necessity. Once your cameras are good enough for telling you if the object is a human, the lidar starts adding less and less value, but hey, I'm sure it doesn't hurt and likely helps while we figure it out. Tesla didn't exclude lidar because it doesn't add value, they just can't afford it (and it was big and ugly) and have access to enough data (and a human safety driver) that I'm sure they are hoping they can develop their training enough to skip the middle lidar step. So there is some chance it will pay off and lidar was not needed, and 0% chance they could have afforded to do it with lidar and they would have failed (at FSD), and in the mean time they have some cool stuff to play with! The choice made itself.

Lidar not only does object detection but also object classification.

 

[Bladerskb]

Can you give me an example of a failure that AP still has 3 years later? I can't quantify if the progress is exponential but I've definitely seen significant progress in AP, especially with V10. And I can give you many examples of cases where it used to fail 3 years ago but now, does not fail anymore. For example, AP used slam on the brakes almost too late when approaching stopped cars at a red light but now brakes smoothly in advance. My AP used to have trouble with lane keeping going through a particular intersection that bends to the left but now it keeps the lane perfectly through the intersection. My AP used to veer to the right when the right lane line would temporarily disappear but now it stays dead center. My AP used to be a bit slow in accelerating after a red light turns green but now it accelerates more quickly. I used to get quite a bit of phantom braking from overpasses, or just a shadow on the road but now I get virtually no phantom braking at all. I could go on. Overall, reliability and performance has definitely increased a lot in 3 years.

Again, I am not saying that Tesla is ahead. I know Waymo is way ahead, Mobileye is ahead in camera vision, etc... I am just saying that Tesla has made a lot of progress compared to where they used to be.

I'm clearly not saying there isn't any progress. I'm saying its not the exponential progress elon and tesla fans claim.

For example:

Tesla didn’t fix an Autopilot problem for three years, and now another person is dead

This was after elon said that they will use radar like lidar to create something better than lidar to eliminate this problem.
Tesla fans went nuts writing articles like.

Tesla Leapfrogs Self-Driving Competitors With Radar That's Better Than Lidar

But the same accident has happened and other similar accidents of running into stuff dozens of times (back of fire trucks/ cars/ barriers/ debris, etc).

 

[croman]

Clearly Tesla has a lot of work to do to deliver meaningful autonomy but they've already made great progress. Even the above cited mobileye slide credits them with 5 kinds of "networks" but we also know about 5 more: traffic signals, traffic signs, road edges, road markings, and road signs which Tesla has developed but not deployed. We've heard about general object detection and they apparently will soon render cones which is cool. That almost closes the gap and I bet Tesla has more being developed for hw3 specifically.

Now the cool thing with Tesla is they aren't conservative and will ship that to us. Maybe it doesn't work well but at least we can watch it improve until it's utility begins to match the hype. I've seen this show for the past 3 years, it's never pretty at the beginning but I have faith in AK to work on it. Smart summon is actually improving but still sucks. Sometimes it manages to work well and I can see a path to it being useful. Let's see mobileye actually deliver.

Also, I'm still waiting on that L3 Audi A8 you hyped worse than Elon, @Bladerskb.