Welcome to Tesla Motors Club
Discuss Tesla's Model S, Model 3, Model X, Model Y, Cybertruck, Roadster and More.
Register

Camera-Based Object Detection - The Limiting Factor to FSD

ZeApelido

Active Member
Jun 1, 2016
3,314
30,245
The Peninsula, CA
Trying to get out of these threads complaining about missed deadlines, as I wanted to discuss more specifically the limiting factors to achieving some level of FSD. We know Tesla has chosen to focus on camera-based vision. They believe vision will have to be solved even if using LIDAR, and if it is solved, it will work well enough on its own.

Competitors have chosen LIDAR as they believe camera-based vision alone will not be reliable enough.

These stands are not necessarily total opposites. There is still no guarantee that camera-based object detection can have sufficient reliability given current hardware & software specs.

Obviously Tesla has the major advantage of amount of training data. But is it enough? Competitors cite the computational complexity of a human vision system as something far more complex and intensive than HW3.

Although I work in machine / deep learning, it is hard to get a sense of where exactly the failure modes are appearing and what it will take to overcome them. For instance, detecting stationary objects in the middle of the road. The reliability needs to be really high. If a current model shows too high of false positive / false negatives rates, Tesla will of course try to train on even more useful data. But what if the failure rates are still too high?

Build a bigger model, of course. But bigger models are harder to train. And after that, of course they have to fit onto the hardware, so can't be too big! Tesla already determined that HW 2.5 was not sufficient enough. But how do they know that 3.0 will be sufficient?
 

powertoold

Active Member
Oct 10, 2014
3,025
6,524
USA
Tesla is building brand new ML development stack for FSD based on their data advantage vs. competitors. It's difficult to assess their progress.

It's possible that once they "activate" their unit-testing and unsupervised learning methods (eluded to by Elon's "Dojo" comment), Tesla will be far ahead of the game in terms of vision-based FSD.

Possibilities are endless with Tesla's data advantage.
 

Bladerskb

Senior Software Engineer
Oct 24, 2016
2,510
4,211
Michigan
Tesla is building brand new ML development stack for FSD based on their data advantage vs. competitors. It's difficult to assess their progress.

It's possible that once they "activate" their unit-testing and unsupervised learning methods (eluded to by Elon's "Dojo" comment), Tesla will be far ahead of the game in terms of vision-based FSD.

Possibilities are endless with Tesla's data advantage.

For you to claim that there's an advantage you actually have to demonstrate that advantage.
Its been almost 4 years and AP is still behind in Computer Vision therefore there can't be an advantage.
AP still fails in the same way it failed 3 years ago therefore there is no exponential increase in reliability or performance which Elon claims time and time again.

 
  • Like
Reactions: motocoder

powertoold

Active Member
Oct 10, 2014
3,025
6,524
USA
I agree, Elon loves to say exponential. It doesn't apply to AP.

However, my other points are valid. I'm not sure what you mean "AP is still behind in CV." AP, in its current form, is vastly superior to any consumer deployed CV based driving system.
 

diplomat33

Average guy who loves autonomous vehicles
Aug 3, 2017
8,401
11,728
Terre Haute, IN USA
Tesla is building brand new ML development stack for FSD based on their data advantage vs. competitors. It's difficult to assess their progress.

It's possible that once they "activate" their unit-testing and unsupervised learning methods (eluded to by Elon's "Dojo" comment), Tesla will be far ahead of the game in terms of vision-based FSD.

Possibilities are endless with Tesla's data advantage.

Believe me, I want Tesla to make progress towards FSD. But potential does not automatically translate into success. Having a lot of data, having a new ML stack or unsupervised learning might give Tesla some powerful tools but that does not automatically mean that Tesla will leap ahead. That would be like me saying "just wait until I get my new Nike running shoes, then I will be the fastest man on the planet". Good shoes are helpful but they don't automatically make me the best runner. it also takes time and training. Tesla still needs to do the work.

AP still fails in the same way it failed 3 years ago therefore there is no exponential increase in reliability or performance which Elon claims time and time again.

Can you give me an example of a failure that AP still has 3 years later? I can't quantify if the progress is exponential but I've definitely seen significant progress in AP, especially with V10. And I can give you many examples of cases where it used to fail 3 years ago but now, does not fail anymore. For example, AP used slam on the brakes almost too late when approaching stopped cars at a red light but now brakes smoothly in advance. My AP used to have trouble with lane keeping going through a particular intersection that bends to the left but now it keeps the lane perfectly through the intersection. My AP used to veer to the right when the right lane line would temporarily disappear but now it stays dead center. My AP used to be a bit slow in accelerating after a red light turns green but now it accelerates more quickly. I used to get quite a bit of phantom braking from overpasses, or just a shadow on the road but now I get virtually no phantom braking at all. I could go on. Overall, reliability and performance has definitely increased a lot in 3 years.

Again, I am not saying that Tesla is ahead. I know Waymo is way ahead, Mobileye is ahead in camera vision, etc... I am just saying that Tesla has made a lot of progress compared to where they used to be.
 

powertoold

Active Member
Oct 10, 2014
3,025
6,524
USA
CV-based FSD is dependent on tons of real life edge case data. Pretty much fact.

No other company has real life edge case data on tap.

Tesla is still developing the development stack and approach to take advantage of this data. I believe they'll figure out the stack soon.

That's why I believe they'll leap far ahead of the competition.
 

alsetym

Member
Apr 28, 2018
276
349
hamilton
Yeah, unfortunately it is hard to tell how close they are to some things. For instance, traffic light recognition. It's on, or its off basically. Same with avoiding objects on the road. It has to be nearly 100% confident to slam on the brakes with a car behind you to avoid something. Then one day, it will be that confident and it will stop. Lane change was like that as well. They went from sort of manual lane change to auto lane change and though it was slow and over cautious etc. I haven't heard of any accidents because it misinterpreted a car.

I'm sure that the dev versions of their software have little boxes around everything with a classification, and it probably pretty accurate because that's something that people (not just tesla) has been working on for decades. Heck, they may also be always correct in identifying a human, but not have the rule set developed for what the heck to do with the information. The AAA test though shows they have some work to do between the detection and decision making. You can see smart summon pretty much stops anytime anything moves (probably) because it is not 100% confident in its classification nor predicting movement. You can do that at 5mph, you can't do that at 50.

Anyway, my understanding of lidar is that it is great for telling you something is there, helping define drivable space, but isn't going to tell you what it is and if you should avoid it and what that object is likely to do (boxes act differently than toddlers). So, I think either way you need the camera to tell you whether it is a human (and which way it is facing, if it is looking at you, running into the road etc), a squirrel or whatever. It definitely can help in training your cameras faster since it has very low false positive rate for detecting that an object is there. So if you have less data it may be a necessity. Once your cameras are good enough for telling you if the object is a human, the lidar starts adding less and less value, but hey, I'm sure it doesn't hurt and likely helps while we figure it out. Tesla didn't exclude lidar because it doesn't add value, they just can't afford it (and it was big and ugly) and have access to enough data (and a human safety driver) that I'm sure they are hoping they can develop their training enough to skip the middle lidar step. So there is some chance it will pay off and lidar was not needed, and 0% chance they could have afforded to do it with lidar and they would have failed (at FSD), and in the mean time they have some cool stuff to play with! The choice made itself.
 

alanbyvale

Member
Aug 11, 2019
53
16
Nottingham
I do not have the knowledge to argue any particular point and in fact, as of today, I don't even have my Tesla. But, saying Waymo is far ahead is slightly misleading (I think) as they map a ring fenced area to give the car prior knowledge of fixed objects and road layouts. I assumer that if you put a Waymo car on a UK road (or any other road) it would not have a clue whereas a Tesla would, and does, make a valiant effort to identify objects based on it's data model.
 

jnads

Member
Oct 17, 2019
37
27
Iowa
For you to claim that there's an advantage you actually have to demonstrate that advantage.
Its been almost 4 years and AP is still behind in Computer Vision therefore there can't be an advantage.
AP still fails in the same way it failed 3 years ago therefore there is no exponential increase in reliability or performance which Elon claims time and time again.


I think it's funny you post a conference / demonstration video and call it perfection.

EVERYONE manicures their data for presentation purposes. Of course they're not going to show their system failing.

Let's not forget Tesla used Mobileye for their early systems and people got killed crashing into cement barriers.


Has it not ever occurred to you Tesla is actually conservative in deploying their technology?

They already demonstrated beyond MobilEye autonomy in the Autonomy Day video. But they know their false positive/negative rate is high so they're collecting data to refine their technology.
 
  • Like
Reactions: mikes_fsd

Daniel in SD

Well-Known Member
Jan 25, 2018
7,060
10,521
San Diego
Let's not forget Tesla used Mobileye for their early systems and people got killed crashing into cement barriers.
The first crash into a semi truck trailer was HW1 (Mobileye/Tesla), the Model X crash into a concrete barrier was HW2 (Tesla).
The Model 3 crash into the side of semi truck trailer was HW2.5.
They already demonstrated beyond MobilEye autonomy in the Autonomy Day video.
There is no evidence that Tesla's system is more advanced yet. You can find tons of video demos of Mobileye's system, some from third parties. Tesla has only released two and the 2019 video doesn't appear any more advanced than the 2016 video.
 

Bladerskb

Senior Software Engineer
Oct 24, 2016
2,510
4,211
Michigan
Let's not forget Tesla used Mobileye for their early systems and people got killed crashing into cement barriers.

And since then dozens have gotten injured and killed crashing into the same cement barriers.
Also the cement barrier crash was all 2.0+ not AP1.

Has it not ever occurred to you Tesla is actually conservative in deploying their technology?

Absolutely not, this is blatantly obvious form their initial release of AP2 and NOA.

They already demonstrated beyond MobilEye autonomy in the Autonomy Day video. But they know their false positive/negative rate is high so they're collecting data to refine their technology.

Not only is mobileye ahead in Neural network development and deployment in production cars today but their demos are far more sophisticated than the basic demo that Tesla did on autonomy day on empty roads.





578gwi8zu5y21.png
 

Bladerskb

Senior Software Engineer
Oct 24, 2016
2,510
4,211
Michigan
Anyway, my understanding of lidar is that it is great for telling you something is there, helping define drivable space, but isn't going to tell you what it is and if you should avoid it and what that object is likely to do (boxes act differently than toddlers). So, I think either way you need the camera to tell you whether it is a human (and which way it is facing, if it is looking at you, running into the road etc), a squirrel or whatever. It definitely can help in training your cameras faster since it has very low false positive rate for detecting that an object is there. So if you have less data it may be a necessity. Once your cameras are good enough for telling you if the object is a human, the lidar starts adding less and less value, but hey, I'm sure it doesn't hurt and likely helps while we figure it out. Tesla didn't exclude lidar because it doesn't add value, they just can't afford it (and it was big and ugly) and have access to enough data (and a human safety driver) that I'm sure they are hoping they can develop their training enough to skip the middle lidar step. So there is some chance it will pay off and lidar was not needed, and 0% chance they could have afforded to do it with lidar and they would have failed (at FSD), and in the mean time they have some cool stuff to play with! The choice made itself.

Lidar not only does object detection but also object classification.
 
  • Informative
Reactions: motocoder

Bladerskb

Senior Software Engineer
Oct 24, 2016
2,510
4,211
Michigan
Can you give me an example of a failure that AP still has 3 years later? I can't quantify if the progress is exponential but I've definitely seen significant progress in AP, especially with V10. And I can give you many examples of cases where it used to fail 3 years ago but now, does not fail anymore. For example, AP used slam on the brakes almost too late when approaching stopped cars at a red light but now brakes smoothly in advance. My AP used to have trouble with lane keeping going through a particular intersection that bends to the left but now it keeps the lane perfectly through the intersection. My AP used to veer to the right when the right lane line would temporarily disappear but now it stays dead center. My AP used to be a bit slow in accelerating after a red light turns green but now it accelerates more quickly. I used to get quite a bit of phantom braking from overpasses, or just a shadow on the road but now I get virtually no phantom braking at all. I could go on. Overall, reliability and performance has definitely increased a lot in 3 years.

Again, I am not saying that Tesla is ahead. I know Waymo is way ahead, Mobileye is ahead in camera vision, etc... I am just saying that Tesla has made a lot of progress compared to where they used to be.

I'm clearly not saying there isn't any progress. I'm saying its not the exponential progress elon and tesla fans claim.

For example:

Tesla didn’t fix an Autopilot problem for three years, and now another person is dead

This was after elon said that they will use radar like lidar to create something better than lidar to eliminate this problem.
Tesla fans went nuts writing articles like.

Tesla Leapfrogs Self-Driving Competitors With Radar That's Better Than Lidar

But the same accident has happened and other similar accidents of running into stuff dozens of times (back of fire trucks/ cars/ barriers/ debris, etc).
 

croman

Active Member
Nov 21, 2016
4,945
7,563
Chicago, IL
Clearly Tesla has a lot of work to do to deliver meaningful autonomy but they've already made great progress. Even the above cited mobileye slide credits them with 5 kinds of "networks" but we also know about 5 more: traffic signals, traffic signs, road edges, road markings, and road signs which Tesla has developed but not deployed. We've heard about general object detection and they apparently will soon render cones which is cool. That almost closes the gap and I bet Tesla has more being developed for hw3 specifically.

Now the cool thing with Tesla is they aren't conservative and will ship that to us. Maybe it doesn't work well but at least we can watch it improve until it's utility begins to match the hype. I've seen this show for the past 3 years, it's never pretty at the beginning but I have faith in AK to work on it. Smart summon is actually improving but still sucks. Sometimes it manages to work well and I can see a path to it being useful. Let's see mobileye actually deliver.

Also, I'm still waiting on that L3 Audi A8 you hyped worse than Elon, @Bladerskb.
 
  • Like
Reactions: Pale_Rider

Products we're discussing on TMC...

About Us

Formed in 2006, Tesla Motors Club (TMC) was the first independent online Tesla community. Today it remains the largest and most dynamic community of Tesla enthusiasts. Learn more.

Do you value your experience at TMC? Consider becoming a Supporting Member of Tesla Motors Club. As a thank you for your contribution, you'll get nearly no ads in the Community and Groups sections. Additional perks are available depending on the level of contribution. Please visit the Account Upgrades page for more details.


SUPPORT TMC
Top