Welcome to Tesla Motors Club
Discuss Tesla's Model S, Model 3, Model X, Model Y, Cybertruck, Roadster and More.
Register

AI experts: true full self-driving cars could be decades away because AI is not good enough yet

Bladerskb

Senior Software Engineer
Oct 24, 2016
2,317
2,647
Michigan
None of these companies have the capabilities to collect a lot of data. Ergo, they will not say they need a lot. It's pretty obvious.

Assessment of what the "correct" approaches are should be made without influence of what any company says (including Tesla), because they are all biased.

What does lots of data means? Before tesla fans say it as acouple thousand cars, then 10k, then 50k, then 100k, then 500k, now 1 million.
Next year they will say anything under 2 million cars collecting data is worthless.

So no, so far, its basically influenced by whatever Tesla is doing.
But I would love to hear from you what lots of data consists of factually without Tesla bias and influence.
 
  • Like
Reactions: diplomat33

DrDabbles

Active Member
Jul 28, 2017
1,092
1,310
NH, US
Birds do very well, better than humans in some cases.

There are zero cases in which birds drive cars.

or, to our knowledge, any great understanding of the world

To your knowledge. Birds have magnetic minerals in their brains, they use air currents, and many of them migrate vast distances based on scent and navigational marks.

So are birds a bad model for a FSD car?

No. Not in any way. Birds don't need to obey the extremely complex rules of the road.

f FSD can achieve a bird intelligence, it would be good enough.

FSD isn't any kind of intelligence. In fact, calling "artificial intelligence" intelligence is a misnomer in the first place.
 
  • Like
Reactions: Tundrok

ZeApelido

Active Member
Jun 1, 2016
3,154
26,031
The Peninsula, CA
What does lots of data means? Before tesla fans say it as acouple thousand cars, then 10k, then 50k, then 100k, then 500k, now 1 million.
Next year they will say anything under 2 million cars collecting data is worthless.

So no, so far, its basically influenced by whatever Tesla is doing.
But I would love to hear from you what lots of data consists of factually without Tesla bias and influence.


I mentioned this:

I've harped on this many times in previous posts, no need to go through this again. Suffice to say, simulation / augmentation is important & necessary to add new types of cases, but it alone is not sufficient.

No one is saying simulation isn't necessary, but only even great simulation will not overcome a deficit of edge case data.

The question becomes - what is a sufficient amount of edge case data? Well based on my experience, it's probably a lot.

Autonomous cars might be the hardest machine learning / algorithmic challenge humans have encountered.

Not only is the 1) dimensionality / complexity extremely high, but 2) the accuracy requirements are also extremely high. The amount of data needed as either 1 or 2 increases scales non-linearly, and both are very high in this case.


So yes, my intuition based on previous ML problems is that no, collecting some data in some cities will not be sufficient. We will need to collect data basically everywhere. My guess is Waymo/Cruise are not on a path to collect enough data to deploy everywhere in the U.S. yet. Their areas of collection are too focused. Maybe Tesla won't even get enough data.



It's entirely speculative.

This is nothing to do with Tesla. It has to do with an educated guess based on my experiences in building statistical models (which were much less complex than autonomous vehicles). The conservative thing to do is assume it will take a lot of data.

The "look at what Tesla has or hasn't done so far" doesn't prove anything IMO about amount of data available to them. I don't think they've taken advantage of all that data for multiple reasons. Whether incompetence along the way or lack of infrastructure / architectures to take advantage of it, I don't believe they've optimised usefulness of that data yet.
 
  • Like
Reactions: mikes_fsd

Bladerskb

Senior Software Engineer
Oct 24, 2016
2,317
2,647
Michigan
I mentioned this:



This is nothing to do with Tesla. It has to do with an educated guess based on my experiences in building statistical models (which were much less complex than autonomous vehicles). The conservative thing to do is assume it will take a lot of data.
You still didn't answer the question, what does "it will take a lot of data" mean.
You need to quantify that statement unless its useless.
The "look at what Tesla has or hasn't done so far" doesn't prove anything IMO about amount of data available to them. I don't think they've taken advantage of all that data for multiple reasons. Whether incompetence along the way or lack of infrastructure / architectures to take advantage of it, I don't believe they've optimised usefulness of that data yet.

You say that Tesla has "lot of data" but you dont want to quantify "lot of data". So you are making it about Tesla.
 

ZeApelido

Active Member
Jun 1, 2016
3,154
26,031
The Peninsula, CA
You still didn't answer the question, what does "it will take a lot of data" mean.
You need to quantify that statement unless its useless.


You say that Tesla has "lot of data" but you dont want to quantify "lot of data". So you are making it about Tesla.

If I had to speculate...

There are at least 6 million intersections in the main 500 or so urbanized areas of the U.S. (So probably even more including rural).

Every intersection should be "tested" N number of times. N is related to the dimensionality and complexity of the problem. N ~ 2^D where D = # dimensions.

Dimensions are things like

1) Is it nighttime vs daytime?
2) Raining vs not
3) Foggy vs not
4) Pedestrian crossing vs not

We could probably come up with a 100 or so dimensions (just a guess). 2^100 = 1267650600228229401496703205376

And right now we are assuming sampling each intersection with each combination of conditions just one time is sufficient. But it probably should be 10, or 100 times.

so 6 million x 100 samples x 1267650600228229401496703205376 combinations.

So we need ~ 7 x 10^32 cases.

Now, most of these cases will be redundant. Probably 99.999999 % will be useless. The problem is, you don't know which ones! So, a test vehicle may not have to save and send the data back to the cloud for training, but the test vehicle has to at least see it and compare it with whatever inference model is running.

This is the conservative approach. Waymo, of course, has nothing close to this. Tesla does not either. But we know with say 100,000 or 1,000,000 vehicles (in a few years) in the U.S., most intersections can be sampled 10 to 100 times daily. Then in one year, each intersection could be sampled 30,000 times. So we could have 200 billion intersection samples per year (2 x 10^10).

2 x 10^10 still doesn't reach the 7 x 10^32 came up with. So no, Tesla would not have enough data to match my conservative estimate of how much data needs to be seen.

But, of course we hope that each intersection doesn't really need to be sampled 2^100 times, hopefully there is a bunch of overlap in edge cases. How many orders of magnitude can be chopped off? Only the performance of the algorithms over time will tell us.

Tesla (and maybe eventually in a few years Mobileye) will be able to run their algorithm on every intersection in the U.S. daily to see when both their training accuracy has reach an ultimate asymptote and see performance on unseen diverse test data to make sure performance on test data matches training data.

Waymo, Cruise do not and will not have that. They are banking on most of those intersections not mattering, most interactions not mattering.

Lot's of assumptions.
 

diplomat33

Well-Known Member
Aug 3, 2017
7,756
9,078
Terre Haute, IN USA
If I had to speculate...

There are at least 6 million intersections in the main 500 or so urbanized areas of the U.S. (So probably even more including rural).

Every intersection should be "tested" N number of times. N is related to the dimensionality and complexity of the problem. N ~ 2^D where D = # dimensions.

Dimensions are things like

1) Is it nighttime vs daytime?
2) Raining vs not
3) Foggy vs not
4) Pedestrian crossing vs not

We could probably come up with a 100 or so dimensions (just a guess). 2^100 = 1267650600228229401496703205376

And right now we are assuming sampling each intersection with each combination of conditions just one time is sufficient. But it probably should be 10, or 100 times.

so 6 million x 100 samples x 1267650600228229401496703205376 combinations.

So we need ~ 7 x 10^32 cases.

Now, most of these cases will be redundant. Probably 99.999999 % will be useless. The problem is, you don't know which ones! So, a test vehicle may not have to save and send the data back to the cloud for training, but the test vehicle has to at least see it and compare it with whatever inference model is running.

This is the conservative approach. Waymo, of course, has nothing close to this. Tesla does not either. But we know with say 100,000 or 1,000,000 vehicles (in a few years) in the U.S., most intersections can be sampled 10 to 100 times daily. Then in one year, each intersection could be sampled 30,000 times. So we could have 200 billion intersection samples per year (2 x 10^10).

2 x 10^10 still doesn't reach the 7 x 10^32 came up with. So no, Tesla would not have enough data to match my conservative estimate of how much data needs to be seen.

But, of course we hope that each intersection doesn't really need to be sampled 2^100 times, hopefully there is a bunch of overlap in edge cases. How many orders of magnitude can be chopped off? Only the performance of the algorithms over time will tell us.

Tesla (and maybe eventually in a few years Mobileye) will be able to run their algorithm on every intersection in the U.S. daily to see when both their training accuracy has reach an ultimate asymptote and see performance on unseen diverse test data to make sure performance on test data matches training data.

Waymo, Cruise do not and will not have that. They are banking on most of those intersections not mattering, most interactions not mattering.

Lot's of assumptions.

1) That is where simulations can be very helpful. You can run 100M simulations far faster than you could test those 100M cases in the real world.

2) Keep in mind that the Tesla fleet is not uniform. Some areas have more Teslas. Other areas have fewer Teslas. So some intersections will probably be over sampled and some intersections will be under sampled.

3) I don't agree that Waymo and Cruise are banking on most interactions not mattering. Quite the contrary, Waymo and Cruise are aggressively testing for all cases that they can in both the real world and simulations. They are not ignoring some cases or hoping they don't matter.

What they are doing is using geofenced areas to make the number of dimensions and cases much smaller and therefore much more manageable. It will be easier to "solve FSD" in a geofenced area than in the entire US. And like you said, there will be a lot of overlap. So once you "solve FSD" in say SF, you can go to another city like say Orlando, and a lot of the cases will already be solved. You will not be starting from scratch. Your FSD will be starting at a pretty good level already. You will only need to solve for any new edge cases.

Also, Waymo and Cruise don't need to encounter all 6M intersections in the US to train their system since they pre map first. Any new intersection that they want to drive at some point, they will simply pre map first before driving. Using HD maps, the system will know how to navigate the intersection. Waymo and Cruise only need to test for other dimensions like weather, other objects crossing, etc...
 

Daniel in SD

Well-Known Member
Jan 25, 2018
6,556
9,307
San Diego
Is there any evidence that Cruise and Waymo need more training data? If you look at Waymo's safety paper it seems like the issue is with driving policy and planning. The biggest issue seems to be that their car drives in unpredictable (i.e. non human) ways that cause human drivers (even a pedestrian!) to run into them.
They did not have any single-vehicle collisions in 6.1 million miles of driving so in that sense they are definitely super human.
Anyway, are there any collisions here that would be solved by more training data?https://storage.googleapis.com/sdc-...Waymo-Public-Road-Safety-Performance-Data.pdf
Of course 6.1 million miles isn't enough to determine safety without extrapolation because the human driver fatality rate is 1 per 100 million miles.
 
  • Like
Reactions: diplomat33

diplomat33

Well-Known Member
Aug 3, 2017
7,756
9,078
Terre Haute, IN USA
Is there any evidence that Cruise and Waymo need more training data? If you look at Waymo's safety paper it seems like the issue is with driving policy and planning. The biggest issue seems to be that their car drives in unpredictable (i.e. non human) ways that cause human drivers (even a pedestrian!) to run into them.
They did not have any single-vehicle collisions in 6.1 million miles of driving so in that sense they are definitely super human.
Anyway, are there any collisions here that would be solved by more training data?https://storage.googleapis.com/sdc-...Waymo-Public-Road-Safety-Performance-Data.pdf
Of course 6.1 million miles isn't enough to determine safety without extrapolation because the human driver fatality rate is 1 per 100 million miles.

I don't think Waymo and Cruise need more training data for like basic perception. Yes, there main issue seems to be with driving policy and planning. So I think they do need more driving miles in order to gain more experience on how to handle these planning and driving policy cases.
 
  • Funny
Reactions: mikes_fsd

ZeApelido

Active Member
Jun 1, 2016
3,154
26,031
The Peninsula, CA
1) That is where simulations can be very helpful. You can run 100M simulations far faster than you could test those 100M cases in the real world.

2) Keep in mind that the Tesla fleet is not uniform. Some areas have more Teslas. Other areas have fewer Teslas. So some intersections will probably be over sampled and some intersections will be under sampled.

3) I don't agree that Waymo and Cruise are banking on most interactions not mattering. Quite the contrary, Waymo and Cruise are aggressively testing for all cases that they can in both the real world and simulations. They are not ignoring some cases or hoping they don't matter.

What they are doing is using geofenced areas to make the number of dimensions and cases much smaller and therefore much more manageable. It will be easier to "solve FSD" in a geofenced area than in the entire US. And like you said, there will be a lot of overlap. So once you "solve FSD" in say SF, you can go to another city like say Orlando, and a lot of the cases will already be solved. You will not be starting from scratch. Your FSD will be starting at a pretty good level already. You will only need to solve for any new edge cases.

Also, Waymo and Cruise don't need to encounter all 6M intersections in the US to train their system since they pre map first. Any new intersection that they want to drive at some point, they will simply pre map first before driving. Using HD maps, the system will know how to navigate the intersection. Waymo and Cruise only need to test for other dimensions like weather, other objects crossing, etc...

1) Again, those simulations are useful but they don't replace the real-world cases that are different than what are put in the simulations. And what is different is unknown.

2) Definitely true, but that can been tuned by dropping data from areas with high frequency with GPS tags. Whereas Waymo / Cruise are even more uniform. Most areas have 0 Waymos.

Yes you can start in SF then apply to Orlando, solve iteratively, city by city. A slower model, but possible.

Premapping is useful but I don't think removes need to encounter different intersections. Driving policy is not just about what to do in an intersection when everything is normal. The hard part is when things aren't normal, which can be abnormal differently at each intersection, and only happen very sparsely.
 

diplomat33

Well-Known Member
Aug 3, 2017
7,756
9,078
Terre Haute, IN USA
1) Again, those simulations are useful but they don't replace the real-world cases that are different than what are put in the simulations. And what is different is unknown.

I think it will take too long to solve all edge cases with just real world driving alone. There are way too many. You said it yourself, there could be as many as 10^37 cases. I think you need BOTH real world driving AND simulations.

Premapping is useful but I don't think removes need to encounter different intersections. Driving policy is not just about what to do in an intersection when everything is normal. The hard part is when things aren't normal, which can be abnormal differently at each intersection, and only happen very sparsely.

Yes, you still need to solve edge cases but with pre-mapping, the car is starting with some pre-knowledge of what the road looks like. So you don't need to start from scratch, teaching the car what the intersection looks like. You don't need to train the car to identify every single intersection since it will already have that information before it drives. You can focus on edge cases with moving objects.
 

rxlawdude

Active Member
Jul 10, 2015
2,709
1,933
Orange County, CA
I think it will take too long to solve all edge cases with just real world driving alone. There are way too many. You said it yourself, there could be as many as 10^37 cases. I think you need BOTH real world driving AND simulations.



Yes, you still need to solve edge cases but with pre-mapping, the car is starting with some pre-knowledge of what the road looks like. So you don't need to start from scratch, teaching the car what the intersection looks like. You don't need to train the car to identify every single intersection since it will already have that information before it drives. You can focus on edge cases with moving objects.
Thanks for admitting that Waymo will never exceed L4. By your own statement.

Agree!
 

rxlawdude

Active Member
Jul 10, 2015
2,709
1,933
Orange County, CA
Where did I admit that?
You've just stated that Waymo uses their Lidar for mapping for modeling. Unless they do that in whiteout conditions on every dirt road in the country, pray tell how a Waymo vehicle would proceed. L5 means every human driveable condition. Waymo, by your own statements, MUST stop in its tracks faster than for traffic cones laid out for a construction zone it didn't "know about."
 

Daniel in SD

Well-Known Member
Jan 25, 2018
6,556
9,307
San Diego
You've just stated that Waymo uses their Lidar for mapping for modeling. Unless they do that in whiteout conditions on every dirt road in the country, pray tell how a Waymo vehicle would proceed. L5 means every human driveable condition. Waymo, by your own statements, MUST stop in its tracks faster than for traffic cones laid out for a construction zone it didn't "know about."
L5 will require a technological breakthrough, there's no way to know whether or not Waymo will be the company to make that breakthrough. One could predict failure for every AV company and be right the vast majority of the time! Most of them will fail at solving the most difficult engineering problem ever attempted...
 
  • Like
Reactions: diplomat33

diplomat33

Well-Known Member
Aug 3, 2017
7,756
9,078
Terre Haute, IN USA
You've just stated that Waymo uses their Lidar for mapping for modeling. Unless they do that in whiteout conditions on every dirt road in the country, pray tell how a Waymo vehicle would proceed. L5 means every human driveable condition. Waymo, by your own statements, MUST stop in its tracks faster than for traffic cones laid out for a construction zone it didn't "know about."

I think you are confused. You don't pre-map in whiteout conditions. You pre-map in good weather. That way, the car has an accurate map of the road even in bad conditions where it can't see.

Second, L5 does not require you to drive in whiteout conditions because whiteouts are not human driveable conditions. The SAE document specifically mentions whiteouts as not being part of L5:

"However, there may be conditions not manageable by a driver in which the ADS would also be unable to complete a given trip (e.g., white-out snow storm, flooded roads, glare ice, etc.) until or unless the adverse conditions clear. " p 32.

Also, Waymo cars don't drive just based on the HD map. They also drive based on the cameras, lidar and radar. Waymo cars don't stop if the HD map is wrong. And Waymo cars detect traffic cones based on lidar and cameras. They don't use HD maps to see traffic cones.

Lastly, nobody has solved L5. If Waymo solves L4, it would be more than anybody else has achieved. So I don't see the problem with Waymo not achieving L5 yet. L4 is real FSD. And FSD does not need to be L5 to be useful.
 

rxlawdude

Active Member
Jul 10, 2015
2,709
1,933
Orange County, CA
I think you are confused. You don't pre-map in whiteout conditions. You pre-map in good weather. That way, the car has an accurate map of the road even in bad conditions where it can't see.

Second, L5 does not require you to drive in whiteout conditions because whiteouts are not human driveable conditions.

Also, Waymo cars don't drive just based on the HD map. They also drive based on the cameras, lidar and radar. Waymo cars don't stop if the HD map is wrong. And Waymo cars detect traffic cones based on lidar and cameras. They don't use HD maps to see traffic cones.

Lastly, nobody has solved L5. If Waymo solves L4, it would be more than anybody else has achieved. So I don't see the problem with Waymo not achieving L5 yet. L4 is real FSD. And FSD does not need to be L5 to be useful.
Define "human driveable" with respect to dirt roads, including long private drives to rural homes.
 

rxlawdude

Active Member
Jul 10, 2015
2,709
1,933
Orange County, CA
I think you are confused. You don't pre-map in whiteout conditions. You pre-map in good weather. That way, the car has an accurate map of the road even in bad conditions where it can't see.

Second, L5 does not require you to drive in whiteout conditions because whiteouts are not human driveable conditions. The SAE document specifically mentions whiteouts as not being part of L5.

Also, Waymo cars don't drive just based on the HD map. They also drive based on the cameras, lidar and radar. Waymo cars don't stop if the HD map is wrong. And Waymo cars detect traffic cones based on lidar and cameras. They don't use HD maps to see traffic cones.

Lastly, nobody has solved L5. If Waymo solves L4, it would be more than anybody else has achieved. So I don't see the problem with Waymo not achieving L5 yet. L4 is real FSD. And FSD does not need to be L5 to be useful.
FSD, in your continuing love of all things Waymo, includes ability to drive where a human can drive. Perhaps "whiteout' is overly broad. Let's say, snow covered, 5 mile winding dirt road. Humans can drive that. Waymo? Ever? Probably not. Ergo, Waymo is not "FSD."
 
  • Disagree
Reactions: diplomat33

rxlawdude

Active Member
Jul 10, 2015
2,709
1,933
Orange County, CA
FSD, in your continuing love of all things Waymo, includes ability to drive where a human can drive. Perhaps "whiteout' is overly broad. Let's say, snow covered, 5 mile winding dirt road. Humans can drive that. Waymo? Ever? Probably not. Ergo, Waymo is not "FSD."
A disagree with no rebuttle. Can Waymo drive a snow covered, dirt road it mapped in good weather? Every human-driveable dirt road? Like the driveway leading to a rural destination?

Yes or no will do. Now you're backing down to "L5 is not needed." We're making progress on winnowing the technical limitations of Waymo. Let's keep it 100.
 

diplomat33

Well-Known Member
Aug 3, 2017
7,756
9,078
Terre Haute, IN USA
FSD, in your continuing love of all things Waymo, includes ability to drive where a human can drive. Perhaps "whiteout' is overly broad. Let's say, snow covered, 5 mile winding dirt road. Humans can drive that. Waymo? Ever? Probably not. Ergo, Waymo is not "FSD."

Yes, Waymo could drive on a snow covered, 5 mile, winding dirt road.
 

About Us

Formed in 2006, Tesla Motors Club (TMC) was the first independent online Tesla community. Today it remains the largest and most dynamic community of Tesla enthusiasts. Learn more.

Do you value your experience at TMC? Consider becoming a Supporting Member of Tesla Motors Club. As a thank you for your contribution, you'll get nearly no ads in the Community and Groups sections. Additional perks are available depending on the level of contribution. Please visit the Account Upgrades page for more details.


SUPPORT TMC
Top