Andrej Karpathy - AI for Full-Self Driving (2020)

[Bitdepth]

Devils advocate here. People take Tesla’s delay to achieve FSD as a sign that it’s nice advanced drivers assist, but it’s all smoke and mirrors beyond it. What makes Karpathy stay if it all truly was a non viable approach? Money can be made elsewhere.

What you are doing with your "devils advocate" argument is called a strawman. For the most part in these threads some arguments are that Tesla's delay to achieve FSD is a sign that it is a lot harder than Elon Musk had led people to believe.

FSD was sold and promised to be released way back in 2016 "pending regulation". Since then, they've conceded that the hardware they had was not enough, they have redesigned the hardware and software stack several times with each improving on the previous.

Objectively looking at it, they broke away from using Mobile Eye and within months were able to achieve software parity with what they had with Mobile Eye. Autopilot is very impressive. But as we can now see that is the easy part and achieving level 4 FSD is a lot difficult.

To answer your question, Karparthy like most engineers i have known in my life love a good challenge. No doubt they are working on it every day as is everyone else in this field trying to tackle this problem. The goal for everyone is to be able to get the amount of sensors needed down to a cost effective price that can be sold to consumers while also solving the software side of it as well.

 

[DanCar]

If you are having a discussion based on a scientific topic, your language should reflect as such.

If we are talking about growing flowers is that scientific? It is only scientific because you deem it so. Others would call it a technical subject.

You are no longer discussing in good faith and calling out such cavalier and disingenuous language is not the same as silencing someone.

Yes, not the same, but has same affect.

you certainly cannot you say a Level 4 autonomous vehicle which uses every sensor modality afforded is just a glorified tram

What would you call something that is labeled as level 4 but isn't? Only level 4 in special marketing stunts. It is not level 4 as a vehicle as a service. Glorified tram sounds humerous too me, and exaggerated language is common speak to get a point across. Doesn't matter if it has a ton of sensors and billions of flops in the cloud available, if it doesn't work, then it doesn't work.

 

[Bitdepth]

If we are talking about growing flowers is that scientific?

Yes very much so. Agricultural science and the food you eat everyday is based on research into cost effective way to mass grow, manufacture foods. Flower husbandry is big business.

It is only scientific because you deem it so. Others would call it a technical subject.

Thanks for the correction. The term i meant to use was technical, when you are having a technical discussion, your language should reflect as such.

Yes, not the same, but has same affect.

Do you see now why language is important?

What would you call something that is labeled as level 4 but isn't? Only level 4 in special marketing stunts. It is not level 4 as a vehicle as a service.

This is the sorta language that does not serve this discuss. It is just argument for argument's sake.

Glorified tram sounds humerous too me, and exaggerated language is common speak to get a point across.

It does not get any point across, it serves to reduce and simplify years of research. It does not move the discussion anywhere. And if you're not here to discuss then what are we here for?

Doesn't matter if it has a ton of sensors and billions of flops in the cloud available, if it doesn't work, then it doesn't work.

But it works. That's like saying Navigate on Autopilot doesn't work because when you pull out of your driveway it does not allow you to engage it until you meet certain criteria before you can engage it. Level 4 Autonomy is domain specific by definition.

 

[DanCar]

Yes very much so. Agricultural science and the food you eat everyday is based on research into cost effective way to mass grow, manufacture foods. Flower husbandry is big business.

Most people will disagree that discussion of planting flowers on a public forum should only be discussed in scientific or technical terms. And many will think that is a ridiculous assertion. Don't be arguing for arguments sake.

It does not get any point across, it serves to reduce and simplify years of research. It does not move the discussion anywhere. And if you're not here to discuss then what are we here for?

It serves to show that your beloved project is a failure, for past few years that they declared it would work. Hopefully they can turn it around, but I'm dubious. A boondoggle of a project so far.

 

[NHK X]

What you are doing with your "devils advocate" argument is called a strawman. For the most part in these threads some arguments are that Tesla's delay to achieve FSD is a sign that it is a lot harder than Elon Musk had led people to believe.

FSD was sold and promised to be released way back in 2016 "pending regulation". Since then, they've conceded that the hardware they had was not enough, they have redesigned the hardware and software stack several times with each improving on the previous.

Objectively looking at it, they broke away from using Mobile Eye and within months were able to achieve software parity with what they had with Mobile Eye. Autopilot is very impressive. But as we can now see that is the easy part and achieving level 4 FSD is a lot difficult.

To answer your question, Karparthy like most engineers i have known in my life love a good challenge. No doubt they are working on it every day as is everyone else in this field trying to tackle this problem. The goal for everyone is to be able to get the amount of sensors needed down to a cost effective price that can be sold to consumers while also solving the software side of it as well.

Yes if presented as an argument. Not my intent. The reasons for cynicism vary widely, and agree much comes from Tesla’s optimism vs how difficult the solution is.

This was presented more as a topic of discussion. Thanks for your input. Didn’t think about the idea of just wanting a challenge.

 

[mrkisskiss]

I've said it before, but I'll say it again... autonomous driving needs to be solved with a collaboration between infrastructure (governments or private construction) and car companies. It's mad to expect individual companies to "solve" this with software, and I don't care how many pictures of stop signs you collection and label.

The systems they're building are too individual, too proprietary - and insanely, dangerously fragile.

Hundreds of billions has already been spent on pursuing autonomy; pick your flavour of sensors or whatever, it's all irrelevant. I would imagine it'll easily surpass trillions of dollars of investment by the time we get somewhere useful.

Governments and car companies (and others) should work together to introduce a standard, safe, secure, programmable and expandable communications protocol for infrastructure and vehicles. Open up some spectrum...

Then, add transmitters and sensors into the infrastructure, and receivers into the cars. Why on earth do you need each car to (hopefully) recognise a stop sign - every time - in all weathers, times of day and so on. The cost and fragility is in the wrong place. Add a transmitter into the road (or nearby) that tells the car to stop there. It can't easily be faked, it can't easily be spoofed and it would never be wrong. Each car would still need a vision system and basic driving policy - but nothing much more complex than we've already got. It also allows for _global fleet_ control; in poor weather, the system could reduce or limit the speed of all the cars on the road.

"What about construction!" I hear you say... well, again, this would require some legislation. But, smart traffic poles/cones, combined with knowledge of where the construction happens (a permit) would be required and solve this fairly easily. Yep, there would be changes in approach needed, but that's just what's required for such a fundamental shift in transportation.

Then it'd be up to the governments to ensure the infrastructure is correct, maintained, and suitable for AVs:

Refresh the lane markings, get sensors in the road that confirm the lane number and the direction of travel - things to help (or even the centre line). Traffic lights become traffic control - the system can manage traffic flow, and avoid congestion by changing the speed of the road fleet as required to optimise (these are the sorts of tasks that computers are good at). Junctions could be redesigned, rebuilt and optimised for AVs.

Bring areas online gradually and the car will know automatically when and where it's safe to drive to autonomously. Even non-AV vehicles would then know where AVs are driving around.

Yes, it's expensive, but the cost responsibility is shared between us/the government, the car companies, construction companies and so on... It'd create a lot lot of jobs in construction (and maintenance) and this is a good thing.

Either way, I'm willing to bet it wouldn't be in the trillions of dollars to refresh infrastructure. , and would mean every car company could produce autonomous cars, which all play by standardised, approved rules, which would mean it would finally make sense to form legislation around it, and there we go.

To expect the single biggest shift in autonomous transportation to happen without any changes to infrastructure and no construction responsibility is totally bananas. In my opinion, of course

 

[MrBadger]

CAV technology is very far advanced. Quite possibly even partially deployed in some form or other - even if not enabled - we had code in traffic light controllers to handle some of the requirements awaiting release/validation and implementation.

Even a few years back, there were a few test/demo sites. Small example of some of the capabilities, Audi Traffic Light Information Feature Launched in Las Vegas - Automobile Magazine (loads of other articles, I picked that randomly from a list of search results) I believe Audi jumped the gun on this (especially wrt to data transmission) and at the point in time of this, global standards was still in state of flux. Not sure what state of play is now. I've been out of the wider loop for several years.

 

[mrkisskiss]

Absolutely. The technology and hardware to do this stuff properly isn't "years away, maybe", or rely on some form of "expected" leap in progress (or a leap of faith). It could be done with the technology of today. But, it would require proper international standards, a lot of construction, and a co-ordinated effort from governments and industry.

That, of course, is a whole other kettle of fish.

 

[MrBadger]

But, it would require proper international standards, a lot of construction, and a co-ordinated effort from governments and industry.

Don't under estimate the amount of work and collaboration that has already been put into this globally. Personally, I think construction/rollout is relatively trivial - just look how quick they rolled out new street lighting, but UTC (traffic control centre software) validation may take longer but can be done in parallel.

I think part of it was waiting for some key future proofing technology and security to become more widely accessible (its not like you can guarantee a mobile signal everywhere like the Audi system expects) and for the motor industry to have a desire to want it. We have had electric vehicles for over a hundred years but only recently has it started being taken seriously mainstream.

I use to work in independent transport research (speciality controlling air quality by influencing traffic flow), so have seen quite a lot of what industry and governments are interested in. There are lots of things being actively considered that may or may not become common place some point in the future, likewise lots of things being looked at for now.

 

[diamond.g]

2020.12.6 appears to bring traffic control to the fleet. So folks in the US should be able to give it a whirl.

 

[willow_hiller]

2020.12.6 appears to bring traffic control to the fleet. So folks in the US should be able to give it a whirl.

I wonder if this will contain the AP rewrite, or parts of it. Given what Karpathy showed us, I would not fully trust old Autopilot to navigate unmarked intersections without the improved bird's eye view the rewrite provides.

 

[diamond.g]

I wonder if this will contain the AP rewrite, or parts of it. Given what Karpathy showed us, I would not fully trust old Autopilot to navigate unmarked intersections without the improved bird's eye view the rewrite provides.

There is an intersection on my way to work where if I am in the right lane auto steer would “float” over to the left lane upon going across the intersection. I’ll have to try the next time I am going into the office.

 

[MP3Mike]

I wonder if this will contain the AP rewrite, or parts of it. Given what Karpathy showed us, I would not fully trust old Autopilot to navigate unmarked intersections without the improved bird's eye view the rewrite provides.

I'm pretty sure Elon said that this doesn't contain the rewrite. And that the rewrite would come with the reverse summon feature...

 

[mongo]

There's legitimate discussion and then there's disingenuous. Saying other SDC systems are just a glorified train because they use HD map is just that. Dishonest. Bad faith. illogical. Its like saying Tic Tac Toe is the same level of complexity as the game Go. You can't let statements like that stand. You have to call it out as it is. Maybe i have less patience because i have been responding to the same thing on this forum since basically 2016 and its still rinse and repeat. Nothing has changed. Its the same argument with lip gloss on it.

So is the following correct or not? Because that is what we are saying is a poor system design, regardless of whether any current company does or does not use it.

The short answer is because Waymo has centimetre level accurate maps and localization which allow it to trace a very precise path for the car to take when proceeding through an intersection or making a turn at an intersection. So with that precise path finding, Waymo cars can always take turns very reliably and very precisely.

 

[pongajim]

First, thanks for posting.

I definitely agree with you about HD maps. I do have a follow-up question: can't HD maps also be used for contextual information like labeling when a stop sign is only if you are turning right? For example, Karpathy mentions needing to use camera vision to understand when a stop sign only applies to certain situations but that seems like something that could be labeled on a HD map.

Based on this new talk, how would you assess Tesla's current state of FSD? What I get from the presentation is that Tesla is using existing machine learning techniques and slowly working through all the complexities and nuances of perception, planning and driving policy, trying to get it all done with vision and machine learning alone. It's taking them longer than they thought because of how complex the problem of autonomous driving is, especially with just camera vision.

Agree. The one reason they will be successful however is they have the fleet; no one even has a fleet to speak of.

 

[diplomat33]

So is the following correct or not? Because that is what we are saying is a poor system design, regardless of whether any current company does or does not use it.

How is that a poor system design? It seems to me that it makes the autonomous driving more reliable. The car is approaching an intersection. It knows it needs to make a right turn based on its nav directions. It takes in data from its HD map, coupled with data from its other sensors (cameras, lidar and radar) and is able to trace a precise path for navigating that right turn.

 

[mongo]

How is that a poor system design? It seems to me that it makes the autonomous driving more reliable. The car is approaching an intersection. It knows it needs to make a right turn based on its nav directions. It takes in data from its HD map, coupled with data from its other sensors (cameras, lidar and radar) and is able to trace a precise path for navigating that right turn.

Is it alway the same cm level path? That is what tracing (vs creating) a precise (as opposed to accurate) path ahead of time (as opposed to while manuvering and reacting) implies.
"Waymo has centimetre level accurate maps and localization which allow it to trace a very precise path for the car to take when proceeding through an intersection or making a turn at an intersection"

'Vision' vs maps (for fun ):

 

[diplomat33]

Is it alway the same cm level path? That is what tracing (vs creating) a precise (as opposed to accurate) path ahead of time (as opposed to while manuvering and reacting) implies.
"Waymo has centimetre level accurate maps and localization which allow it to trace a very precise path for the car to take when proceeding through an intersection or making a turn at an intersection"

I would think it is the same cm level path.

'Vision' vs maps (for fun ):

Thanks for sharing that clip. Hunt for Red October is one of my all-time favorite movies. And that scene does illustrate the advantages of having precise HD maps and localization. Ramius never could have pulled off that maneuver with "vision" alone.

 

[mongo]

I would think it is the same cm level path.

Thus making it like a train. Following the same path every time.

Thanks for sharing that clip. Hunt for Red October is one of my all-time favorite movies. And that scene does illustrate the advantages of having precise HD maps and localization. Ramius never could have pulled off that maneuver with "vision" alone.

There was no vision, other than their imagination. Mapping system: turn turn turn, off course, off course. Rameus as path planner: wait for it, wait for it.
If they had used any sensors, it would have been simple.

 

[diplomat33]

Thus making it like a train. Following the same path every time.

No. Trains are constricted to only stay on a single path and can't maneuver on their own. Waymo cars can maneuver independently on their own. Waymo cars are not constricted to just stay on a single path every time. The path on the HD map is simply to help the car make the maneuver more precisely. So for example, when the Waymo car approaches an intersection, it does not have to "guess" with vision alone how to make the turn but has map data to know how to make the turn accurately.

There was no vision, other than their imagination. Mapping system: turn turn turn, off course, off course. Rameus as path planner: wait for it, wait for it.
If they had used any sensors, it would have been simple.

Well they could have used active sonar to get distance to the undersea cliff. But active sonar works like lidar. Passive sonar (the equivalent of cameras) would not have worked in this scenarios.