FSD v12.x (end to end AI)

[AndrewZ]

Unless it's written that the driver is responsible. To take liability for pothole damage is not gonna happen.

Yeah well you certainly wouldn’t want to own liability for millions of robotaxis driving around hitting potholes, I give my fiancé heck for vaguely curbing the right rim every now and then

 

[Mullermn]

.....and how is this accomplished?

Different frequencies are used which are easier for humans to locate directionally.

In the uk these can take the form of what sounds like a normal siren which switches to what sounds like white noise occasionally. Incredibly easy to tell where it’s coming from.

 

[jeewee3000]

The front cameras have a better view of potholes then humans do since they are higher so it should be doable for the really large potholes.

True.

Biggest challenge for humans is avoiding potholes at night in the rain so those are the real edge cases. Pothole locations will never be mapped reliably since they pop up suddenly and when large get patched pretty quickly since towns want to avoid liability.

To be fair, FSD shouldn't be held to a higher standard than humans. At night you can still see potholes on lit roads or when driving at slower speeds on pitch black roads (one must always adjust one's speed to the situation).

If you are driving on a faster road (say a freeway or straight road through a forest) where only the cars headlights are lighting the road, and you would hit a pothole of large enough size that it would damage your wheel, the local government can be held liable for the road conditions.

As someone in Europe that hasn't seen anything close to FSDbeta city driving but only using handicapped "Euro-FSD-Autopilot", I'm not too worried about these concerns (potholes, school zones, speed bumps, etc).

My main concerns with FSD ever working:
- can the car pass through tight gaps on vision only without ever hitting parked cars/obstacles/oncoming traffic? (many city roads require a European driver to have a great feel for the width of his vehicle, left mirror on left mirror is one of the most frequent of accidents). This is an instance where I don't trust vision will ever be able to measure up to the centimeter, which is definitely needed or FSD will have to stay out of cities.

- can the car learn the rules of the road properly, knowing at what intersections it has to yield and what intersections it has the right of way. I had to roll my eyes hard when Omar said in a recent FSD v12 video that San Francisco is a great testbed for FSD since it has the weirdest situations so if FSD works there it works everywhere. Talk about reality distortion. Not only should he check out South American or Indian roads, but even the more developed EU roads - that are formed historically instead of the grid-town-layouts in the US - provide way more edge cases than San Francisco that seems to be 90% four-way-stop-sign-intersections and roundabouts. Over here many intersections have no traffic lights, signs or road markings. You just have to know (from earlier memory) if you're on a main road (in which case the B-roads have to yield to you) or if you're not on a main road (in which case the right side has right of way).

If FSD v12 is only trained on 10 second clips, this last item should seem near impossible to solve, since one sometimes sees a sign that means 'main road' and only a minute later one comes to an intersection without any markings/signs. FSD v12 needs to learn a pattern but how will it learn a pattern this way?

This is why I think the current E2E architecture might be unfit for all driving conditions. Unless they train on full drive-videos (say 15 mins long), but the compute/data storage needed would explode. I was generally expecting rule based coding to stay in the game because of this (in the sense of learning the rules of the road. They should be hard coded IMO since not everything can be gleaned from video data. (or I'm too uninformed to realize how, could be the case.))

 

[zoomer0056]

Omar said in a recent FSD v12 video that San Francisco is a great testbed for FSD since it has the weirdest situations so if FSD works there it works everywhere.

Yup, he has it very easy and can't see beyond SF. Maybe his own NN needs some video training from another part of the world.

 

[Todd Burch]

Yup, he has it very easy and can't see beyond SF. Maybe his own NN needs some video training from another part of the world.

Agree. For an urban area, San Francisco has some of the easiest roads to navigate in the US. Very squared-off, tons of low-speed 4-way stops. It's not a coincidence that all the self-driving companies started out testing there.

 

[Tronguy]

True.


To be fair, FSD shouldn't be held to a higher standard than humans. At night you can still see potholes on lit roads or when driving at slower speeds on pitch black roads (one must always adjust one's speed to the situation).

If you are driving on a faster road (say a freeway or straight road through a forest) where only the cars headlights are lighting the road, and you would hit a pothole of large enough size that it would damage your wheel, the local government can be held liable for the road conditions.

As someone in Europe that hasn't seen anything close to FSDbeta city driving but only using handicapped "Euro-FSD-Autopilot", I'm not too worried about these concerns (potholes, school zones, speed bumps, etc).

My main concerns with FSD ever working:
- can the car pass through tight gaps on vision only without ever hitting parked cars/obstacles/oncoming traffic? (many city roads require a European driver to have a great feel for the width of his vehicle, left mirror on left mirror is one of the most frequent of accidents). This is an instance where I don't trust vision will ever be able to measure up to the centimeter, which is definitely needed or FSD will have to stay out of cities.

- can the car learn the rules of the road properly, knowing at what intersections it has to yield and what intersections it has the right of way. I had to roll my eyes hard when Omar said in a recent FSD v12 video that San Francisco is a great testbed for FSD since it has the weirdest situations so if FSD works there it works everywhere. Talk about reality distortion. Not only should he check out South American or Indian roads, but even the more developed EU roads - that are formed historically instead of the grid-town-layouts in the US - provide way more edge cases than San Francisco that seems to be 90% four-way-stop-sign-intersections and roundabouts. Over here many intersections have no traffic lights, signs or road markings. You just have to know (from earlier memory) if you're on a main road (in which case the B-roads have to yield to you) or if you're not on a main road (in which case the right side has right of way).

If FSD v12 is only trained on 10 second clips, this last item should seem near impossible to solve, since one sometimes sees a sign that means 'main road' and only a minute later one comes to an intersection without any markings/signs. FSD v12 needs to learn a pattern but how will it learn a pattern this way?

This is why I think the current E2E architecture might be unfit for all driving conditions. Unless they train on full drive-videos (say 15 mins long), but the compute/data storage needed would explode. I was generally expecting rule based coding to stay in the game because of this (in the sense of learning the rules of the road. They should be hard coded IMO since not everything can be gleaned from video data. (or I'm too uninformed to realize how, could be the case.))

Um. So, I drive around quite a bit in Middlesex County in NJ which very definitely has colonial-style roads (i.e., roads at all sorts of funny angles) and in the greater Boston, Massachusetts area. While there are relatively (i.e., 1850's) areas where there are straight roads, most of the greater Boston area was settled in the 1600's-1700's and the roads are.. dinky, funny angles, multiple roads at intersections, and so on. There's a lot of two-way streets with parking on both sides, resulting in one lane down the middle, which leads to Fun when trying to drive through these places. And downtown Boston (motto: Our streets were laid out by drunken cows!) are Other, at least by US standards. As one goes further west, streets tend to get more rectilinear.

Having said all that, Europe, in general, has "streets" that were laid out who-knows-when, some of which probably date back to pre-CE dates. In the worst case, some were laid out for foot traffic, only; others for the occasional horse, and some for Roman Legions. Yeah, there are plenty of decent roads, even by US standards, but, as you point out, inner cities (and maybe not even inner) are typically narrower than what one will see in the US.

For what it's worth: There are Youtube videos of Teslas having narrow spots to get through on FSD, and the car will fold mirrors in order to get through. I've had this happen, maybe, once in six years of driving Teslas and three years of FSD-b. But the car does seem to know where the edges are. (The car that did that had ultrasonic sensors; don't know how the vision versions would handle it.) In any case: When driving around, if there's a garbage truck or delivery van, FSD-b at present looks for opposing traffic and, if there isn't any, smoothly goes around said semi-parked, or double-parked vehicles.

One thing for sure: The more lines painted on the street, the better FSD-b does. On non-marked streets 11.4.9 has a bit of a bias towards the right (good for LHD cars), but will go down the middle of a street where most drivers tend to move a lot more to the right. It does move over for opposing traffic.

Going back to Boston for a minute: There are these roads, as mentioned, that are two-way but are, effectively, one lane when there's cars parked on both sides. When there's no traffic I've run FSD-b through that without trouble. Mostly, when there is traffic, I intervene on general principles. There's been a couple of incidents where I let the car do its thing while keeping a close eye on the opposing traffic; mostly, the other driver stops and lets the Tesla through, but that's not being scared of the Tesla, that's just the locals handling the situation as per normal. Since I don't do this on a regular basis, I'm not sure how the Tesla would react to a car coming at me a bit more aggressively. I'd guess the 11.4.9 software would stop, but haven't actually tried it yet. so can't say.

As regards the general idea that, "With this hardware, the car will never/very far in the future/not this year be able to handle the roads".. I'm more of a glass half-full type and hopeful that the next year will bring some real improvements. We'll see.

 

[Dewg]

My main concerns with FSD ever working:
- can the car pass through tight gaps on vision only without ever hitting parked cars/obstacles/oncoming traffic? (many city roads require a European driver to have a great feel for the width of his vehicle, left mirror on left mirror is one of the most frequent of accidents). This is an instance where I don't trust vision will ever be able to measure up to the centimeter, which is definitely needed or FSD will have to stay out of cities.

I agree with the sentiment here, but not the logic. Can humans negotiate the scenario you laid out? With just two eyes (cameras)? Before anyone jumps in with "we have stereoscopic vision", I'll remind you that there are many people licensed on the road today with mono-vision due to an injury or birth defect. I'll even suggest an experiment: try driving for a bit with one eye closed. Do you crash into things?

can the car learn the rules of the road properly, knowing at what intersections it has to yield and what intersections it has the right of way. I had to roll my eyes hard when Omar said in a recent FSD v12 video that San Francisco is a great testbed for FSD since it has the weirdest situations so if FSD works there it works everywhere. Talk about reality distortion. Not only should he check out South American or Indian roads, but even the more developed EU roads - that are formed historically instead of the grid-town-layouts in the US - provide way more edge cases than San Francisco that seems to be 90% four-way-stop-sign-intersections and roundabouts. Over here many intersections have no traffic lights, signs or road markings. You just have to know (from earlier memory) if you're on a main road (in which case the B-roads have to yield to you) or if you're not on a main road (in which case the right side has right of way).

It's an unfortunate trait that many Americans share: a lack of global perspective. I think you'll find that when many Americans say "everywhere", they, unfortunately, mean in the US. Sad, but true.

 

[JB47394]

I'll even suggest an experiment: try driving for a bit with one eye closed. Do you crash into things?

Motion parallax is the big one there, but there are other cues.

Depth perception - Wikipedia

en.wikipedia.org

Monocular cues include relative size (distant objects subtend smaller visual angles than near objects), texture gradient, occlusion, linear perspective, contrast differences, and motion parallax.

 

[PianoAl]

Before anyone jumps in with "we have stereoscopic vision", I'll remind you that there are many people licensed on the road today with mono-vision due to an injury or birth defect. I'll even suggest an experiment: try driving for a bit with one eye closed. Do you crash into things?

Stereopsis (using two eyes to judge depth) only works for distances below about 20 feet.

I've been blind in one eye since age 7. No problems driving.

 

[Scott Chapman]

am I the only one that is freaked out that we still don't seem to know what version FSD is in 2024.2.2 ?

 

[jeewee3000]

Um. So, I drive around quite a bit in Middlesex County in NJ which very definitely has colonial-style roads (i.e., roads at all sorts of funny angles) and in the greater Boston, Massachusetts area. While there are relatively (i.e., 1850's) areas where there are straight roads, most of the greater Boston area was settled in the 1600's-1700's and the roads are.. dinky, funny angles, multiple roads at intersections, and so on. There's a lot of two-way streets with parking on both sides, resulting in one lane down the middle, which leads to Fun when trying to drive through these places. And downtown Boston (motto: Our streets were laid out by drunken cows!) are Other, at least by US standards. As one goes further west, streets tend to get more rectilinear.

Having said all that, Europe, in general, has "streets" that were laid out who-knows-when, some of which probably date back to pre-CE dates. In the worst case, some were laid out for foot traffic, only; others for the occasional horse, and some for Roman Legions. Yeah, there are plenty of decent roads, even by US standards, but, as you point out, inner cities (and maybe not even inner) are typically narrower than what one will see in the US.

For what it's worth: There are Youtube videos of Teslas having narrow spots to get through on FSD, and the car will fold mirrors in order to get through. I've had this happen, maybe, once in six years of driving Teslas and three years of FSD-b. But the car does seem to know where the edges are. (The car that did that had ultrasonic sensors; don't know how the vision versions would handle it.) In any case: When driving around, if there's a garbage truck or delivery van, FSD-b at present looks for opposing traffic and, if there isn't any, smoothly goes around said semi-parked, or double-parked vehicles.

One thing for sure: The more lines painted on the street, the better FSD-b does. On non-marked streets 11.4.9 has a bit of a bias towards the right (good for LHD cars), but will go down the middle of a street where most drivers tend to move a lot more to the right. It does move over for opposing traffic.

Going back to Boston for a minute: There are these roads, as mentioned, that are two-way but are, effectively, one lane when there's cars parked on both sides. When there's no traffic I've run FSD-b through that without trouble. Mostly, when there is traffic, I intervene on general principles. There's been a couple of incidents where I let the car do its thing while keeping a close eye on the opposing traffic; mostly, the other driver stops and lets the Tesla through, but that's not being scared of the Tesla, that's just the locals handling the situation as per normal. Since I don't do this on a regular basis, I'm not sure how the Tesla would react to a car coming at me a bit more aggressively. I'd guess the 11.4.9 software would stop, but haven't actually tried it yet. so can't say.

As regards the general idea that, "With this hardware, the car will never/very far in the future/not this year be able to handle the roads".. I'm more of a glass half-full type and hopeful that the next year will bring some real improvements. We'll see.

Thanks for the history lesson. It is indeed true not the entire USA consists of only "new" grid layout road systems. Apologies for the generalization.

Regarding the mirror folding, I've seen that but it was always on cars equipped with USS. Still looking for an example on a Vision only test car.

 

[DarkandStormy]

If it was a complete re-write why is it called FSD v12 and not FSD 2.0 v1?

Also if it's completely different, they're starting from scratch and the millions upon millions of people used in all prior versions are useless.

 

[diplomat33]

If it was a complete re-write why is it called FSD v12 and not FSD 2.0 v1?

It is just a naming convention. FSD beta version numbers have been going sequentially, 8, 9, 10, 11, 12. To go to FSD 2.0 v1 would be to completely change the naming convention.

And Elon decided that if the re-write was deemed significant enough, it would be numbered a whole integer above. So when they did this end-to-end rewrite, he decided that it was a big enough re-write to warrant the 12 label.

 

[aronth5]

While avoiding potholes is not the near the top of the list for Tesla to fix when I drive to my son's house once it's pothole season I disable FSD a mile from his house. It's not worth the 50/50 chance of damaging the tire/rim. Last night the town's DPW Dept. put out a number of orange cones in the holes so it was easier to avoid the worst potholes. In a couple of days they will be patched.

 

[aronth5]

Excellent comparison of V11 driven by AI Driver and V12 by Omar. Video is narrated by AI Driver. V12 certainly did a much better job than V11 in fairly heavy vehicle and pedestrian traffic. If you cannot watch the entire video start around 8:30. Definitely makes me look forward even more to trying V12.

 

[AlanSubie4Life]

Excellent comparison of V11 driven by AI Driver and V12 by Omar. Video is narrated by AI Driver. V12 certainly did a much better job than V11 in fairly heavy vehicle and pedestrian traffic. If you cannot watch the entire video start around 8:30. Definitely makes me look forward even more to trying V12.

Who knows what the sliders are set to though…. But maybe the FSD team will YOLO the public v12 release! We can hope.

 

[Twiglett]

.....and how is this accomplished?

This for one example, but there are many others

DIRECTIONAL EMERGENCY SIRENS LTD

Fire & Rescue Sirens Ltd

www.directionalsirens.com

 

[KLam]

This is in Chinese, but with English caption. It's mainly on investment, but the interviewed Tesla FSD Engineer reviewed quite a bit on Tesla's AI/FSD effort. Enjoy.

 

[spacecoin]

This is in Chinese, but with English caption. It's mainly on investment, but the interviewed Tesla FSD Engineer reviewed quite a bit on Tesla's AI/FSD effort. Enjoy.

Propaganda in Chinese is PROPAGANDA in English.

 

[KLam]

Propaganda in Chinese is PROPAGANDA in English.

What propaganda? You have not viewed the clip yet!!