FSD Beta v11.x

The next big milestone for FSD is 11. It is a significant upgrade and fundamental changes to several parts of the FSD stack including totally new way to train the perception NN.

From AI day and Lex Fridman interview we have a good sense of what might be included.

- Object permanence both temporal and spatial
- Moving from “bag of points” to objects in NN
- Creating a 3D vector representation of the environment all in NN
- Planner optimization using NN / Monte Carlo Tree Search (MCTS)
- Change from processed images to “photon count” / raw image
- Change from single image perception to surround video
- Merging of city, highway and parking lot stacks a.k.a. Single Stack

Lex Fridman Interview of Elon. Starting with FSD related topics.

Here is a detailed explanation of Beta 11 in "layman's language" by James Douma, interview done after Lex Podcast.

Here is the AI Day explanation by in 4 parts.

Deep Understanding Tesla FSD Part 1: HydraNet

From Theory to Reality, Analyze the Evolution of Tesla Full Self-Driving

saneryee-studio.medium.com

Here is a useful blog post asking a few questions to Tesla about AI day. The useful part comes in comparison of Tesla's methods with Waymo and others (detailed papers linked).

Twelve Questions about Tesla Autopilot Technologies Exposed at 2021 AI Day

On Aug. 19 2021, Telsa held an AI day , exposing more technologies about Autopilot , from perception, planning, data labeling, simulation…

yuhuang-63908.medium.com

 

[Supcom]

For those keep complaining the dry wipe not being fixed, and not as good as some older cars. I would like to offer the following:

The wiper function in a Tesla no longer just for cleaning due to rain. It also has the function of making sure the camera views are not obstructed due to dust, insects, fog, deteriorating pixels etc. With the recent updates of camera views color, view latencies, hardware 4 more resolution cameras etc. I suspect that Teslas has begun (if not already doing it from the beginning) to look into the cameras data more carefully. As a retired engineer and have worked on CCD sensors I can guess that Teslas has started to implement more checking on the integrity of the camera images. One of the fundamental engineering operation on a CCD is to calibrate each individual pixels performance (some high performance cameras have a pixel map of the entire sensor to compensate for individual gains and bad pixels). That's always done at the beginning of a picture or video being taken. So, if the algorithm and conditions of the camera implementation detected there maybe obstructions of the cameras view, it will try to wipe the windshield to make sure. A single wipe once in a while will not hurt the windshield as we all know that even with the current car wiper implementations, there are usually a few dry wipes before the wiper fluid pump began to start spraying. So be patient and see if they can find a better way to improve the performance of the vision system.

If the car was attempting to clean the windshield in front of the cameras, it would spray washer fluid.

 

[jebinc]

If the car was attempting to clean the windshield in front of the cameras, it would spray washer fluid.

That and camera clear line of sight should not be prioritized over the human driving line of sight….

 

[KLam]

If the car was attempting to clean the windshield in front of the cameras, it would spray washer fluid.

Don't want to argue here since I have no idea what Tesla is trying to do. But if one wipe clears the question of obstruction present. Why do more?

Or maybe that's another way of telling the driver to pay attention and put their hands on the steering wheel.

 

[KLam]

That and camera clear line of sight should not be prioritized over the human driving line of sight….

I don't think there is a prioritization here!

 

[PACEMD]

Unbiased means you report the facts, and let the reader come to their own conclusions. The moment you put in language that conveys an opinion, or attempts to sway the viewpoint of the reader, it's no longer unbiased.

I'll give you two examples:

A 5 year-old boy died when he was struck by a Tesla on the corner of 5th and Elm. Police are investigating the scene and we'll bring you more details when they become available.

A child was tragically struck and killed by a Tesla, possibly using Tesla's Autopilot, on the corner of 5th and Elm. Police are investigating the accident, and we'll bring you more details as they become available.

Notice the 2nd paragraph invokes emotional responses by using child, tragic, and possibly. I'm attempting to lean the reader towards anger, and prejudge the driver and/or Tesla without the facts.

Ok, ok, how about this...........Yet another 5-year-old child was run over and flattened like a bloody pancake by a Tesla as he stood innocently minding his own business on the corner of 5th and Elm Street. The Tesla was said to have been using FSDm, Full Self-Driving murderer. The auto wipers did not engage to clear blood from the not driver's line of sight. Elon Musk could not be reached for comment.

 

[jebinc]

I don't think there is a prioritization here!

I’m not sure what the reason may be for not wiping when it should be. I was out in drizzle, and in auto wipe mode, and it didn’t wipe when it needed to. I, on many occasions, to manually tell it to wipe the F’n windshield because I couldn’t see the damn road well enough. To each!

 

[jebinc]

Ok, ok, how about this...........Yet another 5-year-old child was run over and flattened like a bloody pancake by a Tesla as he stood innocently minding his own business on the corner of 5th and Elm Street. The Tesla was said to have been using FSDm, Full Self-Driving murderer. The auto wipers did not engage to clear blood from the not driver's line of sight. Elon Musk could not be reached for comment.

And when he was (finally reached), he proclaims, “FSD drives safer than a human…” He further professes, “FSD soon will be 10x better than a human driver, so GFY!!!”…

 

[KLam]

I’m not sure what the reason may be for not wiping when it should be. I was out in drizzle, and in auto wipe mode, and it didn’t wipe when it needed to. I, on many occasions, to manually tell it to wipe the F’n windshield because I couldn’t see the damn road well enough. To each!

Well, I don't know what to say. My 2015 Lexus 350 do that also. I often have to start and adjust the wiping speed on that car too. So I am not that sensitive to it. Also I am in Los Angeles, so I don't have to use the wiper that often.

 

[jebinc]

Also I am in Los Angeles, so I don't have to use the wiper that often.

Consider that one benefit of living in SoCal.

 

[LowlyOilBurner]

Yep. My dad's Toyota Prius had AEB with radar. It didn't stop or mitigate a bad accident caused by another car stopped in the middle of an intersection.

Believe it or not, plenty of OEMs have plenty of broken or misleading features. It's not just Tesla.

This. I owned two SPA Volvos (XC90 & XC60 Inscriptions) that had “AEB, Large animal detection”. They never stopped for deers. I did. At least on FSD, my car has identified, slowed, swerved and stopped for deer. That’s kinda neat.

 

[WilliamG]

For those keep complaining the dry wipe not being fixed, and not as good as some older cars. I would like to offer the following:

The wiper function in a Tesla no longer just for cleaning due to rain. It also has the function of making sure the camera views are not obstructed due to dust, insects, fog, deteriorating pixels etc. With the recent updates of camera views color, view latencies, hardware 4 more resolution cameras etc. I suspect that Teslas has begun (if not already doing it from the beginning) to look into the cameras data more carefully. As a retired engineer and have worked on CCD sensors I can guess that Teslas has started to implement more checking on the integrity of the camera images. One of the fundamental engineering operation on a CCD is to calibrate each individual pixels performance (some high performance cameras have a pixel map of the entire sensor to compensate for individual gains and bad pixels). That's always done at the beginning of a picture or video being taken. So, if the algorithm and conditions of the camera implementation detected there maybe obstructions of the cameras view, it will try to wipe the windshield to make sure. A single wipe once in a while will not hurt the windshield as we all know that even with the current car wiper implementations, there are usually a few dry wipes before the wiper fluid pump began to start spraying. So be patient and see if they can find a better way to improve the performance of the vision system.

That’s a pretty fanciful explanation. Can you then explain the wiping frenzy on a beautiful day (without windshield washer) and the lack of wiping to the point of it being dangerous without a manual wipe when it is actually raining?

 

[jebinc]

That’s a pretty fanciful explanation. Can you then explain the wiping frenzy on a beautiful day (without windshield washer) and the lack of wiping to the point of it being dangerous without a manual wipe when it is actually raining?

Not to mention the fine scratches this fine system places upon your windshield. A class leading system, right?!

 

[KLam]

That’s a pretty fanciful explanation. Can you then explain the wiping frenzy on a beautiful day (without windshield washer) and the lack of wiping to the point of it being dangerous without a manual wipe when it is actually raining?

Personally, I have not experienced a single dry wipe let alone a wiping frenzy on a beautiful day (that's plenty in LA). In that case, you may want to schedule a service appointment. If your car do that often, I suggest that you clean your wind shield and wiper often or spray wiper fluid when it happens.
In the case of auto wipe not coming on while raining, I mentioned that I have it on my other car too. So, it could be a detection problem. That case I just turn it on manually. Not a big deal! I have yet to go to Lexus service for diagnostics. There is no off the air update on that I know of.

 

[WilliamG]

Personally, I have not experienced a single dry wipe let alone a wiping frenzy on a beautiful day (that's plenty in LA). In that case, you may want to schedule a service appointment.
In the case of auto wipe not coming on while raining, I mentioned that I have it on my other car too. So, it could be a detection problem. That case I just turn it on manually. Not a big deal! I have yet to go to Lexus service for diagnostics. There is no off the air update on that I know of.

Okay. I get what you’re saying: It’s broken for you, but you don’t care. I mean, fair enough…

 

[EVNow]

I’m not sure what the reason may be for not wiping when it should be. I was out in drizzle, and in auto wipe mode, and it didn’t wipe when it needed to. I, on many occasions, to manually tell it to wipe the F’n windshield because I couldn’t see the damn road well enough. To each!

I almost always put it on manual now when it’s raining (which seems to be always in the last week or so).

 

[WilliamG]

I almost always put it on manual now when it’s raining (which seems to be always in the last week or so).

Same.

 

[SidetrackedSue]

Don't want to argue here since I have no idea what Tesla is trying to do. But if one wipe clears the question of obstruction present. Why do more?

Or maybe that's another way of telling the driver to pay attention and put their hands on the steering wheel.

If I'm parked and in camp mode and reading, why do I have to have my hands on the steering wheel?

(Someone else informed me that it was an undocumented 'mosquito swatting mode' to try and make me less annoyed!)

 

[AndrewZ]

I wouldn't even doubt that the wiper issues are the system trying to clean some non-existent thing from the windshield, maybe it's interpreting glints of sunlight as specks of stuff on the windshield. Why it happens when there isn't rain but tends to not wipe when there is rain is another question but...

Bigger question for me is why windshield wipers even need neural nets when there are already tried and tested solutions out there, feels like it's adding complexity rather than reducing it or you'd think the problem would be solved by now, but who knows.

 

[sleepydoc]

For those keep complaining the dry wipe not being fixed, and not as good as some older cars. I would like to offer the following:

The wiper function in a Tesla no longer just for cleaning due to rain. It also has the function of making sure the camera views are not obstructed due to dust, insects, fog, deteriorating pixels etc. With the recent updates of camera views color, view latencies, hardware 4 more resolution cameras etc. I suspect that Teslas has begun (if not already doing it from the beginning) to look into the cameras data more carefully. As a retired engineer and have worked on CCD sensors I can guess that Teslas has started to implement more checking on the integrity of the camera images. One of the fundamental engineering operation on a CCD is to calibrate each individual pixels performance (some high performance cameras have a pixel map of the entire sensor to compensate for individual gains and bad pixels). That's always done at the beginning of a picture or video being taken. So, if the algorithm and conditions of the camera implementation detected there maybe obstructions of the cameras view, it will try to wipe the windshield to make sure. A single wipe once in a while will not hurt the windshield as we all know that even with the current car wiper implementations, there are usually a few dry wipes before the wiper fluid pump began to start spraying. So be patient and see if they can find a better way to improve the performance of the vision system.

I've considered this, the problem is the facts just don't back it up. Example - sitting at an intersection. Cloudy day, no precipitation whatsoever, no change in lighting and the wipers suddenly turn on. I've also tried cleaning the glass over the cameras prior to leaving with no change in behavior. Now, is it possible a stray fleck of debris hit the windshield right over the camera leaving the rest of the windshield clear? Yes, but one would expect this to be an extremely rare event. The more common occurrence should be an even distribution over the entire windshield yet what we reliably see is a completely clear windshield with the wipers turning on.

Your hypothesis is also disproved by the converse - the wipers regularly fail to engage when there is precipitation.

None of us can definitively say that they aren't trying to improve camera vision but currently there's no good evidence to support it.

Personally, I have not experienced a single dry wipe let alone a wiping frenzy on a beautiful day (that's plenty in LA). In that case, you may want to schedule a service appointment. If your car do that often, I suggest that you clean your wind shield and wiper often or spray wiper fluid when it happens.

Honestly, this statement strains credulity. (others have made appointments and been told it will be addressed via software update, indicating that Tesla knows it's a software flaw)

 

[GWord]

I've also tried cleaning the glass over the cameras prior to leaving with no change in behavior.

My 7yr old Model S has dozens of little tiny nicks and chips in the glass that looks like little water drops under certain lighting conditions. Obviously not something that can be cleaned. With that said…. my wife’s new Model Y also has wiper shennanigans and that glass is pristine so that would seem to agree with you and indicate glass condition isn’t the sole root cause.