Welcome to Tesla Motors Club
Discuss Tesla's Model S, Model 3, Model X, Model Y, Cybertruck, Roadster and More.
Register

FSD AP improvements in upcoming v11 from Lex Fridman interview

This site may earn commission on affiliate links.
The recent Lex Fridman interview of Elon had some interesting tidbits about the next major FSD beta version (https://www.youtube.com/watch?v=DxREm3s1scA).

First, something simple. Tesla’s AI inference chip (each car has two of them) has an image signal processor that processes each image frame from the eight cameras, and the resulting processed image is what the car neural net then sees, and more importantly, what it was trained on. This processing is similar to what any digital camera does. The unprocessed image looks like what a raw image file looks like (for those photo buffs) and isn’t human useful since a human really can’t see much in the resulting image.

But that initial processing takes a huge 13 milliseconds. For a system that runs at a 27ms frame rate, that’s like half your time budget. Moreover, the processed image has a lot less data in it that the raw image (which is basically photon counts). In particular, the raw image would allow a computer to see much better in very low light situations than with a processed image.

So, Tesla is bypassing the image processor entirely in V11 and the neural net will use just the raw, photon count, images. Doing this means Tesla will have to completely retrain its neural nets from scratch since the input is so different. This is yet another example of “the best part is no part” thinking and should offer huge improvements since they will now have more accurate input and more time for other kinds of processing. BTW, the top left of this image of the Tesla inference chip is what they are bypassing. It isn’t a complete waste since they still need the processed images for sentry mode and whatnot, but it won’t be part of the critical FSD time loop.

r/teslamotors - FSD AP improvements in upcoming v11 from Lex Fridman interview
Tesla FSD inference chip

v11 will also push even more C code into the neural net. Currently, the neural net outputs what Elon called “a giant bag of points” that is labeled. Then C code turns that into vector space which is a 3D representation of the world outside the car. V11 will expand the neural net so that it will produce the vector space itself, leaving the C code with only the planning and driving portions. Presumably this will be both more accurate and faster.

Not all the neural nets in the car use the surround video pipeline yet. Some still process perception camera by camera, so that’s getting addressed as well.

In the end, lines of code will actually drop with this release.

Other interesting things. They have their own custom C compiler that generates machine code for the specific CPUs and GPUs on the AI chip. All the hardcore time sensitive code is written in C.

So, that’s the info Elon told us from the interview. As I wrote in my last deep dive about FSD (Layman's Explanation of Tesla AI Day), Tesla has a lot of optimizations it can still do, and this is an example of a couple of them.
Time sensitive code is written in the native machine language unless the C Compiler produces very tight code
 
Yes the C optimizer surpassed the ability of a human assembly language programmer to optimize code a good 25 years ago.
That's not what that link is saying though, it's arguing that "C is faster than assembly" in the sense that the programmer can run experiments more quickly in a high-level language (i.e. compile with loop unrolled vs not unrolled) and view the generated assembly code much faster than writing the assembly directly. The "faster" part is the speed in which the programmer can make adjustments and test them out, not that the code is faster. It helps you get to faster code more quickly, trying out different approaches that would takes ages to test in hand-coded assembly.

If you have all the time in the world, you can definitely write faster hand-tuned assembly code than a C compiler, it's just no one has time for that.
 
  • Like
Reactions: pilotSteve
That's not what that link is saying though, it's arguing that "C is faster than assembly" in the sense that the programmer can run experiments more quickly in a high-level language (i.e. compile with loop unrolled vs not unrolled) and view the generated assembly code much faster than writing the assembly directly. The "faster" part is the speed in which the programmer can make adjustments and test them out, not that the code is faster. It helps you get to faster code more quickly, trying out different approaches that would takes ages to test in hand-coded assembly.

If you have all the time in the world, you can definitely write faster hand-tuned assembly code than a C compiler, it's just no one has time for that.
I will admit I didn’t read the article. But my point stands…compilers will outperform hand optimized code. They can detect patterns at a greater scale than one programmer can keep in their mind. Sure a programmer and unroll a loop or elide a copy. But the compiler can optimize caches across more code than a programmer can possibly comprehend.
 
  • Informative
Reactions: wknickless
Sounds like they are very close to what will be v11 already. However, as far as I can tell, they are still working to finish implementing the overall 8-camera vision system and hydranets that Karpathy talked about like two or three years ago. I have always been skeptical of how far FSD can go since I started driving on EAP/NOA in 2018, but I am still surprised that these "foundational rewrites" are taking so long. And now, yet again, Elon is predicting they won't be done until the end of the year (2022), which makes me think they still aren't close to "complete," whatever that means. At some point I imagine they will accept the limitations of the current sensor suite, pop a bottle of champagne and call it "done," and then move on to HW4 or HW5 with new sensors that will be the base on the L5/Robotaxi that Elon has been promising. Maybe by the end of 2023? ;)
Wow. Didn't they start over with a rewrite in 2022?

Meanwhile the competition has fairly useful systems built on maps and sensors. FSD seems like an experiment that will never end. I know, it's somewhat useful for a lot of people. Elon should be resigned to giving refunds pretty soon.
 
  • Like
Reactions: ArchHamster
Wow. Didn't they start over with a rewrite in 2022?

Meanwhile the competition has fairly useful systems built on maps and sensors. FSD seems like an experiment that will never end. I know, it's somewhat useful for a lot of people. Elon should be resigned to giving refunds pretty soon.
Quite a few already have through small claims courts, according to some users here.