Neural Networks

[pkodali]

I was surprised to read that Tesla is not using 4K cameras, but then again, there's a trade off between the amount of detail that can be seen vs processing power required...

Also really poor low light performance, as some of the new blackvue dashcam have been showing

 

[Vitold]

Once perspective and lens corrected, images of the same object from different cameras should not vary much hence all can be fed to same NN. I always thought that such correction was part of the calibration.

 

[Vitold]

I was surprised to read that Tesla is not using 4K cameras, but then again, there's a trade off between the amount of detail that can be seen vs processing power required...

In place of higher resolution sensors Tesla is using different focal lengths for front vision.

Shutter speed especially in low light is more important than anything else.

 

[Joel Weber]

Sensor size and quality can also make a big difference in low light performance. If one brand of relatively cheap 4K camera has poor low light performance, that doesn't necessarily imply that all 4K cameras will have poor low light performance.

 

[DGDanforth]

"Monster" is not a scientific term. Please don't use "click bait" on this forum.

 

[Fact Checking]

I’ve tried in the past. Not enough karma so it won’t let me post.

Please make a single reply to @LordAstinus's post on Reddit, perhaps to his shout-out to you:

Amazing post on big Neural Network Changes in V9 - Potentially and order of magnitude or more better • r/teslamotors

and you'll promptly gain more than enough karma to post articles in the future!

 

[jimmy_d]

Please make a single reply to @LordAstinus's post on Reddit, perhaps to his shout-out to you:

Amazing post on big Neural Network Changes in V9 - Potentially and order of magnitude or more better • r/teslamotors

and you'll promptly gain more than enough karma to post articles in the future!

Thanks for the suggestion.

 

[verygreen]

Also really poor low light performance, as some of the new blackvue dashcam have been showing

Actually dynamic range on their aptina sensors is pretty good and works decently well in reasonable low light (i.e. it depends on headlights and such). This was discussed back in the hw2.0 camera thread ages ago.

Also keep in mind the dashcam thingie overcompresses the video and cuts the dynamic range considerably.

 

[EvanLin]

Lateral control is tighter and it’s been able to beat all the V8 failure scenarios I’ve collected over the last 6 months.

I am not sure about that. You can see below video at 7:49. The car "wobbles". V8 didn't do this.
V9 has more realistic lane approach than V8, but it may just cause the problem - following the inaccurate lane instead of going straight.

 

[jimmy_d]

I am not sure about that. You can see below video at 7:49. The car "wobbles". V8 didn't do this.
V9 has more realistic lane approach than V8, but it may just cause the problem - following the inaccurate lane instead of going straight.

Nice video. It's great that you're demonstrating stuff in such an easy to digest format. The camera angles are amazing.

I've seen a few places myself where the new lane representation leads to behavior that differs significantly from V8. Not all of those are improvements. On balance it's been a substantial win for me, but I'm only one guy with one situation and I live where there are tons of Teslas. I'm sure there are edge cases that I'm not well acquainted with and people who have different priorities. I meant it as opinion and I hope it came across that way. My experience with how AP changes suggests that the new warts will get cleaned up.

 

[strangecosmos]

High-level thought that's been recurring to me. In a lecture on YouTube, Andrew Ng said when you're stuck on a deep learning problem, there are always two things you can try:

• A bigger model
• A bigger dataset

So, it makes sense that Tesla is training bigger models on bigger datasets. Why would anyone do anything else?

 

[Fact Checking]

Thanks for the suggestion.

Also one of the Reddit /r/teslamotors moderators wrote a comment offering to approve your posts manually:

Amazing post on big Neural Network Changes in V9 - Potentially and order of magnitude or more better : teslamotors

"Jimmy! If you see this, PM me. It only takes a few comments elsewhere on reddit to not receive the automod message anymore but we can work to approve/upvote your comments to make it permanent. I pinged him."​

But just a single comment from you anywhere in that thread should be enough to propel your Reddit karma high enough!

 

[jimmy_d]

Thanks for the amazing write up!!!!

Would you say that this monster NN in V9 is the NN for "FSD"? I am not saying that the cars have FSD but it would seem like this new monster NN would be the NN that FSD will eventually use. The fact that it is so big and uses all 8 cameras, would seem like a good indication that FSD will use the V9 NN. If so, that is pretty exciting if with V9, we have switched from the "strictly EAP" NN to a "FSD capable" NN. Or do you think this just a "pre-FSD" NN and we will get yet another new NN when V10 comes out that will be the "FSD" NN?

The fact I can relate right now is that this network is present in the V9 software distribution. It might drive the car, or it might be some kind of test NN that is only enabled in some vehicles or under certain circumstances. Possibly it runs in parallel with another network or networks that does the real driving. If this latter item is true then it could be an evaluation for the FSD perception system. I understand that some people who have the ability to observe V9 in operation might be looking into this, so maybe we will find out. I myself only analyze what's in files that other people have shared with me.

One item that might support the interpretation that this particular network is part of the FSD effort is the compute requirements of this network (which is incidentally named AKNET_V9 in the architecture file). It requires so much computation that you might not be able to run it at a fast enough framerate on the current APE hardware. If that is the case then this network would have to be run on Tesla's own "NN computer" in HW3 which, according to the last earnings call, is already being driven around in some Tesla vehicles already. Elon has said that HW3 is planned for FSD so maybe those are related.

Another fact I can attest to is that there is another set of networks which is compiled into the vision binary. This was true in V8 and it is still true in V9. Based on symbol names and blob sizes that set of networks has separate weights and different network types for each kind of camera (one for pillars, one for repeaters, one for narrow...). Those networks support all 8 cameras, collectively, and include variations for 3 and S/X. These 'binary embedded' networks are substantially different from the ones that were present in the V8 software and also quite different from AKNET_V9. In time I hope to tease out more details about them from the vision binary but for now that's about all the solid data I have.

 

[diplomat33]

The fact I can relate right now is that this network is present in the V9 software distribution. It might drive the car, or it might be some kind of test NN that is only enabled in some vehicles or under certain circumstances. Possibly it runs in parallel with another network or networks that does the real driving. If this latter item is true then it could be an evaluation for the FSD perception system. I understand that some people who have the ability to observe V9 in operation might be looking into this, so maybe we will find out. I myself only analyze what's in files that other people have shared with me.

One item that might support the interpretation that this particular network is part of the FSD effort is the compute requirements of this network (which is incidentally named AKNET_V9 in the architecture file). It requires so much computation that you might not be able to run it at a fast enough framerate on the current APE hardware. If that is the case then this network would have to be run on Tesla's own "NN computer" in HW3 which, according to the last earnings call, is already being driven around in some Tesla vehicles already. Elon has said that HW3 is planned for FSD so maybe those are related.

Another fact I can attest to is that there is another set of networks which is compiled into the vision binary. This was true in V8 and it is still true in V9. Based on symbol names and blob sizes that set of networks has separate weights and different network types for each kind of camera (one for pillars, one for repeaters, one for narrow...). Those networks support all 8 cameras, collectively, and include variations for 3 and S/X. These 'binary embedded' networks are substantially different from the ones that were present in the V8 software and also quite different from AKNET_V9. In time I hope to tease out more details about them from the vision binary but for now that's about all the solid data I have.

Thanks for the detailed response.

 

[jimmy_d]

Thanks for the detailed response.

I hope that was useful in proportion to how long it took to read.

 

[kbecks13]

which is incidentally named AKNET_V9 in the architecture file

Sorry if i've missed this, but are we assuming that "AKNET" is for Andrej Karpathy or perhaps its for Alex Krizhevsky who created the infamous AlexNet? Works either way i suppose (nice for Andrej)

 

[KyleDay]

Sorry if i've missed this, but are we assuming that "AKNET" is for Andrej Karpathy or perhaps its for Alex Krizhevsky who created the infamous AlexNet? Works either way i suppose (nice for Andrej)

It's definitely Karpathy - the wizard behind the curtain.

 

[bebbiXpress]

Thanks @jimmy_d! Very interesting stuff!

I do have a question that is not directly related to the NNs (I guess) but to V9 and maybe you've got an answer to that. Do you think, the reason only HW2.5 got the dash cam feature is because of the limited color channels on HW2.0 cameras or because of computing power?

 

[bebbiXpress]

all cameras: 3 color channels, 2 frames (2 frames also has very interesting implications)

Another question arose. As of my knowledge HW2.0 cameras finally only have 2 channels, grey and red. How is the NN dealing with that?

 

[conman]

"To be clear, actual NN improvement is significantly overestimated in this article. V9.0 vs V8.1 is more like a ~400% increase in useful ops/sec due to enabling integrated GPU & better use of discrete GPU."
Elon Musk on Twitter