Once perspective and lens corrected, images of the same object from different cameras should not vary much hence all can be fed to same NN. I always thought that such correction was part of the calibration.
In place of higher resolution sensors Tesla is using different focal lengths for front vision. Shutter speed especially in low light is more important than anything else.
Sensor size and quality can also make a big difference in low light performance. If one brand of relatively cheap 4K camera has poor low light performance, that doesn't necessarily imply that all 4K cameras will have poor low light performance.
Please make a single reply to @LordAstinus's post on Reddit, perhaps to his shout-out to you: Amazing post on big Neural Network Changes in V9 - Potentially and order of magnitude or more better • r/teslamotors and you'll promptly gain more than enough karma to post articles in the future!
Actually dynamic range on their aptina sensors is pretty good and works decently well in reasonable low light (i.e. it depends on headlights and such). This was discussed back in the hw2.0 camera thread ages ago. Also keep in mind the dashcam thingie overcompresses the video and cuts the dynamic range considerably.
I am not sure about that. You can see below video at 7:49. The car "wobbles". V8 didn't do this. V9 has more realistic lane approach than V8, but it may just cause the problem - following the inaccurate lane instead of going straight.
Nice video. It's great that you're demonstrating stuff in such an easy to digest format. The camera angles are amazing. I've seen a few places myself where the new lane representation leads to behavior that differs significantly from V8. Not all of those are improvements. On balance it's been a substantial win for me, but I'm only one guy with one situation and I live where there are tons of Teslas. I'm sure there are edge cases that I'm not well acquainted with and people who have different priorities. I meant it as opinion and I hope it came across that way. My experience with how AP changes suggests that the new warts will get cleaned up.
High-level thought that's been recurring to me. In a lecture on YouTube, Andrew Ng said when you're stuck on a deep learning problem, there are always two things you can try: A bigger model A bigger dataset So, it makes sense that Tesla is training bigger models on bigger datasets. Why would anyone do anything else?
Also one of the Reddit /r/teslamotors moderators wrote a comment offering to approve your posts manually: Amazing post on big Neural Network Changes in V9 - Potentially and order of magnitude or more better : teslamotors "Jimmy! If you see this, PM me. It only takes a few comments elsewhere on reddit to not receive the automod message anymore but we can work to approve/upvote your comments to make it permanent. I pinged him." But just a single comment from you anywhere in that thread should be enough to propel your Reddit karma high enough!
The fact I can relate right now is that this network is present in the V9 software distribution. It might drive the car, or it might be some kind of test NN that is only enabled in some vehicles or under certain circumstances. Possibly it runs in parallel with another network or networks that does the real driving. If this latter item is true then it could be an evaluation for the FSD perception system. I understand that some people who have the ability to observe V9 in operation might be looking into this, so maybe we will find out. I myself only analyze what's in files that other people have shared with me. One item that might support the interpretation that this particular network is part of the FSD effort is the compute requirements of this network (which is incidentally named AKNET_V9 in the architecture file). It requires so much computation that you might not be able to run it at a fast enough framerate on the current APE hardware. If that is the case then this network would have to be run on Tesla's own "NN computer" in HW3 which, according to the last earnings call, is already being driven around in some Tesla vehicles already. Elon has said that HW3 is planned for FSD so maybe those are related. Another fact I can attest to is that there is another set of networks which is compiled into the vision binary. This was true in V8 and it is still true in V9. Based on symbol names and blob sizes that set of networks has separate weights and different network types for each kind of camera (one for pillars, one for repeaters, one for narrow...). Those networks support all 8 cameras, collectively, and include variations for 3 and S/X. These 'binary embedded' networks are substantially different from the ones that were present in the V8 software and also quite different from AKNET_V9. In time I hope to tease out more details about them from the vision binary but for now that's about all the solid data I have.
Sorry if i've missed this, but are we assuming that "AKNET" is for Andrej Karpathy or perhaps its for Alex Krizhevsky who created the infamous AlexNet? Works either way i suppose (nice for Andrej)
Thanks @jimmy_d! Very interesting stuff! I do have a question that is not directly related to the NNs (I guess) but to V9 and maybe you've got an answer to that. Do you think, the reason only HW2.5 got the dash cam feature is because of the limited color channels on HW2.0 cameras or because of computing power?
Another question arose. As of my knowledge HW2.0 cameras finally only have 2 channels, grey and red. How is the NN dealing with that?
"To be clear, actual NN improvement is significantly overestimated in this article. V9.0 vs V8.1 is more like a ~400% increase in useful ops/sec due to enabling integrated GPU & better use of discrete GPU." Elon Musk on Twitter