Neural Networks

[S4WRXTTCS]

Did one of you guys cheat on each other?? Trent Change your avatar picture please...

Keep up the awesome drama guys, it’s great reading and currently I’m more sympathetic and leaning towards bladder but I love jimmy’s info so.....

I’m confused

The problem is neither of these people own the car.

So their entire perspective is often skewed towards things that have absolutely no short or medium term value to any owner.

I rarely agree with extreme positions, and odds are pretty high that people acting from extremely positions aren't doing us any favors. In fact the best way to disrupt democracy seems to be planting extreme positions on social media because this causes people to lose common ground.

I like Jimmy because while he might go out on a limb here, and there he is clear about the limits of his knowledge.

I try not to make too much of it especially since there isn't really anyone that can vet it. Jimmy can't release the Neural Network code because he doesn't have any rights to it.

There is also the obvious question of why that code is there to begin with, and not encrypted. If Tesla doesn't care why not just let people share it? It's not like Elon doesn't know people have hacked cars. So it's almost on their just to create excitement, but why that way versus simply having Karpathy do some talk about it?

 

[strangecosmos]

I apologize for derailing the thread. I’ve asked others not to do that, and here I am doing that exact thing. I naively thought my post would lead to a resolution.

I posted my response in the designated Mobileye vs. Tesla megathread.

 

[strangecosmos]

I try not to make too much of it especially since there isn't really anyone that can vet it. Jimmy can't release the Neural Network code because he doesn't have any rights to it.

There is also the obvious question of why that code is there to begin with, and not encrypted. If Tesla doesn't care why not just let people share it? It's not like Elon doesn't know people have hacked cars. So it's almost on their just to create excitement, but why that way versus simply having Karpathy do some talk about it?

I might be wrong about this, but I don’t think Jimmy has access to the neural network source code. Comma.ai or whoever can’t just hack a Tesla and steal AKnet_V9. Right?

I think the reason Tesla doesn’t do more to restrict the software in its cars is that nothing bad enough has happened to justify the added work/cost. It’s been pretty harmless so far. Outside of the places wheee hardcore Tesla fans go — TMC, r/teslamotors, Electrek, Tesla Daily, the Tesla Show, etc. — and even in them, I don’t think most people are even aware of the hacking that’s going on. Encrypting everything might also prevent its bug bounty program from being effective.

If Tesla has spies in this forum, I would argue that you should encourage the work of verygreen and others. Similar to the bug bounty program, there is a chance people hacking their cars can discover a flaw that your spies can find out about here.

 

[Tezlanian]

I might be wrong about this, but I don’t think Jimmy has access to the neural network source code. Comma.ai or whoever can’t just hack a Tesla and steal AKnet_V9. Right?

You can learn a lot by dissecting the binary and the associated scripts. Once HW3 comes out this will become a lot more difficult as it's all custom and not using known CNN architectures potentially. Hopefully @jimmy_d or others will still be able to discern something after the switch and deprecation of the legacy hardware.

I think the reason Tesla doesn’t do more to restrict the software in its cars is that nothing bad enough has happened to justify the added work/cost. It’s been pretty harmless so far. Outside of the places wheee hardcore Tesla fans go — TMC, r/teslamotors, Electrek, Tesla Daily, the Tesla Show, etc. — and even in them, I don’t think most people are even aware of the hacking that’s going on. Encrypting everything might also prevent its bug bounty program from being effective.

If Tesla has spies in this forum, I would argue that you should encourage the work of verygreen and others. Similar to the bug bounty program, there is a chance people hacking their cars can discover a flaw that your spies can find out about here.

They're honestly way less hostile to the whitehats than many companies, especially legacy auto manufacturers. Read up here Tesla’s bug bounty program | Bugcrowd

 

[strangecosmos]

You can learn a lot by dissecting the binary and the associated scripts.

But it’s not like Tesla is giving away valuable intellectual property by not encrypting this stuff. Another company can’t just copy Tesla’s neural network architecture by hacking a Tesla. Right?

 

[electronblue]

The problem is neither of these people own the car.

So their entire perspective is often skewed towards things that have absolutely no short or medium term value to any owner.

I’m not sure about this. Ownership of car sounds irrelevant to discussing progress of autonomous vehicles. Ownership may even be misleading if currently owned car is not representative of what companies are working on and what can appear soon on the market?

 

[Vitold]

But it’s not like Tesla is giving away valuable intellectual property by not encrypting this stuff. Another company can’t just copy Tesla’s neural network architecturey hacking a Tesla. Right?

Of course Tesla is giving away valuable IP however Musk does not like moats and prefers to lead with speed of innovation. This strategy will work for a while as technology is far from mature - from any vendor.

 

[chillaban]

But it’s not like Tesla is giving away valuable intellectual property by not encrypting this stuff. Another company can’t just copy Tesla’s neural network architecture by hacking a Tesla. Right?

"encrypting" this stuff is a ton more complex than one expects, on a device that has to use the data it has.

iOS has encrypted its firmware since the dawn of time yet someone on twitter has it decrypted before long. Encryption just means you have to root the device at the right point in time to see the contents, and that's not going to deter a determined adversary.

I agree that there is an intellectual property risk to Tesla given the ability to extract the neural net, and how external entities like Damien, verygreen, etc are able to piece together this level of understanding of how Autopilot works. But at the same time:

(1) THIS is what leads us as customers to have faith in Tesla's progress towards higher levels of autonomy. It is the first independently-verified, customer-controlled autonomy stack. I don't trust manufacturer-curated demonstration footage. I spent so many years at an automotive supplier being on the other side, coming up with the demos that end up in datasheets and marketing slides. You can make it look as amazing as you want.
(2) Tesla doesn't seem to have a lot of resources in the software department, and I'd rather them spend less of their time on attempting to counter people rooting their own cars, and more time developing their actual system. It's often just as effective to outpace competition than it is to prevent them from reverse engineering your product. These days, practically nobody wants 2018.10.4 level Autopilot, even though in March that was such a huge improvement. If Tesla keeps improving at this rate, competition cannot bootstrap themselves by just copying Tesla's approach with many months of lag.

 

[jimmy_d]

"encrypting" this stuff is a ton more complex than one expects, on a device that has to use the data it has.

iOS has encrypted its firmware since the dawn of time yet someone on twitter has it decrypted before long. Encryption just means you have to root the device at the right point in time to see the contents, and that's not going to deter a determined adversary.

I agree that there is an intellectual property risk to Tesla given the ability to extract the neural net, and how external entities like Damien, verygreen, etc are able to piece together this level of understanding of how Autopilot works. But at the same time:

(1) THIS is what leads us as customers to have faith in Tesla's progress towards higher levels of autonomy. It is the first independently-verified, customer-controlled autonomy stack. I don't trust manufacturer-curated demonstration footage. I spent so many years at an automotive supplier being on the other side, coming up with the demos that end up in datasheets and marketing slides. You can make it look as amazing as you want.
(2) Tesla doesn't seem to have a lot of resources in the software department, and I'd rather them spend less of their time on attempting to counter people rooting their own cars, and more time developing their actual system. It's often just as effective to outpace competition than it is to prevent them from reverse engineering your product. These days, practically nobody wants 2018.10.4 level Autopilot, even though in March that was such a huge improvement. If Tesla keeps improving at this rate, competition cannot bootstrap themselves by just copying Tesla's approach with many months of lag.

Agree.

There are certain aspects to the use of NNs in software which is a kind of IP protection that is very powerful in and of itself. First - NNs are famously very black box in that they cannot be precisely understood in any straightforward way from an examination of their structure or behavior. This is a disadvantage in many ways but one useful feature (useful to some people, anyway) is that reverse engineering them is practically impossible. This contributes to IP protection when considered alongside another feature of NNs, which is that because a production NN is the product of a system that automatically generates the NN from a large database of training data and is quite sensitive to the details of the system that produces it you can't reproduce it without a complete and detailed set of all the data that went into producing it. What this means is that you can distribute your NN all you want and nobody is going to be able to extract any useful feature of it's overall behavior which is smaller than the complete NN.

Of course anybody who can use the NN exactly as is can just take a binary copy of it and use it directly. But if the target use differs in any significant way, no matter how minor, you cannot get much value from taking the NN. For instance, if you built another car that was mostly like a Tesla but not *identical* to a Tesla in terms of it's sensor suite, sensor placement, and control dynamics then the NN would probably not work very well. And you cannot modify the NN without access to all the data, tools, and settings that are used to produce it.

Of course there are some corner cases where you can train a modifier NN to take the output of an NN and adapt it to a slightly different situation, but even this is a major undertaking in the large majority of cases. You would still need an enormous curated database of training examples for your target vehicle - it would just be somewhat smaller than the database needed for the original network.

So in a sense, just using an NN in your design gets you a very useful level of IP protection so long as you keep tight control of the training data and training parameters. Since all that stuff lives in a well secured data center and never leaves your facility it's not particularly vulnerable to hacking. This would not be like hacking into google and trying to steal a file - it would be like hacking into google and trying to steal the contents of the entire data center - not impossible but quite infeasible.

As an aside - this is the method that Comma AI is using in their business model. They have open sourced their software and they give away almost all the details of their system except for the NN's production system. By keeping control of the development details and training database for the NN they can ensure that they will control their ecosystem without having to use any encryption or other form of DRM.

 

[lunitiks]

Makes a ton of sense IMO

 

[jimmy_d]

I might be wrong about this, but I don’t think Jimmy has access to the neural network source code. Comma.ai or whoever can’t just hack a Tesla and steal AKnet_V9. Right?

I think the reason Tesla doesn’t do more to restrict the software in its cars is that nothing bad enough has happened to justify the added work/cost. It’s been pretty harmless so far. Outside of the places wheee hardcore Tesla fans go — TMC, r/teslamotors, Electrek, Tesla Daily, the Tesla Show, etc. — and even in them, I don’t think most people are even aware of the hacking that’s going on. Encrypting everything might also prevent its bug bounty program from being effective.

If Tesla has spies in this forum, I would argue that you should encourage the work of verygreen and others. Similar to the bug bounty program, there is a chance people hacking their cars can discover a flaw that your spies can find out about here.

I hear from the folks who provide me with files that hacking Tesla's has been getting harder and harder. There seems to be an expectation that at some point it'll be outside the reach of amateurs. I recall that the iPhone was relatively easy to hack for several years when it came out but that has changed quite dramatically of late. Even well resourced organizations (like the FBI) complain that they can't get into recent iPhones.

But I think it's probably also true that hacking Tesla's hasn't so far presented Tesla with a serious IP threat for reasons detailed in my previous post. Tesla seems to be getting more hacker friendly recently too which seems like a positive development.

 

[jimmy_d]

Makes a ton of sense IMO

If you're talking about Comma AI's strategy I wholeheartedly agree. It's really very clever. They get all the advantages of being open source along with IP control that hollywood can only dream about. And the ability to sell their hardware as a 'dashcam' lets them avoid control by the NHTSA. In parallel they provide the NN as a free download for hobbyists, which the NHTSA has little ability to control (at least so far).

 

[CK_Stuggi]

Wouldn't "nabbing" the NN be rather simple given an identical/ closely matched camera placement, given the net being camera agnostic?

 

[jimmy_d]

Was listening to Karpathys most recent talk:

He starts at 3:09.

At 11:45 and forward he says that he believes in "Building a single model". I assume that the bigger FSD NN that we have seen might is the single model he talks about. And I assume this is some common layers and then a lot of different layers for the different tasks. Something like A->(B,C,D,E->(F,G->H))) or something like this:

Thanks for sharing that - really interesting. AK's talks are one of the best windows we have into the technical details of AP right now. He seems to intentionally not talk about Tesla in particular but the examples he provides from his experience there provide us with some insight nonetheless.

Apologies to anyone for whom the following is obvious - I'm going to provide a little explanation of the talk for the use of people who aren't very familiar with NNs. I've received quite a bit of feedback from them that such explanations are appreciated so I'm going to try to translate some of this stuff for their use.

AK talk translation:

Talk is for audience of people who work with NNs all the time and think in NN jargon. Am translating the overall points of it here:

Of course AK is an NN guy but recently he’s been promoting this idea that NNs are just one example of what he calls SW2.0 (software 2.0). So here he talks about SW2.0 but you can understand it right now as being mainly talking about making NNs for use in real products.

’SW 2.0’ is a new and very different method of making computer programs. Instead of writing a very precise and detailed recipe for the steps you want the program to perform you instead create a detailed description of what the program’s output should be. The SW2.0 process will produce a system that will then ‘learn’ / ‘discover’ / ‘search’ for a program that has the desired behavior.

Side note from jimmy_d: I say learn/discover/search because people in the field tend to use these terms interchangeably even though the plain english meaning is quite different. The process of training an NN is similar at a high level to ‘learning’ but at a low level the detailed algorithm uses a search method called gradient descent, so people in the field often talk about searching.

To do software 2.0 development you have to also do a lot of regular software development because the code that ‘searches’ is a regular software 1.0 program and there is also a lot of software 1.0 code used to create the very detailed behavior description. That means you have to write and maintain a lot of regular software which you use as part of the SW 2.0 (neural network) development process.

jimmy_d note: that ‘description’ is frequently in the form of a large database of particular behaviors you want from the SW2 program (or NN, in this case). For instance, your database could have a lot of examples of ‘in situation Y the correct output is X’. In learning parlance we call that the training set, but in Karpathy’s SW2 terminology he thinks of it as the software specification.

AK: Tesla uses a lot of NNs in AP
AK: Want to take best practices from regular code development and use them for SW2 code. For example, in SW1 test driven development you write the tests first then write your SW1 code to pass the tests.
AK then goes on to talk about various lessons from SW1 development and how there are good parallels in SW2 development. It’s really very interesting if you’re in the business.

So what do we learn about AP from this talk? Mainly we learn that Tesla’s AP development effort is very heavily invested in NNs to the point that they are pioneering a novel development methodology and a lot of new tools in order to maximally leverage the traits of NNs for the benefit of AP. We learn that they are using a production system that is heavily focused on version control, data management, and automated regression testing. AK also implies that a certain amount of automated deployment is going on.

 

[jimmy_d]

Wouldn't "nabbing" the NN be rather simple given an identical/ closely matched camera placement, given the net being camera agnostic?

Now *that* is an excellent observation.

To be sure, it seems that the advanced AKNET_V9 network - the camera agnostic one - is probably not the one most of us are using in V9 right now. So for the time being this issue probably doesn't apply. But the observation that making the network less sensitive to input distortion also makes it easier to steal is a great point. Of course being camera agnostic only removes one kind of system dependency. The network could still, for instance, be very sensitive to the placement of the cameras. So if you tried to use it on a vehicle that had different physical dimension there could be problems.

Along the lines of IP protection it's probably worth considering that HW3 could make the network even more difficult to steal if they decided to put even a little bit of anti-theft effort into it. The HW3 IC is wholly Tesla internal and reverse engineering IC's is very hard and duplicating them even harder. If Tesla were to, for instance, use a hardware filter to unscramble the NN weights on the fly as they are pulled from memory then they could provide scrambled weight files which would be unusable to anyone without an identical hardware unscrambler implemented in their chip IO.

 

[chillaban]

Now *that* is an excellent observation.

To be sure, it seems that the advanced AKNET_V9 network - the camera agnostic one - is probably not the one most of us are using in V9 right now. So for the time being this issue probably doesn't apply. But the observation that making the network less sensitive to input distortion also makes it easier to steal is a great point. Of course being camera agnostic only removes one kind of system dependency. The network could still, for instance, be very sensitive to the placement of the cameras. So if you tried to use it on a vehicle that had different physical dimension there could be problems.

Along the lines of IP protection it's probably worth considering that HW3 could make the network even more difficult to steal if they decided to put even a little bit of anti-theft effort into it. The HW3 IC is wholly Tesla internal and reverse engineering IC's is very hard and duplicating them even harder. If Tesla were to, for instance, use a hardware filter to unscramble the NN weights on the fly as they are pulled from memory then they could provide scrambled weight files which would be unusable to anyone without an identical hardware unscrambler implemented in their chip IO.

Right if they have custom silicon, they could theoretically have their neural accelerator take in encrypted neural nets and then software running on the device would never get to see the actual innards of it.

But with that said, it sounds to me like IP theft of neural nets would be blatantly obvious — it sounds difficult to obfuscate a neural net in a way that would conceal the fact that it was stolen from Tesla. And that turns it into a simple matter of legal enforcement.

 

[lunitiks]

One thing though, @jimmy_d, are you sure about the part about sensor placement? I mean isn’t S, X and 3 using the same NN architecture and weights? (Granted, the camera placement are generally the same but they’re certainly not identical considering body dimensions! I imagine it’d make sense for Tesla to have the Semi, Roadster2020, Y and the pickup use the same NN as well?)

 

[heltok]

Wouldn't "nabbing" the NN be rather simple given an identical/ closely matched camera placement, given the net being camera agnostic?

Car manufacturers usually buy competing cars, take them apart, compare every detail of them and see what they can learn from them. This is very standard. In this case they will learn the sensor suite Tesla is using, the ethernet network layout they use, the compute hardware, the neural network input, the neural network model and the weights. So they could probably spend a year to rebuild the same in their own car using the same weight, but that will be tricky as the weights are very tuned to the specific system. They could try to duplicate the data Tesla has also and start to train their own weights using the same model, but that will take a lot of time. It is very hard to be a copy cat, you will always be behind and the engineers will not feel very creative and look for options. And customers are willing to spend less on it. And there might be some legal actions Tesla can take to defend themselves against this.

Likely what they will do is learn what Tesla has done, then bring it into their design discussions. And they don't have access to the same data gathering capabilities that Tesla has so likely they will have to find a different approach and maybe network architecture tailored to their data. And the resistance to learning from what Tesla is doing can be heavy among the old companies, I say this from experience having held presentation to a car manufacturer about what I thought Tesla's strategy was. Maybe the Chinese and the Silicon Valley companies are more open to learning from Tesla's solutions.

Fwiw, the network is not camera agnostic. It is maybe somewhat calibration agnostic, but is likely very much CMYK dependent and probably dependent on many other things also.

 

[Tezlanian]

One thing though, @jimmy_d, are you sure about the part about sensor placement? I mean isn’t S, X and 3 using the same NN architecture and weights? (Granted, the camera placement are generally the same but they’re certainly not identical considering body dimensions! I imagine it’d make sense for Tesla to have the Semi, Roadster2020, Y and the pickup use the same NN as well?)

It's just of matter training and preprocessing the camera input. A single network if trained well enough can cover all vantage points arbitrarily.

Fwiw, the network is not camera agnostic.

I think they can accomplish it.

 

[jimmy_d]

One thing though, @jimmy_d, are you sure about the part about sensor placement? I mean isn’t S, X and 3 using the same NN architecture and weights? (Granted, the camera placement are generally the same but they’re certainly not identical considering body dimensions! I imagine it’d make sense for Tesla to have the Semi, Roadster2020, Y and the pickup use the same NN as well?)

Yes, as far as I know that is true. And I don't know what the actual geometry sensitivity is as it's not something I can work out from first principles in this case. The vehicle dimensions thing was just a quick example of a different sensitivity that the networks might have. But your point stands that vehicle dimensions alone probably don't matter for the networks that are being deployed right now.

AKNET-V9 seems to have an omnibus output that collates everything from all the cameras to create a single 360 degree situation representation. It seems like that might be more likely to care about stuff like the relative placement of cameras than the stand alone camera networks we seem to have right now because it could be fusing, for instance, the FOV of the pillars with the main and fisheye - in which case the difference in spacing between the 3 and the S could have an effect on objects that can be perceived by both the main and a pillar camera simultaneously.

Though of course that is also just speculation. It's fun to think about but it might be wrong.