Tesla's new FSD patent to optimize image processing for deep learning

[diplomat33]

Tesla has a new patent called “Data Pipeline and Deep Learning System for Autonomous Driving”. Essentially, the patent optimizes the process of image processing for deep learning.

Some interesting quotes from the article:

"The flowchart below describes what the process of the vehicle learning the information would look like. “Receive Sensor Data” is the first portion of this process. Then, data will be broken down and pre-processed for the system to then begin its “Deep Learning Analysis.” The results will then be passed along to the vehicle’s Artificial Intelligence Processor to be utilized during vehicle control."

"In another process, the series of information that is retrieved from these images will be compared to data compiled from other Tesla users on a global scale. This will alleviate concerns that drivers may have that the system could perform the wrong process when driving autonomously. The aim of the patent is to create a safe driving experience and improve upon the already solid performance of Tesla’s autonomous driving software, and do so in a process that is more efficient than before."

"By using this process, Tesla is able to maintain as much resolution as possible from the images captured by its vehicles’ cameras and sensors. This then allows the Neural Network to more efficiently learn from the data packets that it is receiving. This allows the Neural Network to work with better images in a more efficient manner as well, which opens the doors to faster autonomous driving improvements."

Tesla's new data pipeline and deep learning patent paves way for quicker autonomous driving improvements

Hopefully, this patent helps Tesla speed up their FSD development.

.

 

[Eno Deb]

At a cursory glance that looks like a very broad patent that will be difficult to defend in litigation. Interesting that they are covering Lidar in one of their claims. But generally, filing a patent doesn't necessarily mean that any or all of the claims will end up in the product.

 

[strangecosmos2]

The language is extraordinarily generic and legalistic. Hard to tell if there's anything in this patent we didn't already know.

 

[Eno Deb]

The language is extraordinarily generic and legalistic. Hard to tell if there's anything in this patent we didn't already know.

If you look at the claims the key is really in claim 1 (the other claims are either dependent claims or more detailed versions of claim 1), specifically this:

"providing each component image of the plurality of component images as a different input to a different layer of a plurality of layers of an artificial neural network to determine a result; and using the result of the artificial neural network to at least in part autonomously operate the vehicle. "

This seems to be essentially what Karpathy has recently talked about in some presentations about the structure of their neural nets (obviously after Tesla filed he patent). Multi-layer neural nets are actually well known prior art and thus probably not patentable, but perhaps not in the context of autonomous cars (although that seems doubtful too).

 

[diplomat33]

If you look at the claims the key is really in claim 1 (the other claims are either dependent claims or more detailed versions of claim 1), specifically this:

"providing each component image of the plurality of component images as a different input to a different layer of a plurality of layers of an artificial neural network to determine a result; and using the result of the artificial neural network to at least in part autonomously operate the vehicle. "

This seems to be essentially what Karpathy has recently talked about in some presentations about the structure of their neural nets (obviously after Tesla filed he patent). Multi-layer neural nets are actually well known prior art and thus probably not patentable, but perhaps not in the context of autonomous cars (although that seems doubtful too).

While multi-layered NN's are not new, it seems what Tesla did is develop a method or technique for feeding different component images to different layers in the NN. So one layer of the NN gets one component image, another layer of the NN gets a different component image. That's what earned them this patent. So presumably, this specific technique for distributing different components images to different layers of a NN is something new.

 

[DanCar]

... That's what earned them this patent. ...

Not sure where the word "earned" comes from. From my perspective they haven't earned anything. They are just claiming something in a patent filing. If they win a court case defending this, or someone decides to pony up something based on the patent, then they will have earned something.
Someone might claim if they are granted a patent they have earned something but the bar is low for filing patents and many frivolous patents are filed each year.

 

[strangecosmos2]

Multi-layer (i.e. deep) neural networks have been ubiquitous since around 2012 and today “neural network” is synonymous with “deep neural network” or “multi-layered neural network”.

What is notable about Tesla's neural network is that it's a multi-task neural network. (See Karpathy's ICML 2019 talk.) Tesla didn't invent the idea of multi-task networks, so I would guess/hope that idea by itself isn't patentable.

 

[diplomat33]

Multi-layer (i.e. deep) neural networks have been ubiquitous since around 2012 and today “neural network” is synonymous with “deep neural network” or “multi-layered neural network”.

What is notable about Tesla's neural network is that it's a multi-task neural network. (See Karpathy's ICML 2019 talk.) Tesla didn't invent the idea of multi-task networks, so I would guess/hope that idea by itself isn't patentable.

Again, the patent is not for multi-layered NN. The patent is for distributing composite images to different layers in the NN, among other things.

 

[diplomat33]

Not sure where the word "earned" comes from. From my perspective they haven't earned anything. They are just claiming something in a patent filing. If they win a court case defending this, or someone decides to pony up something based on the patent, then they will have earned something.
Someone might claim if they are granted a patent they have earned something but the bar is low for filing patents and many frivolous patents are filed each year.

By earned, I just mean that the patent was granted.

 

[strangecosmos2]

Was the patent actually granted or is the patent application still pending?

 

[diplomat33]

Was the patent actually granted or is the patent application still pending?

The patent was published yesterday.

 

[strangecosmos2]

The patent was published yesterday.

Does published mean granted?

The patent is for distributing composite images to different layers in the NN, among other things.

Isn't it the opposite of composite images, i.e. images decomposed into parts?

 

[diplomat33]

Does published mean granted?

Not necessarily. Patents are published before they are granted.

Isn't it the opposite of composite images, i.e. images decomposed into parts?

The patent says "decomposing the received image into a plurality of component images". So I interpret that to mean that it takes one image image and breaking it up into several images.

 

[strangecosmos2]

The patent says "decomposing the received image into a plurality of component images". So I interpret that to mean that it takes one image image and breaking it up into several images.

Right, and a composite image is made by taking parts of different images and stitching them together into one image. It's the opposite of a composite image.

 

[Eno Deb]

Was the patent actually granted or is the patent application still pending?

This is an international patent application (PCT process), which cannot in itself turn into granted patent (only national patent offices can grant patents). Basically, it's a process of preparing a unified application that can later be used to apply for a patent in multiple treaty countries of your choice. The publication (together with the results of an international search) is the final step of this "international phase" before the potential national patent applications.

 

[Bladerskb]

By earned, I just mean that the patent was granted.

No the patent hasn't been granted. Plus It takes years to get a patent granted.
Secondly this is an absolutely useless patent that is being as usual hyped up by the Tesla fanbase as being some 'game changer'.

While others have hundreds some even thousands of SDC related patents.
Tesla's SDC related patents can almost be counted on one hand. Yet every single one of them have be hyped.

''paves way for quicker autonomous driving improvements" lol seriously? is there any truth in the Tesla Land?

 

[diplomat33]

No the patent hasn't been granted. It takes years to get a patent granted.
Secondly this is an absolutely useless patent that is being as usual hyped up by the Tesla fanbase as being a game changer.

'quicker autonomous driving improvements' lol wtf? is there any truth in the Tesla Land?

I stand corrected on the first part. I was wrong about the patent being granted.

But how is the process described in the patent of "providing each component image of the plurality of component images as a different input to a different layer of a plurality of layers of an artificial neural network to determine a result; and using the result of the artificial neural network to at least in part autonomously operate the vehicle." useless? Surely it would have some use to Tesla.

 

[Bladerskb]

I stand corrected on the first part. I was wrong about the patent being granted.

But how is the process described in the patent of "providing each component image of the plurality of component images as a different input to a different layer of a plurality of layers of an artificial neural network to determine a result; and using the result of the artificial neural network to at least in part autonomously operate the vehicle." useless? Surely it would have some use to Tesla.

Because others have hundreds some even thousands of SDC related patents. Google/Waymo has hundreds, maybe thousands. Mobileye for example has hundreds maybe thousands and these are useful patents that could block an actual competitor from doing something vital.

Tesla's SDC related patents can almost be counted on one hand. Yet every single one of them is hyped up as some game changer. The article you posted said this patent 'paves way for quicker autonomous driving improvements' and that it 'making sure that it will evolve at an even faster rate in the future.'

This patent has nothing to do with that, its simply a patent on prepping data before feeding it into a NN. The spin that is put on any little thing that Tesla does to push agenda and narrative that their NN is on an exponential growth or that they are ahead is so gross.

 

[Jack6591]

Because others have hundreds some even thousands of SDC related patents. Google/Waymo has hundreds, maybe thousands. Mobileye for example has hundreds maybe thousands and these are useful patents that could block an actual competitor from doing something vital.

Tesla's SDC related patents can almost be counted on one hand. Yet every single one of them is hyped up as some game changer. The article you posted said this patent 'paves way for quicker autonomous driving improvements' and that it 'making sure that it will evolve at an even faster rate in the future.'

This patent has nothing to do with that, its simply a patent on prepping data before feeding it into a NN. The spin that is put on any little thing that Tesla does to push agenda and narrative that their NN is on an exponential growth or that they are ahead is so gross.

You weren’t the sunny kid in your class. Were you?

 

[S4WRXTTCS]

The article you posted said this patent 'paves way for quicker autonomous driving improvements' and that it 'making sure that it will evolve at an even faster rate in the future.'

Where are you seeing that?

This is what he linked to, and it's not an article at all.
WO2019245618 DATA PIPELINE AND DEEP LEARNING SYSTEM FOR AUTONOMOUS DRIVING

its simply a patent on prepping data before feeding it into a NN

That's not really what they're trying to patent. It's more about how the data is optimized/customized for the layer it's used on.

At the end of the day it's just another SW Patent which typically is a patent on obvious things. I'm sure you've dealt with the Head of IP asking people to think of anything that can be protected. Where the patent is mostly defensive posturing. Where you try to line up your patents with a competitors patent in some kind of Cold War like stand off. So they're often about extremely trivial things, and probably won't hold up to scrutiny.