The leap from L2 (which is where EAP sits) to L3 is pretty big, and in the absence of LIDAR, rock-solid vision (way better than Mobileye) is an absolute minimum. But it's more on top of that -- it's how to deal with a host of different potential scenarios. To turn an L3 system into an L4 system is a bit like training a termite colony how to build a concrete and steel skyscraper. You can sort of see the similarities between a termite mound and an office tower, but come on...
-
Want to remove ads? Register an account and login to see fewer ads, and become a Supporting Member to remove almost all ads.
HW2.5 capabilities
- Thread starter verygreen
- Start date
-
- Tags
- Autonomous Vehicles
The leap from L2 (which is where EAP sits) to L3 is pretty big, and in the absence of LIDAR, rock-solid vision (way better than Mobileye) is an absolute minimum. But it's more on top of that -- it's how to deal with a host of different potential scenarios. To turn an L3 system into an L4 system is a bit like training a termite colony how to build a concrete and steel skyscraper. You can sort of see the similarities between a termite mound and an office tower, but come on...
jimmy_d
Deep Learning Dork
I’m still a little unclear as to why we can’t brainwash the robot for a while into acting like the cycloptic cat? How is it possible for MobileEye to take their tech with them? Another analogy would be helpful
It doesn't look like anyone else here is going to give you a straight answer, so how about this one: the particular behavior of the mobileye part is encoded in the silicon directly to a large degree. In order to reuse the AP1 code that Tesla themselves developed they need something that would be an extremely similar substitute. Essentially they'd need to build an emulator for the mobileye part. That could be risky legally, because there's probably a boilerplate agreement to not reverse engineer the mobileye part in their agreement and they'd certainly be in violation of that. But that is really moot because developing an accurate emulation would take longer than getting AP2 working from scratch, unless mobileye shared the details of how their silicon is architected and that is definitely not happening.
Here's a less clear and yet wordier and vastly more sanctimonious version of the above:
AP1's critical functionality ran through a pipeline where the camera feeds straight into the mobileye chip. That chip runs a suite of visual pattern recognition heuristics on the video feed and outputs a feature vector. Basically the feature vector is a chunk of data that say "yellow lane marking at position X with probability Y, stop sign of size A at location B with probability C ...". When the mobileye part was developed they expected that a backend processor would take that and run a bunch of regular code on it to understand the situation the car was in and react appropriately. But then neural networks happened and it turned out that putting a NN software running on a separate cpu/gpu into the stream after the mobileye processor made more sense so Tesla switched their AP1 software design over to doing that. And to make it work they had to develop that NN software and all the succeeding architectural bits themselves. The upside is they got a system that worked a lot better than what you could do with mobileye's original approach while getting most of the advantages of power savings, cost savings and so forth, but it meant developing all the post-mobileye stuff without much help from them. Luckily NN's a really good at that kind of thing. In a sense the AP2 hardware right now is just using the NVIDIA gpus in the APU to replace the mobileye functionality.
Now Tesla likely has rights to everything except what's inside the mobileye chip. But that neural network that operates on the mobileye output is custom tailored to the exact behavior of that chip - including all it's quirks, because that's what neural networks do. And the low level visual features are all being extracted by the silicon in that mobileye chip. The pipeline is completely broken without it. And the precise behavior of the mobileye chip is encoded into the silicon design. There's code that runs on the mobileye chip, and Tesla probably had access to that, but because the mobileye is a custom processor that code isn't good for anything except running on the mobileye and you can't emulate it precisely without knowing the silicon design in great detail.
So if mobileye takes their ball you have to just play a different game.
A year ago Tesla probably looked at recent progress in using NN's to process images directly and estimated that getting AP1 performance out of a from-scratch NN running on their fancy new NVIDIA GPU's, (which are much more powerful than the raw numerical performance of the mobileye part), was something that would be doable in 3 or 6 months. That's certainly what I expected. But then AP1 was pretty mature and the mobileye guys weren't actually morons so that turned out to be a high bar for all but the most mundane situations. Software maturity means that you've had the time to track down and deal with the corner case surprises that piss people off. AP2 is not yet mature in terms of its autosteer performance at higher speeds. And AP1 kept getting better too because they didn't stop working on it. AP1 a year ago was not nearly as good as it is today. I have used both of them (25k miles on AP1 but only about 1k on AP2 so far) and AP2 today is a lot better than AP1 was in Dec of 2015. It's probably about where AP1 was in Dec of 2016. But AP1 today is a lot better than it was last Dec and AP2 is just catching up to where AP1 was several months ago. The bright side is that AP2's rate of improvement is much faster than AP1 so I expect it to move ahead of AP1 in the next few months, but it might not be mature for another year yet. Maturity takes time.
And for an even bigger digression:
It took Musk years longer to land an orbital booster than he originally predicted. But he did land it - something the "experts" had long derided as so difficult as to be effectively impossible and which competitors considered to be so unlikely that it wasn't even worth seriously thinking about until after Musk had delivered proof. And now he's landed 16 of them including everything so far this year. And the rest of the world is many years, perhaps decades behind. FSD cars won't go the same way because competitors are already pouring money into the space and a lot of big companies are taking it very seriously. But in a similar fashion to what happened with the rockets, Tesla is audaciously pursuing what should be possible without hedging their bets in ways that slow their advance into the market. A Tesla bought today may not have FSD right now, but very likely it will eventually have that capability. There is no other vehicle for sale, or which has been announced, which can claim anything close.
jimmy_d
Deep Learning Dork
Well it's hard to know for sure how ME changed their code over time, but the IC in question was developed without consideration for neural networks. Their part isn't useless for NN's, but it certainly isn't optimized for them. It lacks features that you'd want to have in an NN chip. You could run an NN on it, but that's not what ME was doing when Tesla started using it. Senior folks at ME used to argue that their 'hand coded' features were better than what an NN could come up with. And there's probably some truth to that in certain respects, but it misses the point. In any case, when Tesla announced that they were going with ME I searched all of ME's publicly available news and data, including their part specs and youtube's of their engineer's giving talks, and the words 'neural network' didn't appear anywhere in that corpus.
As a neural network guy I found that kind of annoying.
Quite true. Having ME to build on made the job of building AP1 a lot easier. Losing it forced them to start over and that had a cost.
But it's also true that a pure NN solution will surpass what could have been built on ME's hardware - given some time. And not having a partner that's actively trying to undermine your business has some advantages. I think George Hotz's opinion of those guys is probably right on the mark.
He's a clown, but he knows his stuff that guy.
Very interesting, I didn't know who George Hotz was, but I remember reading about this interaction he had with Elon.
After watching the video, it is a very good example of how a brilliant person's immaturity and narcissism can end up sabotaging himself and greatly limiting his options and voice.
Perhaps Elon's best trait is being likable in addition to being really smart. Hotz's personality is so awful in this clip (I'm thinking a far worse version of the Zuckerberg in the Social Network), that it distracts from his otherwise fantastic work.
A good example of how being really smart doesn't protect a person from one's own douchey narcissism. Can you imagine how many willing folks would've given this guy the reigns and the money if he didn't make people want to vomit?
Instead he's basically someone's uncle at thanksgiving who insists on wearing his Mensa t-shirt and kicking everyone's ass at chess, meanwhile he's on unemployment and spends his time convincing your 13 year old cousin that he wasn't fired, but he did have to leave his last 4 jobs because his bosses were massive idiots.... "pass the gravy Jared, because you just got check mated for the 5th time today"
Mobileye eyeq3 was built on deep neural networks. You were clearly looking at all the wrong places or not even looking at all.
Mobileye most certainly was using a neural net in the EyeQ3 chip that does object & path recognition and path planning in AP1. It was simply a mature, pre-trained neural net (or rather a whole group of specific neural nets used for different specific purposes) provided by Mobileye to Tesla.
Tesla may also have used their own neural nets in conjunction - we don't really know. But Mobileye was certainly providing a trained neural net also - they've said so themselves. This isn't some big secret or theory. Furthermore buried in the EyeQ3 specs from Mobileye is a tantalizing sentence that claims that chip was capable of actual learning on the fly in individual cars - nobody knows whether the feature was implemented but it was in Mobileye's specs. What know for sure is they provided curated, trained neural nets to Tesla as the foundation for autopilot 1. I think perhaps what you are referring to is the difference between "supervised" and "unsupervised" learning or at least the difference between Mobileye's compartmentalized neural nets and Nvidia's demonstration of an end-to-end neural net.
Mobileye had one neural net for detecting the shape of a car in front of you, for example. You take a neural net and feed it pictures of cars and other objects. If the net guesses right you say "good job" if it does not you say "wrong." Over time the net learns to recognize cars. At some point you are done "training it" and the weighting of the variables in the net are "fixed" rather than adjusting to more training images. This "trained" net goes into the EyeQ3 chip and goes to work in your Tesla.
An "end to end" solution like Nvidia's demo takes a single giant neural net and feeds it an image of the world outside the windshield and then the net does nothing more than observe the angle of the steering wheel through time. No objects are labeled, no human supervises except to say "good job" or "bad job" to the entire task of driving. Object and road recognition are not separate tasks and neural nets, for example - with this method.
Now - what Tesla is doing now is probably *not* the end-to-end Nvidia demo despite the fact that Tesla is using Nvidia's Drive PX2 chip in the cars and Nvidia's training supercomputer back in the lab. Nobody knows exactly what Tesla is doing but some members here can speak about the training method much more coherently than I can. I believe it is multiple compartmentalized neural nets (based on other discussions and hints dropped like people such as Tesla's current autopilot head on Reddit a wile back) combined with some hard coded steering algorithms.
I'm blown away by the knowledge of folks on this thread, but then again, I'm not a computer science or engineer (but I did charge at a Holiday Inn Express last night).... Anyway, it brings to life the complexity of the subject.
As a non-programmer, it helps me realize that I was assuming complexity in all the wrong areas. I've studied some aspects of neuroscience, and I've learned that people assume the brain is too much like a computer circuit with hard wiring, and they think its possible to model neural systems digitally, but they don't really understand the how the nature of brain development, neuronal pruning, strategic cell death, and the fact that the brain sits in an ever-changing broth of fluid that changes and breathes and that this makes the whole endeavor much different. At best it seems, perhaps we can eventually impersonate the life like brain with a computer, but it's gonna be awkward.
So I digress into my dilemma.... as I'm still befuddled by exactly why it's so hard to see and impersonate the structure of the silicon architecture in ME, if I am using the correct terminology.
I can't wrap my buttershrimpy brain around why it can't be effectively replicated. I mean, eventually Jesse could replicate Walt's meth amphetamine in breaking bad. Why can't Tesla make good crystal?
The whiskey analogy makes sense to me, but it introduces the idea of their being random complexities and variations that even the whiskey brew masters are unaware of.... they just have such a sense of how to combine ingredients for the most part and how to combine variations in the various parts of the process so that it's impossible to replicate because of climate and other physical limitations in space... Whereas the mobileye chip seems to me to be more like an iPhone chip that could be produced over and over and over again.
Am I incorrect in assuming that the issue isn't just the recipe, but it's that AP2 is trying to do more in the long run? Kind of like what @lunitiks was saying?
By the way.... can we please please please get some more vivid and disturbing analogies from him?
I mean that dude busted out a picture of a birth defected cycloptic cat for god's sake. I know Tesla posted a job opening for a visual graphic artist to represent the world on autopilot.... can we please convince @lunitiks to accept that job? Screw AP1 truck and motorcycle representations, I want to see what he can do with roadkill representations on autopilot!
As a non-programmer, it helps me realize that I was assuming complexity in all the wrong areas. I've studied some aspects of neuroscience, and I've learned that people assume the brain is too much like a computer circuit with hard wiring, and they think its possible to model neural systems digitally, but they don't really understand the how the nature of brain development, neuronal pruning, strategic cell death, and the fact that the brain sits in an ever-changing broth of fluid that changes and breathes and that this makes the whole endeavor much different. At best it seems, perhaps we can eventually impersonate the life like brain with a computer, but it's gonna be awkward.
So I digress into my dilemma.... as I'm still befuddled by exactly why it's so hard to see and impersonate the structure of the silicon architecture in ME, if I am using the correct terminology.
I can't wrap my buttershrimpy brain around why it can't be effectively replicated. I mean, eventually Jesse could replicate Walt's meth amphetamine in breaking bad. Why can't Tesla make good crystal?
The whiskey analogy makes sense to me, but it introduces the idea of their being random complexities and variations that even the whiskey brew masters are unaware of.... they just have such a sense of how to combine ingredients for the most part and how to combine variations in the various parts of the process so that it's impossible to replicate because of climate and other physical limitations in space... Whereas the mobileye chip seems to me to be more like an iPhone chip that could be produced over and over and over again.
Am I incorrect in assuming that the issue isn't just the recipe, but it's that AP2 is trying to do more in the long run? Kind of like what @lunitiks was saying?
By the way.... can we please please please get some more vivid and disturbing analogies from him?
I mean that dude busted out a picture of a birth defected cycloptic cat for god's sake. I know Tesla posted a job opening for a visual graphic artist to represent the world on autopilot.... can we please convince @lunitiks to accept that job? Screw AP1 truck and motorcycle representations, I want to see what he can do with roadkill representations on autopilot!
skitown
Member
Please read this in the Stephen Hawking voice:
"funny-you-should-mention-that-@skitown... as I-haven't-regained-bowel-functions-since-leaving-that-thread"
One thing no one has brought up-
@verygreen mentions evidence of a new NN for the front facing wide angle camera. Remember that EAP only was advertised at using two of the front facing cameras, so a third is indicative of coming FSD functionality, if the third camera is indeed not for EAP.
@verygreen mentions evidence of a new NN for the front facing wide angle camera. Remember that EAP only was advertised at using two of the front facing cameras, so a third is indicative of coming FSD functionality, if the third camera is indeed not for EAP.
stopcrazypp
Well-Known Member
It may be used for rain sensing too. The wide and narrow cameras are not as suitable for that.
Perhaps it can. There's a whole academic field devoted to doing so. Problems become more difficult the longer you look at them. One of Chomsky's books relates an anecdote of the state of AI in the 1940's or 50's. Someone assigned the task of replicating vision to an undergrad researcher as a summer project.
You don't think Tesla is making good crystal? They've been at it only 12 months - I think they've made amazing crystal for how long they've been doing this. See my attached photo. Remember AP1 - they're progressing far faster than that system did. Oct 2014 was the AP1 hardware surprise release. A full year passed before autosteer was released. A FULL YEAR - and Tesla was working with pre-trained neural nets.
AP2 is more reliable now than the first release of AP1 was in October 2015. AND - this time Tesla had to start from zero. AP1 they had a huge head start. I understand folks are frustrated but looked at objectively this progress is stunningly fast. See my little chart below - which I update whenever a new release comes along so I can remind myself how fast this development really is.
AP1 -> 12 months from hardware to autosteer using a pre-trained neural net from mobileye.
AP2 -> 3 months from hardware to autosteer using a blank neural net that knew absolutely nothing.
20 days from now will be the equivalent of the AP1 hardware to software delay - and AP2 is already better than AP1 7.0 was in October 2015.
AP1 has gotten better since then - it's a moving target Tesla is chasing with AP2 (and Tesla is catching up). Remember AP1 we love so much is a full 2 years post-release.
AP2 has progressed faster and we have every reason to believe its progress will only accelerate. Tesla has only one quarter of AP2 video uploads going right now and I have already seen my car definitely learn in one trouble spot between releases. Not perfect but it's dramatic learning. We know these video uploads are being used for reinforcement neural net learning.
No I think you are spot on. Tesla is taking a more difficult approach than, say Cadillac - which "cheated" by simply mapping the roads with lidar. But in the long run Tesla's system should be extremely robust.
I don't remember Tesla being specific about which 4 cameras EAP would use, and there is no reason why the system cannot dynamically choose which 4 cameras are most appropriate for a scenario (for example, 3 forward + right rear facing when overtaking on a highway).
Also, you could argue that rain detection is a different function to AP, and the wide angle cam is active just for that. Lots of wriggle room
lunitiks
Cool James & Black Teacher
Who cares how many cameras Tesla is using. Just get the damn things to work properly!
I don't remember Tesla being specific about which 4 cameras EAP would use, and there is no reason why the system cannot dynamically choose which 4 cameras are most appropriate for a scenario (for example, 3 forward + right rear facing when overtaking on a highway).
Also, you could argue that rain detection is a different function to AP, and the wide angle cam is active just for that. Lots of wriggle room
The blog post wasn't specific. The email that some folks got announcing it was - front normal, front long, and left/rear and right/rear. There are copies of this in a bunch of old threads.
I certainly won't say faster nor a head start. Why? Because while Tesla had NN models from EYEQ for AP1 but didn't for AP2.
They actually had all the refined control algorithms from AP1. So they didn't need to re-code that and can reuse not only the source but all gps map data, radar data, and all the engineering knowledge, tips & tricks, workaround, trials and err they have learned during AP1 dev the last 2 years.
So no it hasn't been progressing further.
when you take everything into account you will find that AP2 actually got a bigger head start.
Thats not cheating...
JeffK
Well-Known Member
I believe it's cheating. A good system should be able to work no matter if you place it in a city or on a back country road it's never seen before.
Yesterday I was navigating for a group and an entire neighborhood was not on Google maps, highly doubt it's been mapped by some lidar mapping service. Would self driving simply stop working in this neighborhood if it wasn't previously mapped? If your system relies on lidar data then all your cars need lidar. There's no way around that for lvl 5 when relying on such data.
The easiest way to avoid that situation is to have the cars not rely on lidar data
Granted, in the above situation with a neighborhood not being mapped, something that relies on other kinds of mapping wouldn't be able to navigate either, but at least it'd be able to drive, make turns, etc.
Actually I think it was. They said EAP would use 4 of the 8 cameras and we already know two of cameras are the front normal and front narrow. We also know EAP includes automatic lane changes which means it also needs to two side cameras in the front left and right quarter panel.
So for EAP the cameras are: Front Narrow, Front Normal, Front Left Rear Facing, Front Right Rear Facing.
People that bought from August-October and hadn't taken delivery were forced into hardware upgrades to AP2. They were offered sticking with AP1 software or getting EAP. In that email, I believe Tesla elaborated on the particular cameras for EAP vs. FSD and they track @BigD0g 's analysis above.
Similar threads
- Article
