-
Want to remove ads? Register an account and login to see fewer ads, and become a Supporting Member to remove almost all ads.
Neural Networks
- Thread starter lunitiks
- Start date
That was Karp's (as we call him) neural net for simulating a classical ballet. That was beta, will probably be finished in 3...What am I looking at?
jimmy_d
Deep Learning Dork
And with that - time to return to our regularly scheduled programming…
So with the introduction of 2018.10.4 we got big changes in the behavior of AP2 but we really didn’t see much change in the neural network metadata. If anything it seemed like there were less resources being deployed for vision than with earlier vision versions aside from the main/narrow camera which got fatter but didn’t get a lot more complicated otherwise. So why the big changes? Maybe there’s some important stuff we’re not seeing.
In addition, Karpathy described 10.4 as the result of an extensive rewrite, but not a lot of that was appearing in the network definition files. Did he mean rewriting Tesla’s internal training stuff? It seemed like he was saying that the stuff going into the cars got a rewrite.
So, I looked deeper into the datadump I received from my anonymous benefactor - it contains stuff beyond the easily interpretable neural network metadata files. And I found… more networks! A lot more.
So now that I’m looking closer I find that metadata files are only present for a subset of the networks in use in the system. Previously that metadata was the only stuff I was seeing, so I was only seeing a part of what was there. Looking back at older versions it seems like the ‘binary only’ (without metadata) networks have been present in the car for a long time.
Digging around in those ‘binary’ networks I find that there are a bunch of other networks for which there is no metadata but for which there is complete execution code, network weights and so forth. At a minimum there are networks for all the cameras except the backup camera, with independent networks for the main, narrow, fisheye, repeater, and pillar networks. It looks like the pillar cameras both use a single network and the repeater cameras both use a single network - so at a minimum that makes 5 different camera networks shipping in 10.4. That's just the binary ones, the metadata ones are separate.
And further, there seem to be networks that perform processing beyond the single frame analysis that is going on in the camera networks. From symbol names and embedded strings it looks like these nets are analyzing lanes, identifying drivable space, picking out landmarks to help the vehicle locate itself, and detecting moving objects - which is something that you can’t do with the previously identified networks alone. There also seems to be code that integrates radar and vision to match up what the two are seeing and also code that estimates the distance to things based only on vision. Maybe this latter is used to match against radar, or maybe it gets used for the pillar and repeater cameras - i don’t know yet.
There’s lots of stuff that I don’t know yet. It’s unlikely that all these networks are used in all cars because some of them are redundant. So maybe there are different network for dev cars included in the regular firmware. Or maybe some of these networks are true ‘shadow’ networks that are being run in parallel with the currently deployed network in order to evaluate their performance or otherwise gather data. I suppose it’s even possible that geographic differences could account for redundancy (do you need a different network in right hand drive countries?).
So this last comment is my own judgement from the overall class of stuff that seems to be getting processed in these networks: I don’t think there’s any FSD stuff in this regular firmware - not even in shadow mode. All the functions I've seen so far seem to be focused on supporting lane keeping and lane changing. There’s support for identifying traffic signals and traffic signs which is stuff that doesn’t yet appear in the behavior of AP2 in any obvious way, but that by itself isn’t enough to tell me that this code base is being used for FSD related development. The sign and signal features might be there to support EAP in terms of not-yet-usable features (stop sign assist? red light assist?) or they could be there as part of helping the car to locate itself or in support of gathering map data. FSD seems to require a lot more than what I see here right now.
So that means the FSD stuff would be a separate effort for which we have only a promo video and one lone twitter leak to guess from.
Looked at this way 10.4 seems like a much bigger change. For one thing the total size of the neural networks dedicated to vision related tasks has more than doubled, and most of that gain is not in processing the camera output frames directly, but in post-processing that provides higher level understanding of vehicle context. In fact about half of the vision processing in 10.4 is ‘post processing’ and not the camera networks themselves.
So with the introduction of 2018.10.4 we got big changes in the behavior of AP2 but we really didn’t see much change in the neural network metadata. If anything it seemed like there were less resources being deployed for vision than with earlier vision versions aside from the main/narrow camera which got fatter but didn’t get a lot more complicated otherwise. So why the big changes? Maybe there’s some important stuff we’re not seeing.
In addition, Karpathy described 10.4 as the result of an extensive rewrite, but not a lot of that was appearing in the network definition files. Did he mean rewriting Tesla’s internal training stuff? It seemed like he was saying that the stuff going into the cars got a rewrite.
So, I looked deeper into the datadump I received from my anonymous benefactor - it contains stuff beyond the easily interpretable neural network metadata files. And I found… more networks! A lot more.
So now that I’m looking closer I find that metadata files are only present for a subset of the networks in use in the system. Previously that metadata was the only stuff I was seeing, so I was only seeing a part of what was there. Looking back at older versions it seems like the ‘binary only’ (without metadata) networks have been present in the car for a long time.
Digging around in those ‘binary’ networks I find that there are a bunch of other networks for which there is no metadata but for which there is complete execution code, network weights and so forth. At a minimum there are networks for all the cameras except the backup camera, with independent networks for the main, narrow, fisheye, repeater, and pillar networks. It looks like the pillar cameras both use a single network and the repeater cameras both use a single network - so at a minimum that makes 5 different camera networks shipping in 10.4. That's just the binary ones, the metadata ones are separate.
And further, there seem to be networks that perform processing beyond the single frame analysis that is going on in the camera networks. From symbol names and embedded strings it looks like these nets are analyzing lanes, identifying drivable space, picking out landmarks to help the vehicle locate itself, and detecting moving objects - which is something that you can’t do with the previously identified networks alone. There also seems to be code that integrates radar and vision to match up what the two are seeing and also code that estimates the distance to things based only on vision. Maybe this latter is used to match against radar, or maybe it gets used for the pillar and repeater cameras - i don’t know yet.
There’s lots of stuff that I don’t know yet. It’s unlikely that all these networks are used in all cars because some of them are redundant. So maybe there are different network for dev cars included in the regular firmware. Or maybe some of these networks are true ‘shadow’ networks that are being run in parallel with the currently deployed network in order to evaluate their performance or otherwise gather data. I suppose it’s even possible that geographic differences could account for redundancy (do you need a different network in right hand drive countries?).
So this last comment is my own judgement from the overall class of stuff that seems to be getting processed in these networks: I don’t think there’s any FSD stuff in this regular firmware - not even in shadow mode. All the functions I've seen so far seem to be focused on supporting lane keeping and lane changing. There’s support for identifying traffic signals and traffic signs which is stuff that doesn’t yet appear in the behavior of AP2 in any obvious way, but that by itself isn’t enough to tell me that this code base is being used for FSD related development. The sign and signal features might be there to support EAP in terms of not-yet-usable features (stop sign assist? red light assist?) or they could be there as part of helping the car to locate itself or in support of gathering map data. FSD seems to require a lot more than what I see here right now.
So that means the FSD stuff would be a separate effort for which we have only a promo video and one lone twitter leak to guess from.
Looked at this way 10.4 seems like a much bigger change. For one thing the total size of the neural networks dedicated to vision related tasks has more than doubled, and most of that gain is not in processing the camera output frames directly, but in post-processing that provides higher level understanding of vehicle context. In fact about half of the vision processing in 10.4 is ‘post processing’ and not the camera networks themselves.
Fourth I believe, but who's keeping track anymore to be honest.
It seems like the most "successful" rewrite so I'm willing to cut them some slack. The previous rewrites either led to no user visible improvements or improved one thing at severe cost to other things.
This one is simply better. Pretty much all around. And I think that's a good thing that they were willing to undertake it.
Ah, great. I thought the initial release, speed limited. Then that x-apple guy had a re-write, released as "silky smooth" - was that 17.26 or something? And it culminated as a 17.50.3. But I really don't have a clue, you are probably right.
I can't imagine this work won't be used by FSD. In a big ice storm, my radar conked and I lost all AP but the car clearly saw everything just fine.
Tesla -- please allow L2 using vision only. I can understand putting added constraints but it could clearly identify all the vehicles and lane lines and do its AP lane keep job with vision alone. All systems should be operational for FSD but radar could be the "regulatory" redundancy needed by SAE. Or maybe its many vision networks that act as a redundant system. I don't care. Make it happen. Push the envelope and I'll create some legal cover for you if need be. I've already got my Congressman on barking at NHSTA and his useless colleagues. Lets drop the excuses and ship some alpha software (lets face it, its all alpha from Tesla anyways).
I'll even promise not to throw up stupid clips of my dog driving my car.
Tesla -- please allow L2 using vision only. I can understand putting added constraints but it could clearly identify all the vehicles and lane lines and do its AP lane keep job with vision alone. All systems should be operational for FSD but radar could be the "regulatory" redundancy needed by SAE. Or maybe its many vision networks that act as a redundant system. I don't care. Make it happen. Push the envelope and I'll create some legal cover for you if need be. I've already got my Congressman on barking at NHSTA and his useless colleagues. Lets drop the excuses and ship some alpha software (lets face it, its all alpha from Tesla anyways).
I'll even promise not to throw up stupid clips of my dog driving my car.
Nah, there were 4 rewrites by my count (5 major revisions) by my count.
1. 17.3.15 (TACC) and 17.5.28 (AS). Initial ship (useless garbage up to 35mph which slowly increased).
2. Second rewite was 17.17.4 (full highway speed AS and it worked reasonably well even if it did linear curves).
3. Then the disaster rewrite of 17.26 (silky smooth) (complete loss of confidence in AP's ability to do anything useful).
4. Then Karpathy salvage job (4th) 17.50.3 (ho ho ho, finally something worth using again).
5. Then major rewrite (2018.10.4) (wow, so this is AP1 like parity?).
I'm starting to believe too that as of 10.4, "lane lines" really aren't lane lines, they're more where the vision algorithm thinks the car should be driving (e.g. the input to the control algorithm, the 'where to be' while the control algorithm decides how to get there).
I'm seeing more and more cases where the car cannot navigate a hilly zero-visibility intersection without a lead car (take over immediately), but when there's cars ahead, it draws lane lines largely based off where the cars in front are, and successfully gets through the intersection.
It seems like the "blue car" following mode is gone, and in its place is some smarter variant of that which infers driving location taking cues from the environment.
I'm seeing more and more cases where the car cannot navigate a hilly zero-visibility intersection without a lead car (take over immediately), but when there's cars ahead, it draws lane lines largely based off where the cars in front are, and successfully gets through the intersection.
It seems like the "blue car" following mode is gone, and in its place is some smarter variant of that which infers driving location taking cues from the environment.
Thank you @jimmy_d for bringing this thread out of the jazzercise plane of hell we’ve journeyed through. Lol
Great info as always! Anyway to tell which networks are being executed currently or is that knowledge lost with our unicorn model X?
Thanks!
Great info as always! Anyway to tell which networks are being executed currently or is that knowledge lost with our unicorn model X?
Thanks!
I was thinking that exact thought (lane lines are simply identified drivable path). Blue car is definitely replaced with something better. I can drive on any road where there is a car in front and even after the car leaves, it keeps driving well. (from today).
NOTE: Earlier builds, particularly 17.17.4 could also do roads without any lane lines but it was lost after that version and has only sporadically appeared. AP1 will do some of these roads but not all and weaves a lot more. AP2 was perfect. It drove better than most humans. The only issue, so far, is that it stops for parked cars and will not move around (in fact, it slowly was rolling me into one when I left it alone and only stopped due to ultrasonic ping which resulted in HOLD (for 1 minute before I gave up and disengaged and moved it past the parked car)).
In this picture, I think the car is treating that big black line on the left side of the car as the lane and the curb on the right as the other lane marker. This is what caused the 101 deaf IMHO.
bebbiXpress
Member
What I observed a lot is that before you activate AS there is only one gray line (most of the time it's the center marking) visible
But the second you enable AS, left and right lines turn blue.
I think that supports the assumption that AP is seeing more a drivable path than lines.
But the second you enable AS, left and right lines turn blue.
I think that supports the assumption that AP is seeing more a drivable path than lines.
That's how it used to operate. I could get it to work if there was a fold in the asphalt (middle of the road also highest point for drainage). It stopped doing that sometime before 17.42.1 (maybe 17.40?).
Now it refuses to engage and it does on perfectly smooth concrete roads without lane markings (it might use the curb but it has to use something else too).
It can do curves as well (without a lead car).
It will NOT engage AS without a car in front but earlier builds would recognize lane lines and allow AS by using the crease and the curb. This has the lane lines but still refuses to present AS unless it has that magic lead car (even if its a good distance down the road and is actually a parked car off to the side).
So I understand what you are saying but I'm not sure its the answer anymore. I'll test more and see but the two roads without the center fold/crease still work (they do have curbs). I know of one road with just a grassy verge and no fold and see if it still works with a lead car.
Oh I'm well aware every time I get an AP1 loaner. AP2's first AS "local road" FW would allow AS on all roads. It was magical (even if it was limited to 35mph). They probably directly tried to emulate AP1 back then and so AP1 allowed it and so did AP2. It did such a bad job at it though but it still tried.
This feels way different than AP1 and the earlier AP2. Its so damn good you have to experience it. No wobbles. No hesitation. Just smooth driving right down the center of the "lane".
jimmy_d
Deep Learning Dork
Functionally there's no difference and I don't know why some have external descriptions (metatdata) and others don't. One thought I had was that the external networks might be loadable alternatives to some of the 'built in' ones. That could be useful if Tesla wanted to update the networks without changing the executable binaries. Another possibility is that there are other programs that need to know the structure of some of the networks and the metadata is just there to provide that structural information to those other programs. But I don't know why some of the networks would be like that and others wouldn't. It is kind of notable that only the camera networks ever seem to have metadata versions. I'm unaware of any of the postprocessing networks ever appearing in metadata.
I'm a total noob at taking apart a binary to see what's going on inside. I'm probably missing a lot. At first I wasn't even going to share this because it seems so inconsequential, but I previously made some statements that seem to be contradicted by this new info and I wanted at least to correct that.
I agree completely. I was going to note this, too.
I believe "blue car" tracking is mostly gone. It only appears for a minuscule flash of time very rarely but for the most part I believe 2018.10.4 is using the environment (including cars on the side) to dynamically change the lane. This explains how the car will now "veer" to avoid parked cars in certain cases.
It still seems with 2018.10.4 though that you cannot active AS purely based on a lead car (whereas AP1 does allow you to do this, at least from a stop). So there are improvements to be made for parity reasons.
But I agree: blue car mode seems mostly gone, a lot more environmental factors going into lane creation.
Similar threads
- Article
