Welcome to Tesla Motors Club
Discuss Tesla's Model S, Model 3, Model X, Model Y, Cybertruck, Roadster and More.
Register

Neural Networks

This site may earn commission on affiliate links.
Never seen a blue car in AP2, but often see a "highlighted" car directly in front, which I assumed was more or less the same thing?

Kudos to @jimmy_d for outing "shadow mode" when we all thought it was a myth. The interesting thing is that nobody has found evidence (yet) of any feedback from these binary nets to the mothership.
 
  • Like
Reactions: Cirrus MS100D
Never seen a blue car in AP2, but often see a "highlighted" car directly in front, which I assumed was more or less the same thing?

Kudos to @jimmy_d for outing "shadow mode" when we all thought it was a myth. The interesting thing is that nobody has found evidence (yet) of any feedback from these binary nets to the mothership.

Those highlighted cars (in white) were described in the firmware as "relevant for control", which basically means the car is adjusting its distance regulation or speed based off that highlighted car.

Cars being shown in black are basically decorative in nature, which is one of the reasons why I think there's a lot of misguided attention towards AP1 showing other cars and how important that actually is. You can already tell AP2 is using other cars to get an implied driving path. It's just not displaying them on the screen, but even though AP1 displays them on the screen, it's not really acting upon that information in any meaningful way.
 
  • Informative
Reactions: croman
Those highlighted cars (in white) were described in the firmware as "relevant for control", which basically means the car is adjusting its distance regulation or speed based off that highlighted car.

Cars being shown in black are basically decorative in nature, which is one of the reasons why I think there's a lot of misguided attention towards AP1 showing other cars and how important that actually is. You can already tell AP2 is using other cars to get an implied driving path. It's just not displaying them on the screen, but even though AP1 displays them on the screen, it's not really acting upon that information in any meaningful way.

How do you know Mobileye’s NN doesn’t use other cars? Isn’t it a black box as far as Tesla is concerned?
 
How do you know Mobileye’s NN doesn’t use other cars? Isn’t it a black box as far as Tesla is concerned?
It may very well use other cars for lane line cues, but what I'm saying is that it doesn't use any of the "black cars" on screen to do things like offset itself to avoid cars slightly encroaching on your lane proactively, etc. It's not part of the control decision, which is purely based off the lead car in front or a car to the side encroaching so much that it turns into a white car on screen.

I more am referring to the implication made by many comments here that there's some sort of assumption that because AP1 displays cars to the side, it's somehow driving more safely because it's aware of stupid things that cars to your front corners in adjacent lanes are doing. It's really not doing anything terribly different compared to AP2 in that regard.
 
It may very well use other cars for lane line cues, but what I'm saying is that it doesn't use any of the "black cars" on screen to do things like offset itself to avoid cars slightly encroaching on your lane proactively, etc. It's not part of the control decision, which is purely based off the lead car in front or a car to the side encroaching so much that it turns into a white car on screen.

I more am referring to the implication made by many comments here that there's some sort of assumption that because AP1 displays cars to the side, it's somehow driving more safely because it's aware of stupid things that cars to your front corners in adjacent lanes are doing. It's really not doing anything terribly different compared to AP2 in that regard.

Got it. The AP1 neural net may (or may not be) be using other cars for path detection, but the Tesla code that moves the car doesn't make use of other car positions. In a similar vein, it is annoying in stop and go traffic in my AP1 when I'm following a driver that likes to weave all over the lane, my car ends up weaving all over the lane too, even though the car knows it is weaving over the white lines.
 
Got it. The AP1 neural net may (or may not be) be using other cars for path detection, but the Tesla code that moves the car doesn't make use of other car positions. In a similar vein, it is annoying in stop and go traffic in my AP1 when I'm following a driver that likes to weave all over the lane, my car ends up weaving all over the lane too, even though the car knows it is weaving over the white lines.

Ditto — that was something that bothered me about AP1 too. I half jokingly say sometimes I'm convinced AP1 would follow the lead car over a cliff.

I actually much prefer AP2's behavior as it's far less influenced by the behavior of the lead car versus all the other environmental cues. In that sense, it feels more on the pathway to being a FSD system, rather than just being a really really advanced driver assist.
 
It may very well use other cars for lane line cues, but what I'm saying is that it doesn't use any of the "black cars" on screen to do things like offset itself to avoid cars slightly encroaching on your lane proactively, etc. It's not part of the control decision, which is purely based off the lead car in front or a car to the side encroaching so much that it turns into a white car on screen.

I more am referring to the implication made by many comments here that there's some sort of assumption that because AP1 displays cars to the side, it's somehow driving more safely because it's aware of stupid things that cars to your front corners in adjacent lanes are doing. It's really not doing anything terribly different compared to AP2 in that regard.

I’m driving AP1 and on controlled access highway car won’t pass cars which are on the left lane (it isn’t legal) unless I press the accelerator. So it isn’t purely for decoration.
 
Interesting. We don’t have that behavior in the US. Do you know if AP2 does the same thing or not in your locale?
I don’t know about AP2. But as I edited to my post, it is not legal here on controlled access highway to pass cars which are on the left lane.

The only reason I sometimes need to pass cars which are on my left side is, that rules are different on true controlled acces highways and on the other roads with multiple lines and AP1 sometimes applies this “no overtake from right side” rule also on roads where it is not forbidden.
 
Actually there's quite a bunch of constituents required to have driving faster in the right lane qualify as "overtaking in the right lane", such as specific minimum speed differences, a return to the left lane and such. The ruleset was somewhat restrictively implemented on AP1, so I'm happy it's not implemented (yet) on AP2.

Another reason AP1 displayed vehicles and their type in adjacent lanes was to build confidence with the driver.
Considering how poorly AP2 still handles slow trucks in the right lane when auto lane changing from the left to the center lane (which means it brakes because it thinks the trucks are in the center lane; .12), I assume the display of vehicles in adjacent lanes would look more like a disco and anything but confidence inspiring.
The currently deployed network still seems to have big difficulties in properly matching vehicles to their respective lanes (which would make sense under the condition it's more looking at drivable space rather than "hard" lanes).
 
Very interesting stuff. Is it possible the metadata is just lint from internal documentation? Or does it look like parameter data for something?

An acquaintance suggested that the metadata might just be cruft from a build process that isn't fully optimized, or maybe something that's legacy. That makes some sense except that there appear to be functions in the code that access those files. If it was really vestigial I'd expect the code itself to not have references to the metadata files. Of course the code doing the access could itself be legacy from earlier versions or only employed during testing.
 
  • Helpful
Reactions: strangecosmos
So this last comment is my own judgement from the overall class of stuff that seems to be getting processed in these networks: I don’t think there’s any FSD stuff in this regular firmware - not even in shadow mode. All the functions I've seen so far seem to be focused on supporting lane keeping and lane changing. There’s support for identifying traffic signals and traffic signs which is stuff that doesn’t yet appear in the behavior of AP2 in any obvious way, but that by itself isn’t enough to tell me that this code base is being used for FSD related development. The sign and signal features might be there to support EAP in terms of not-yet-usable features (stop sign assist? red light assist?) or they could be there as part of helping the car to locate itself or in support of gathering map data. FSD seems to require a lot more than what I see here right now.

So that means the FSD stuff would be a separate effort for which we have only a promo video and one lone twitter leak to guess from.

Looked at this way 10.4 seems like a much bigger change. For one thing the total size of the neural networks dedicated to vision related tasks has more than doubled, and most of that gain is not in processing the camera output frames directly, but in post-processing that provides higher level understanding of vehicle context. In fact about half of the vision processing in 10.4 is ‘post processing’ and not the camera networks themselves.

Any idea what is in the latest update? I have a Model 3 and just got 14.7 yesterday (4/18), was on 12.1 before that (got that on 4/13) and before that was on 10.5. On 10.5 my car was slowing when coming into turns after 4/9 when similar behavior was coming out with Model S/X on a new 14.x update.. So my car with AP had new behavior without an update! I thought that was pretty cool, got 10.5 from SC on 3/28. Prior to 4/9 it would continue to go as fast as I had set the speed when turning, it was overshooting lane markings by a lot, and fixing itself but still [yes I was ready to take over :) ].
Since the latest update to 14.7 I haven't noticed any new update to AP since 4/9 and 10.5 AP behavior changes, just curious if you have gotten any new info on the latest updates?
 
At least as far as Tesla has announced/advertised, stop sign and stop light actions are purely in the realm of FSD. EAP has always been advertised as freeway-only.

Yeah it seems that way to me too - EAP being just a freeway/highway feature.

And the sign stuff is mixed in with other landmark identification features so it's possible that signs are just being used as one of a variety of location landmarks. It's also possible that they are mapping the location of signs and signals to support future features (I think the language in the video permission agreement dialog suggests they are doing this somehow).

I've thought a few times that a feature that helps you avoid accidentally running a stop sign or a stop light at speed could be a useful and doable safety improvement. If so then maybe, like AEB, it would be rolled out into all cars or at least into EAP cars. It wouldn't necessarily have to even actuate the brakes - maybe it could just warn the driver.

Those events result in a lot of serious accidents - I've seen it happen myself more than once. My mom was nearly killed in an accident like that two years ago - she was distracted by her passenger and misread a stoplight. Her little Honda ended up getting T-boned by a Chevy Tahoe. Nobody except my mom was seriously injured but it could have been so much worse.
 
  • Informative
  • Like
Reactions: hiroshiy and T-Will
Any idea what is in the latest update? I have a Model 3 and just got 14.7 yesterday (4/18), was on 12.1 before that (got that on 4/13) and before that was on 10.5. On 10.5 my car was slowing when coming into turns after 4/9 when similar behavior was coming out with Model S/X on a new 14.x update.. So my car with AP had new behavior without an update! I thought that was pretty cool, got 10.5 from SC on 3/28. Prior to 4/9 it would continue to go as fast as I had set the speed when turning, it was overshooting lane markings by a lot, and fixing itself but still [yes I was ready to take over :) ].
Since the latest update to 14.7 I haven't noticed any new update to AP since 4/9 and 10.5 AP behavior changes, just curious if you have gotten any new info on the latest updates?

Personally I only study the neural network related features. The driving state machine is pretty opaque and I think it's detailed behavior depends on a lot of external configuration parameters, map data and so forth. There are several other people on TMC who know a lot more about it - maybe one of them will respond to your question.

But I've seen similar changes in behavior on the same version of firmware and other people have reported them as well. I think the general sense is that it's the result of map updates, which happen continuously in the cloud, and of non-firmware configuration changes which get pushed to the cars without notification.
 
  • Helpful
Reactions: immunogold
Seeing the same thing. There's a road near my house where the striping ends and the road is unmarked for another half-mile. I've never been able to drive that last half mile on any hardware/software combination (I've tried them all) without some sort of lurch for the left curb. But now my AP2 Model X just hugs the right curb and track right along without hesitation or steering uncertainty.
 
Never seen a blue car in AP2, but often see a "highlighted" car directly in front, which I assumed was more or less the same thing?

Kudos to @jimmy_d for outing "shadow mode" when we all thought it was a myth. The interesting thing is that nobody has found evidence (yet) of any feedback from these binary nets to the mothership.

Blue car is different than highlighted car. Blue car implies (in AP1), radar only tracking of the lead car (lost lane line recognition). It uses the radar returns and some visual tracking to literally follow the lead car. If lane lines are detected, it will stop but otherwise AP1 (at least it used to) follow the blue car no matter what.

AP2 never quite did that. Blue car was definitely present since 17.11.3. If you never used AP2 on local roads, you probably never saw it because it always focused on lane lines and will show red hands immediate disengagement if it got over its skis. At around 17.40, it started doing blue cars through intersections and that sorted out the herky jerky lane hunting that would happen if someone attempted to go through very large intersections.

Now it sees so well, they either hide blue car mode or it doesn't need it.

Found blue car in AP2 !

Move Over Tesla. Here Comes Cadillac.
 
  • Helpful
Reactions: immunogold
Any idea what is in the latest update? I have a Model 3 and just got 14.7 yesterday (4/18), was on 12.1 before that (got that on 4/13) and before that was on 10.5. On 10.5 my car was slowing when coming into turns after 4/9 when similar behavior was coming out with Model S/X on a new 14.x update.. So my car with AP had new behavior without an update! I thought that was pretty cool, got 10.5 from SC on 3/28. Prior to 4/9 it would continue to go as fast as I had set the speed when turning, it was overshooting lane markings by a lot, and fixing itself but still [yes I was ready to take over :) ].
Since the latest update to 14.7 I haven't noticed any new update to AP since 4/9 and 10.5 AP behavior changes, just curious if you have gotten any new info on the latest updates?

The new behavior is tied to the ADAS tiles and TomTom speed limit database which uses GPS to update speed limit information in the IC. The ADAS tiles have curve information for AP to know how to slow down.

You can read about it in the Tesla Autopilot Maps thread by @verygreen in this particular sub-forum. We are now locked out from accessing the tiles but it was cool seeing the data populate thanks to the awesome work by @DamianXVI