Welcome to Tesla Motors Club
Discuss Tesla's Model S, Model 3, Model X, Model Y, Cybertruck, Roadster and More.
Register

Were HW3-gen NNs running passively since July?

croman

Active Member
Nov 21, 2016
4,646
6,592
Chicago, IL
I believe green's claim is nothing more than that HW2/3 Teslas aren't recording video or object detections all the time and uploading all of them.

I think continuous recording is what green interprets as “shadow mode”, but I don't interpret it that way.

I interpret shadow mode as being just another set of event-based triggers of the sort that green has confirmed exist. In the context of shadow mode, events that trigger snapshots are likely disagreements between the trajectory outputted by the Autopilot planner and the car's trajectory while being fully human-driven.

Your beliefs are groundless. Green has explicitly debunked your interpretation of his own findings. You have no car or primary sources.
 

Bladerskb

Senior Software Engineer
Oct 24, 2016
2,177
2,465
Michigan
You are just making stuff up without a real basis in reality.

Well actually he's right. Here's what verygreen said about it. He has truly debunked it

"We'll start with the bitter truth. The "shadow driver that just sits there in the computer comparing notes and sending discrepancies and interesting events to Tesla" is a myth. I used to think people just misunderstood Elon, but now I believe Tesla lies about it on purpose 2/"
Twitter

Your beliefs are groundless. Green has explicitly debunked your interpretation of his own findings. You have no car or primary sources.

"...There is a relatively small chance any particular hard brake/disengagement event is going to generate a snapshot for upload but only if all the other stars align right."
green on Twitter

It has been debunked but some people can't let it go.
 
Last edited:

NHK X

Member
Nov 18, 2017
766
626
PNW
Your beliefs are groundless. Green has explicitly debunked your interpretation of his own findings. You have no car or primary sources.

I believe he has also admitted he doesn’t see everything.


I do enjoy reading about what he discovers though.
 

croman

Active Member
Nov 21, 2016
4,646
6,592
Chicago, IL
I believe he has also admitted he doesn’t see everything.

Sure but my points still stand. I said he isn't God and isn't all knowing but he's the best source and he sees enough to suss out shadow mode, as Tesla sees it, cannot exist.


Actually blader pulled a lot of them but his Twitter feed contains other examples of debunking shadow mode. It's just not there and even hw3s second node being activated just indicates redundant take over capability not AB comparisons or dueling minds like some fantasize about.
 

tyson

Member
Nov 13, 2016
479
386
IA
Karpathy talks about the real fleet learning mechanism during the autonomy day. Starts at about the 2:05:00 mark. Basically they ask the fleet for images or situations using triggers. This is something green verified.

Now, some things do run behind the scenes, again something green verified when he posted about the stop light/stop sign detection being present some months ago. I had this feature turned on myself for a while before Tesla disabled it. The specific flag was "GUI_stopAndGo". It wasn't as advanced as what was just released but it detected stop signs and lights along with their states.
This wasn't some full NN running in the background just a portion that wasn't exposed to the GUI or used by the path planner/driving policy. So yes, of course some things run in the background without the user knowing or seeing, but the biggest method for fleet learning is still the triggered data snips, usually just pictures from what karpathy talked about.
 

NHK X

Member
Nov 18, 2017
766
626
PNW
there's no "background shadow NN" pretty much makes this thread asked and answered.

green on Twitter

Then for those of us not in the programming/NN scene, can someone differentiate the difference between people talking about how visualizations have been around for the last year and shadow mode? As a layperson, to me that sounds like a method of running a program in the background. Which sounds like shadow mode. Or is the debate that path decision making is not compared and sent back to Tesla for analysis?

sorry if this is a very elementary question.
 
  • Helpful
Reactions: strangecosmos2

strangecosmos2

Koopa Troopa
Nov 24, 2019
177
120
New Donk City
green's tweets (which I already linked to earlier in this thread) are consistent with how I envision shadow mode. I don't know where people are getting the idea that shadow mode means a car records data of every second it's in motion and uploads it all to Tesla. When did Elon ever say that?

In my mind, shadow mode is just event-based triggers of small data snapshots (e.g. maybe 5-second video clips) where the event is a disagreement between the (passively running) Autopilot planner's path and the human driver's path.

This wouldn't necessarily be a separate neural network running in the background; it could just be the same neural network(s) that run when Autopilot is active running in the background.

Aurora describes a quite similar idea in their ICML 2019 talk on slides 14-17 from 13:15 to 16:30.
 
Last edited:
  • Disagree
Reactions: croman

tyson

Member
Nov 13, 2016
479
386
IA
This wouldn't necessarily be a separate neural network running in the background; it could just be the same neural network(s) that run when Autopilot is active running in the background.
I think this is the sticking point. I cannot speak for the planner, but object detection does run when autopilot is not engaged. This I have seen. As to the rest there is no evidence for it and all indications from Tesla are that disengagements are used to correct the path planner/driving policy. This was also discussed during the autonomy day speech by karpathy.
 
  • Helpful
Reactions: strangecosmos2

strangecosmos2

Koopa Troopa
Nov 24, 2019
177
120
New Donk City
I think this is the sticking point. I cannot speak for the planner, but object detection does run when autopilot is not engaged. This I have seen.

Indeed. green's videos demonstrate this. So, we know for a fact that computer vision NNs run passively in the background. Whether you want to call this "shadow mode" is purely semantic and irrelevant to the OP of this thread.

As to the rest there is no evidence for it and all indications from Tesla are that disengagements are used to correct the path planner/driving policy. This was also discussed during the autonomy day speech by karpathy.

The evidence for it is simply that Elon's mentioned it in the past and that Aurora has also described a similar approach, lending credibility to the idea that this is something Tesla would try. This isn't definitive evidence that it is being implemented, but there also isn't any evidence to the contrary, as far as I know.
 
  • Disagree
Reactions: croman

NHK X

Member
Nov 18, 2017
766
626
PNW
Also, what is Tesla uploading all to the time? I. The last month about a gig of data has been uploaded on my WiFi network, not sure how much gets uploaded on cellular.

I imagined most of this data was somehow relevant data to help improve AP. Is that magical thinking?
 

tyson

Member
Nov 13, 2016
479
386
IA
The evidence for it is simply that Elon's mentioned it in the past
Link? Also Elon has said A LOT of things that never ended up coming true. So maybe add a touch of salt. I trust what Karpathy says in relation to AP far more than Musk. Karpathy writes the code ...
 
  • Like
Reactions: croman

tyson

Member
Nov 13, 2016
479
386
IA
And again, they do gather a LOT of data... I just see no evidence of path planning/driving policy running full time when AP is not engaged.

Most times you can see this yourself if you force a disengagement of AP and monitor the cars data usage. It will wait for WiFi to upload. I have seen it not send data after a disengagement though as well.
 
  • Like
Reactions: croman

strangecosmos2

Koopa Troopa
Nov 24, 2019
177
120
New Donk City
If Tesla uploads a data snapshot for some/most/all crashes or "crash-like events" (which it apparently tracks, per its Safety Reports), and if that snapshot includes the "orange ribbon" and other planning-related data, then this would allow Tesla to determine whether Autopilot would have averted a crash or "crash-like event".

For example, if a Tesla owner drives off a cliff, and the orange ribbon is not pointing off the cliff, that's a good indication the crash would have been avoided on Autopilot.

Determining when Autopilot would have caused a crash when the human did not crash is trickier. That's why I hypothesize the system takes note whenever a human driver does not follow the orange ribbon.

Elon didn't say that the fleet would upload 100% of the instances where Autopilot would have either prevented or caused a crash. He just said the fleet would be used to collect a statistically significant sample size of such instances.

I also don't know if Elon ever said shadow mode would be used to collect data to train neural networks, as opposed to simply serve as a form of safety validation. The training part might just be my own hypothesis, extrapolating from what Elon and others have said.

TL;DR: Shadow mode probably exists in some form. :p
 
Last edited:
  • Disagree
Reactions: croman

Bladerskb

Senior Software Engineer
Oct 24, 2016
2,177
2,465
Michigan
Yeah, I suppose that is probably some form of it. Have to ask him.

No that's the path prediction NN model that the classical control cpu path planning system uses as one of its input to drive.
Its not the actual path planning or driving policy system. It doesn't make decisions, it doesn't say go around this object or avoid this object or don't hit this. It simply draws a line straight ahead through a path. The line literally goes through an object in-front of it. its not the path planning system. The entire path planning is a classical control algorithm. cpu code.

In my mind, shadow mode is just event-based triggers of small data snapshots (e.g. maybe 5-second video clips) where the event is a disagreement between the (passively running) Autopilot planner's path and the human driver's path.

This is directly contradicted by this

"We'll start with the bitter truth. The "shadow driver that just sits there in the computer comparing notes and sending discrepancies and interesting events to Tesla" is a myth. I used to think people just misunderstood Elon, but now I believe Tesla lies about it on purpose 2/"
Twitter


 
Last edited:

About Us

Formed in 2006, Tesla Motors Club (TMC) was the first independent online Tesla community. Today it remains the largest and most dynamic community of Tesla enthusiasts. Learn more.

Do you value your experience at TMC? Consider becoming a Supporting Member of Tesla Motors Club. As a thank you for your contribution, you'll get nearly no ads in the Community and Groups sections. Additional perks are available depending on the level of contribution. Please visit the Account Upgrades page for more details.


SUPPORT TMC
Top