Welcome to Tesla Motors Club
Discuss Tesla's Model S, Model 3, Model X, Model Y, Cybertruck, Roadster and More.
Register

Autonomy Investor Day - April 22 at 2pm ET

This site may earn commission on affiliate links.
Sure about that?

Karpathy mentions shadow mode in the fleet during the autonomy investor presentation, 2:16:35 mark:


Actually he talks about shadow mode in detail at 2:10:00 mark.

Basically he says that shadow mode is them deploying a NN that they are working on and having it run in the car.
For example a NN that detects and predicts that a car is gonna cut in in-front of us. So the NN is always making that predictions.

So lets say the NN makes a prediction that a car to the right is gonna cut in. Then there is a program that takes that predicition and then checks for example three second later to see if the car actually ended up in-front of us/the lane that we are in. If the car did end up infront of it then that counts as a successful prediction.

If it didn't then that's a failed prediction.

So if they deployed a NN for something like this for a week and there were 10 million car cut in predictions for example and then only 85% were successfully predicted as a cut-in by the NN, it would mean they need to improve the NN more.

That's shadow mode.

This is quite different from what Elon Musk said shadow mode was

New Tesla Model S and Model X automobiles will run Autopilot in “shadow mode” and collect driving data that pits a human versus computer. Autopilot vehicles running in shadow mode will not take any driving-assist or self-driving actions. Rather they will only log instances when Autopilot would have taken action and compare those results to the real life actions taken by human drivers. Musk told the press.

So, the car will run Autopilot in “shadow mode” in order for Tesla to gather statistical data to show false positives and false negatives of the software. In shadow mode, the car isn’t taking any action, but it registers when it would have taken action. Then, if the Tesla is in an accident, the company can see if the autonomous mode would have avoided the accident (or the other way around, with the self-driving system potentially causing an accident).

It will record how the car would have acted if the computer was in control, including information about how the car might have avoided an accident (or caused one).​
 
  • Like
Reactions: croman
Actually he talks about shadow mode in detail at 2:10:00 mark.

Basically he says that shadow mode is them deploying a NN that they are working on and having it run in the car.
For example a NN that detects and predicts that a car is gonna cut in in-front of us. So the NN is always making that predictions.

So lets say the NN makes a prediction that a car to the right is gonna cut in. Then there is a program that takes that predicition and then checks for example three second later to see if the car actually ended up in-front of us/the lane that we are in. If the car did end up infront of it then that counts as a successful prediction.

If it didn't then that's a failed prediction.

So if they deployed a NN for something like this for a week and there were 10 million car cut in predictions for example and then only 85% were successfully predicted as a cut-in by the NN, it would mean they need to improve the NN more.

That's shadow mode.

This is quite different from what Elon Musk said shadow mode was

New Tesla Model S and Model X automobiles will run Autopilot in “shadow mode” and collect driving data that pits a human versus computer. Autopilot vehicles running in shadow mode will not take any driving-assist or self-driving actions. Rather they will only log instances when Autopilot would have taken action and compare those results to the real life actions taken by human drivers. Musk told the press.

So, the car will run Autopilot in “shadow mode” in order for Tesla to gather statistical data to show false positives and false negatives of the software. In shadow mode, the car isn’t taking any action, but it registers when it would have taken action. Then, if the Tesla is in an accident, the company can see if the autonomous mode would have avoided the accident (or the other way around, with the self-driving system potentially causing an accident).

It will record how the car would have acted if the computer was in control, including information about how the car might have avoided an accident (or caused one).​

Yes. Karpathy's explanation is what I mean by shadow mode. It is one the tools that Tesla uses to refine the software.
 
tKDv3VXmIWJMFFwL.jpg

Tesla on Twitter

That's not a breakthrough, thats called SLAM which is decades old. Its already on your phones, its what your phone uses to do AR. Its what's being used to do inside out tracking in AR and VR headsets
 
Their NNs are so good differentiating moving and stationary objects. :)


Not only that it can detect minor movements and predict intentions of cars and pedestrians. Like @diplomat33 said this learning can all be done in shadow mode. The NN in cars will make predictions and to compare them to real movements moments later and to use the data to further train the neural net.
 
That's not a breakthrough, thats called SLAM which is decades old. Its already on your phones, its what your phone uses to do AR. Its what's being used to do inside out tracking in AR and VR headsets

Yeah, I know what it is. Assuming it is a near real-time model built by HW2x using passive vision only, then who else is doing this? Intel/ME are using radar for depth, Waymo et al are using LiDAR.
 
Yeah, I know what it is. Assuming it is a near real-time model built by HW2x using passive vision only, then who else is doing this? Intel/ME are using radar for depth, Waymo et al are using LiDAR.
Many companies are working on this. Anyway, it's highly unlikely that Tesla is doing realtime visual SLAM in their production cars now, otherwise they wouldn't have problems detecting stationary obstacles on the road such as stopped firetrucks etc. If I remember correctly Karpathy even mentioned that what they use right now is "approximate" and "more sparse" that what was shown in that video clip.
 
A short summary from CNET. What showed at 0:51 is what the Tesla vision sees and what mentioned in the #HyperChanger video. This I believe is the first time we see that. Size of those yellow dots seem to indicate distance of objects (or degree of hazard?). Very cool.



Yeah, I know what it is. Assuming it is a near real-time model built by HW2x using passive vision only, then who else is doing this? Intel/ME are using radar for depth, Waymo et al are using LiDAR.

Tesla also uses radar data to train and refine vision NN, all done automatically. It's mentioned in Karpathy's presentation in the investor event.
 
Last edited:
A short summary from CNET. What showed at 0:51 is what the Tesla vision sees and what mentioned in the #HyperChanger video. This I believe is the first time we see that. Size of those yellow dots seem to indicate distance of objects (or degree of hazard?). Very cool.
This was shown in Karpathy's presentation. The yellow dots are radar detections overlaid over the bounding boxes produced by the computer vision. It's about the fusion of visual and radar input.
 
  • Informative
Reactions: Cirrus MS100D
This might be a little off topic but since we were discussing Elon's claims of L5 autonomy, I thought I would bring this up.

Back in Jan 2018, GM promised a Chevy Bolt with no steering wheel at all for this year (2019). They promised they would produce 2,500 Bolts with no steering wheel, even released a teaser image too.
Look, Ma, No Steering Wheel Or Pedals In GM's Robo-Taxi, Coming In 2019

It's now April 2019 and as far as I know, GM does not have L5 and is nowhere close to producing Bolts with no steering wheel. Let's see if GM can keep their promise or if they will miss it.
 
  • Like
Reactions: croman
This might be a little off topic but since we were discussing Elon's claims of L5 autonomy, I thought I would bring this up.

Back in Jan 2018, GM promised a Chevy Bolt with no steering wheel at all for this year (2019). They promised they would produce 2,500 Bolts with no steering wheel, even released a teaser image too.
Look, Ma, No Steering Wheel Or Pedals In GM's Robo-Taxi, Coming In 2019

It's now April 2019 and as far as I know, GM does not have L5 and is nowhere close to producing Bolts with no steering wheel. Let's see if GM can keep their promise or if they will miss it.

Did I miss the part where GM sells these Level 5 capable cars to customers since 2016?
 
This might be a little off topic but since we were discussing Elon's claims of L5 autonomy, I thought I would bring this up.

Back in Jan 2018, GM promised a Chevy Bolt with no steering wheel at all for this year (2019). They promised they would produce 2,500 Bolts with no steering wheel, even released a teaser image too.
Look, Ma, No Steering Wheel Or Pedals In GM's Robo-Taxi, Coming In 2019

It's now April 2019 and as far as I know, GM does not have L5 and is nowhere close to producing Bolts with no steering wheel. Let's see if GM can keep their promise or if they will miss it.

We've been over this before. There is a huge difference between talking about future products -- which every company does and every company often misses on -- and selling future products to paying customers. Promise all you like, but if you accept payment for it, you have crossed a line. In particular, if you do more than talk about the future but put out a faked video showing it working right now in the present and suggesting that "validation and regulatory approval" is all that's left to do, and then accept payment, you are committing fraud.

The bragging from GM is not even close.
 
Just as an aside guys, I must say I’ve never been quite as concerned about demand before this. Not that I believed the production constrained mantra for years but something more than before has happened.

And it must be feeding itself too because who wouldn’t have heard concerning stories about Tesla now. Are people hesitant to buy or what is going on? The stock is going absolutely to heck as we speak and even many longs are jumping ship.

They need to stay afloat to deliver my Level 5! :)
 
  • Like
Reactions: Daniel in SD
By the way, who is surprised that Tesla delayed their FSD price increase from May 1 to May 10?

Demand levers losing steam?
Only for those unable to order by May 1st due to not having an order page....
We will delay $1k full self-driving option price increase until May 10 for those unable to order by May 1
UK order page will go live around May 1 or 2
 
We've been over this before. There is a huge difference between talking about future products -- which every company does and every company often misses on -- and selling future products to paying customers. Promise all you like, but if you accept payment for it, you have crossed a line. In particular, if you do more than talk about the future but put out a faked video showing it working right now in the present and suggesting that "validation and regulatory approval" is all that's left to do, and then accept payment, you are committing fraud.

The bragging from GM is not even close.

No, I get the difference between promising a future product and accepting payment in advance for a future product. Yes, it's much worse to miss on a product you accepted payment on. But 1) Tesla did deliver on the hardware part of the L5 promise (AP3). 2) They were always clear that it would take time to deliver the software part of L5. 3) Tesla has delivered incremental new features and value to customers as they get closer to L5. Lastly, 4) Tesla is working hard and making progress towards L5, doing a big autonomy investor day with test drives and a demo to show they are making progress towards L5. Accepting payment for a future product, delivering some value to the customer and showing you are making progress towards the final product is fine. Of course, it does not absolve Tesla of their responsibility to actually deliver on L5.

Another way to look at it:
1) GM promises a future product (robotaxis) with the only evidence being a few demos of full self-driving we've seen. No evidence of production or that they can deliver the future product. GM will probably be late at delivering their robotaxis or, more likely, just forget about it completely. I can't tell you how sick I am of these legacy auto makers always showing concept cars and future products that never see the light of day.
2) Tesla promises the same future product (robotaxis) but actually delivers cars with the hardware AP3), delivers incremental features towards the final goal, and shows steady progress towards that goal. That is better in my opinion.
 
This might be a little off topic but since we were discussing Elon's claims of L5 autonomy, I thought I would bring this up.

Back in Jan 2018, GM promised a Chevy Bolt with no steering wheel at all for this year (2019). They promised they would produce 2,500 Bolts with no steering wheel, even released a teaser image too.
Look, Ma, No Steering Wheel Or Pedals In GM's Robo-Taxi, Coming In 2019

It's now April 2019 and as far as I know, GM does not have L5 and is nowhere close to producing Bolts with no steering wheel. Let's see if GM can keep their promise or if they will miss it.

Where did you see GM say they were developing L5? No steering wheel ISN'T Level 5.

Level 5 = no geo-fencing or limitations, that means the car can drive anywhere, on any roads, in any weather condition and any road condition.

This has been repeated 1,000 times...
 
  • Like
Reactions: J1mbo