AP2.0 Cameras: Capabilities and Limitations?

[stopcrazypp]

Furthermore, $20 sensors do not cost $0 to integrate and test. And when they find issues they're pretty much SOL unless the supplier is willing to play ball (not at all likely for a small fish like Tesla working with German behemoths like HELLA)

I think this is more for the FSD side, where as per recent discussion it may be desirable to wipe more than just rain. Using the cameras gives them more flexibility and gives coverage of the field of view of the cameras (not just the rain sensor).

 

[cartwright]

Drove in the rain yesterday for the first time with the auto wipers. I feel like they performed better than my Lexus. There were times at night when it probably could have been more aggressive, but never a point where I felt like I needed to manually intervene.

Also, as we use these and manually override what they are doing, I'm sure Tesla is gathering that data, so they will only get better. Pretty darn cool and, IMO, far superior to a sensor that is what it is, until it breaks.

 

[lunitiks]

A dedicated rain sensor doesn't 'break'.

But I agree with the rest. The superiority lies in the OTA-upgradeability (its potential)

 

[Stoneymonster]

A dedicated rain sensor doesn't 'break'.

But I agree with the rest. The superiority lies in the OTA-upgradeability (its potential)

Each additonal sensor requires additional wiring, harness, routing, CAN bus connection, etc. it’s not just the sensor.

 

[lunitiks]

Fair enough

 

[alexdav]

Was driving back in the Model 3 from AZ and got caught in the rain. I felt the auto wipers worked well but were too slow in the heavy downpours. Manually turning it to fast worked much better.

Also, this car is my first experience with AP2.0. Seems to me the biggest improvement (at this point) is TACC. I hardly ever drive on a 1 or 2 on AP1.0 because it's scary but AP2.0 handles slowing down so much better.

Only thing that I've really noticed that isn't on par with AP1.0 is curves on hills. The car seems to take those very wide and seemed a tad bouncy. I think AP1.0 tends to be wide as well, but tracks it slightly better.

 

[Bladerskb]

lmao you Tesla fans never cease to amuse me. you are happy for a feature that's supposed to come standard and is actually 16 months late!

Are you ppl really this naive? your guys isn't getting better as you think, its just finally getting the features that were promised almost 2 years ago.

Wake up!

 

[daud]

Greater potential, but safety first: maybe Elon can tell us more about why the anti-rain AI (which turns out to also be cornflake and chicken-enchilada phobic) took so much time youtu.be/7zcAMyNcAk8?t=18m8s

 

[Stoneymonster]

lmao you Tesla fans never cease to amuse me. you are happy for a feature that's supposed to come standard and is actually 16 months late!

Are you ppl really this naive? your guys isn't getting better as you think, its just finally getting the features that were promised almost 2 years ago.

Wake up!

Yeah, but we’re still driving Teslas, which thus far is enough to cover a multitude of sins.

 

[SucreTease]

There's exactly zero evidence anything of the sort was happening on any sort of scale outside of development cars.

How is this for evidence? Tesla wrote the following in its report to the California DMV:

Tesla conducts testing to develop autonomous vehicles via simulation, in laboratories, on test tracks, and on public roads in various locations around the world. Additionally, because Tesla is the only participant in the program that has a fleet of hundreds of thousands of customer-owned vehicles that test autonomous technology in “shadow-mode” during their normal operation (these are not autonomous vehicles nor have they been driven in autonomous mode as defined by California law), Tesla is able to use billions of miles of real-world driving data to develop its autonomous technology.

 

[verygreen]

How is this for evidence? Tesla wrote the following in its report to the California DMV:

Personally I think this is a lie. Remember I had full control of ape in my car for 8 months, I can see what the car was doing, what it was reporting and such.

To make a credible claim Tesla needs to detail what does the "shadow mode" do exactly and it then needs to align with my observations. So far nobody ever came with any sort of credible explanation of what the "shadow mode" is doing that matches against empirical observations.

The closest to "Shadow mode" I saw was the "create a snapshot if we think we should be doing AEB now".

 

[MarcusMaximus]

The closest to "Shadow mode" I saw was the "create a snapshot if we think we should be doing AEB now".

That could be exactly what they mean by "Shadow mode". It would make sense to just upload relevant data about an event and train on simulations based on that event on a server somewhere rather than unnecessarily taxing the APE to run simulation code. Even if it has spare cycles right now, they'd eventually have to do that anyway, since it's extremely unlikely to have enough processing power for both eventual FSD *and* running simulations for future software.

 

[lunitiks]

Good thing Tesla invented the term shadow mode so it could mean whatever they want ;-)

 

[chillaban]

That could be exactly what they mean by "Shadow mode". It would make sense to just upload relevant data about an event and train on simulations based on that event on a server somewhere rather than unnecessarily taxing the APE to run simulation code. Even if it has spare cycles right now, they'd eventually have to do that anyway, since it's extremely unlikely to have enough processing power for both eventual FSD *and* running simulations for future software.

Except when people say "shadow mode", they seem to be thinking it is more significant than that, like it learns or records clips during disengagements to be trained against. AEB snapshots don't make for better Autosteer performance, which is what most people are desiring.

 

[stopcrazypp]

Except when people say "shadow mode", they seem to be thinking it is more significant than that, like it learns or records clips during disengagements to be trained against. AEB snapshots don't make for better Autosteer performance, which is what most people are desiring.

It's not only doing AEB snapshots, but I remember it also taking snapshots that fit some classification profiles (I remember hills specifically).

 

[chillaban]

It's not only doing AEB snapshots, but I remember it also taking snapshots that fit some classification profiles (I remember hills specifically).

Sure, and it was collecting video clips in certain areas too. They post new trigger events once in a while to collect things that they care about collecting, but that sounds very different to me compared to "billions of miles" of shadow mode. It's more like out of those billions of miles, only a small (probably fraction of a percent) subset of those miles are collected.

FWIW I've been using AP2 on a regular basis on challenging suburban/residential roads, and have at least a disengagement or two per day. Back on 2017.42/.48 it used to upload hundreds of MB per day on wifi when I got home, but lately it's not collected much of anything.


I'm not saying that shadow mode is completely made up. More that it's a lot more limited in scope compared to how it's represented (or what people think it is doing).

 

[lunitiks]

The illusion of «shadow mode»: Your car ‘observes’ your every move. It ‘plans ahead’ and ‘determines what it would do’ if it was in control. But it isn’t - it’s in the secret «shadow mode». Your 40 x mobileye, easily upgradeable super computer in the glove box either ‘learns’ or ‘gloats’ when you deviate from what it’s planned. Keep this up long enough, and AI will rise from it!

Guess what? The car doesn’t learn anything.

It doesnt get better by itself. Theres no training going on inside your Tesla. Not in ‘live’ mode, not in shadow mode, not in any mode.

Your car can LOG stuff you do (did), and send the logs or snapshots back to tesla at an appropriate time later if/when your wifi connected.

God knows what they do (or dont do) with this data!!!

 

[stopcrazypp]

Sure, and it was collecting video clips in certain areas too. They post new trigger events once in a while to collect things that they care about collecting, but that sounds very different to me compared to "billions of miles" of shadow mode. It's more like out of those billions of miles, only a small (probably fraction of a percent) subset of those miles are collected.

FWIW I've been using AP2 on a regular basis on challenging suburban/residential roads, and have at least a disengagement or two per day. Back on 2017.42/.48 it used to upload hundreds of MB per day on wifi when I got home, but lately it's not collected much of anything.


I'm not saying that shadow mode is completely made up. More that it's a lot more limited in scope compared to how it's represented (or what people think it is doing).

I remember shadow mode mentioned in a couple of different contexts, but mostly when a software release isn't ready and Tesla is still doing testing. For example, AEB had been in "shadow mode" a couple of times, as had AP2 before they finally released a functional version. There is no second FSD instance that runs (which is what some people seem to assume), but just AEB and AP2 instance that runs even when it is not active and logs data it deems interesting.

 

[MarcusMaximus]

It's more like out of those billions of miles, only a small (probably fraction of a percent) subset of those miles are collected.

It's exactly that. Collecting everything wouldn't just be wasteful, it'd be actively harmful. The vast majority of the time your car is just traveling normally. If you trained on all of that, you're going to overfit the model to those circumstances and completely wash out the challenging instances you actually care about.

Choosing how much of which type of data to feed to a model during training is actually a big topic generally in ML. You have to carefully balance the different classes of input to ensure you don't overfit the model to specific circumstances just because of selection bias.

 

[AnxietyRanger]

It's exactly that. Collecting everything wouldn't just be wasteful, it'd be actively harmful. The vast majority of the time your car is just traveling normally. If you trained on all of that, you're going to overfit the model to those circumstances and completely wash out the challenging instances you actually care about.

Choosing how much of which type of data to feed to a model during training is actually a big topic generally in ML. You have to carefully balance the different classes of input to ensure you don't overfit the model to specific circumstances just because of selection bias.

I guess the big question there is: Is Tesla actually feeding anything from the "shadow mode" into an NN at all? Or is the data, whatever they collect, used for something else.

Also, the idea of shadow mode wasn't necessarily ever that it would teach NNs, but that it would verify NN behavior. NN driving shadow, driver driving rea. Obviously you'd want to know the car's drive would match the driver's drive during normal miles as well if that case, and get data on that. No?

Not all (if any?) of the shadow mode data, whatever it is, is destined to teaching NNs.