AP2.0 Cameras: Capabilities and Limitations?

[Reciprocity]

2 days ago I was driving during medium snow fall and neither TACC nor EAP could be engaged almost right from the start of the drive. There was no chance that snow or slug already blocked the sensor on the bumper so my guess is there was too much radar reflection by the falling snow or camera visibility was too low. I lean to the first as I was driving in worse condition (heavy rain) and EAP was still available.

I had a similar situation. What version are/were you on? I believe this is a new "Feature" to deal with very large snow flakes. I know it sounds crazy, but I was on 50.3 I believe and it had just started snowing. I had EAP on and I would see huge flakes, some the size of a baseball only flat, I kid you not. Every now and again, the car would see one of the huge flakes as a vehicle and slam on the brakes. It totally freaked me out. It would do that thing on the screen when cars are crossing in front of you as you approach them, only there was no car. Only huge snow flakes. During the same drive, the snow flakes become more normal in size and no more phantom cars. It happened 3 times in the span of 5 minutes then never again. Since updating to 18.4.1 and 18.6, I have not been able to engage EAP in heavy snow fall. Before this recent incident, I never had any problems with heavy snow or never not been able to engage EAP, even with a snow build up on the bumper. This seems new, but whats really odd is how fast Tesla was able to limit EAP while they work on a fix for the huge flakes? I know I sound crazy, but it was clear as day. Huge flake.. phantom car and breaking, then nothing until more huge flakes. These where abnormally large flakes, like I said 2-3" in diameter.

 

[bebbiXpress]

What version are/were you on?

2018.4.1

 

[Pale_Rider]

One interesting point made by the service dude was that the last few updates have been significantly bigger in file size. He said that they used to need about 15 mins to transfer to the car over wifi in the SvC but recently they are between 1 and 2 hours.
He stated that since late December, the cars are now 'using' the 4 side cameras. By this, he clarified, the AP is now in ghost mode, not actively using them to influence the drive. When you brake, it checks if it wanted to brake. When you shift left in the lane, it checks if it would have done the same. etc etc. He said that he knows the cameras have just started being used as they had a flurry of service visits in January for cameras not working. They had in fact never worked but never needed to. There was even one where the cables were not plugged into the camera in the factory but had gone unnoticed until now as there were no calls for the camera to do anything. All that has changed now. Fingers crossed, this means that the learning will turn into doing in the near future.
I am aware this is not new news as this ghost mode has been discussed on here before but I thought I'd share how the service tech explained it to me. Sounds like progress to me.

Probably just lack of understanding by a service tech but maybe....

 

[BigD0g]

Probably just lack of understanding by a service tech but maybe....

No, siggy or his tech are wrong or he misheard. The side cameras aren’t even calibrated yet as of 18.4, and the NN haven’t changed since Nov. I can provide proof later if there’s a question, I’m sure @verygreen can verify this as well

 

[Pale_Rider]

No, siggy or his tech are wrong or he misheard. The side cameras aren’t even calibrated yet as of 18.4, and the NN haven’t changed since Nov. I can provide proof later if there’s a question, I’m sure @verygreen can verify this as well

Yeah that’s what I figured based on what y’all have been saying. One day hopefully.

 

[BigD0g]

Yeah that’s what I figured based on what y’all have been saying. One day hopefully.

Three months maybe, six months definitely...

 

[croman]

Three months maybe, six months definitely...

At least there probably isn't a rear radar that Tesla concluded we all needed but can't get.

Otherwise it'd be AP 3 maybe, AP 6 definitely.

 

[bradbrok]

No, siggy or his tech are wrong or he misheard. The side cameras aren’t even calibrated yet as of 18.4, and the NN haven’t changed since Nov. I can provide proof later if there’s a question, I’m sure @verygreen can verify this as well

I think the NN doesn't have to change much between major revisions even. The NN could be fine but the programming based on the NN output can be drastically changed to provide different results. Like when the NN is considered finished, it won't change at all most likely, and only the procedural code changes. I have a feeling that this is more likely the case.

 

[chillaban]

I think the NN doesn't have to change much between major revisions even. The NN could be fine but the programming based on the NN output can be drastically changed to provide different results. Like when the NN is considered finished, it won't change at all most likely, and only the procedural code changes. I have a feeling that this is more likely the case.

Ditto. Lately my biggest issues have been what I'd consider failures/safety limits in the control algorithm. The lane lines drawn on screen correctly show the car's placement but the placement itself is wrong.

Most of the times this is on sweeping curves or on very sharp curves where it sure looks like the system has a max turning angle limit for a given speed.

They can make very noticeable performance improvements just based off addressing these control algorithm limitations. NN updates aren't the only form of improving AP2.

 

[BigD0g]

I think the NN doesn't have to change much between major revisions even. The NN could be fine but the programming based on the NN output can be drastically changed to provide different results. Like when the NN is considered finished, it won't change at all most likely, and only the procedural code changes. I have a feeling that this is more likely the case.

We'll yes of course, I was only stating the NN has not changed at all since early Nov, which is what the OP was alluding to which is patently incorrect. As for algorithm changes, I'm not sure, as I haven't been tracking that, but that's probably something to add to my list of interesting files to watch

 

[run-the-joules]

Ditto. Lately my biggest issues have been what I'd consider failures/safety limits in the control algorithm. The lane lines drawn on screen correctly show the car's placement but the placement itself is wrong.

Most of the times this is on sweeping curves or on very sharp curves where it sure looks like the system has a max turning angle limit for a given speed.

They can make very noticeable performance improvements just based off addressing these control algorithm limitations. NN updates aren't the only form of improving AP2.

Yeah, the propensity for the car to hug the *outside* of a lane during a turn in some situations makes me uncomfortable.

 

[chillaban]

Yeah, the propensity for the car to hug the *outside* of a lane during a turn in some situations makes me uncomfortable.

Yeah. If they fixed that, I would credit that AP2 build as the next big leap after that 2017.40 update. The main thing wrong with today's builds (at least up to 2018.4.x) is their propensity to bias inappropriately on curves…. and that's been going on since like October.


P.S. Also wanted to say, up to 2017.50.x, there were huge problems with hill cresting and going over bumps causing Autosteer to wig out. I am happy to say, with 2018.4.5, I've not had that happen a single time.

 

[BigD0g]

Yeah. If they fixed that, I would credit that AP2 build as the next big leap after that 2017.40 update. The main thing wrong with today's builds (at least up to 2018.4.x) is their propensity to bias inappropriately on curves…. and that's been going on since like October.


P.S. Also wanted to say, up to 2017.50.x, there were huge problems with hill cresting and going over bumps causing Autosteer to wig out. I am happy to say, with 2018.4.5, I've not had that happen a single time.

Interesting, mine still wigs out all over the place.... hmm.....

 

[chillaban]

Interesting, mine still wigs out all over the place.... hmm.....

Could this be the California maps theory? Recent builds have been making my jaw drop when it comes to local driving. On highways, I routinely make it between San Jose and San Francisco at 70-80mph with zero interventions.

I've set up a batch job to monitor the live map tile date stamps but I haven't seen any updates past the 2017-05 maps.

 

[Reciprocity]

Ditto. Lately my biggest issues have been what I'd consider failures/safety limits in the control algorithm. The lane lines drawn on screen correctly show the car's placement but the placement itself is wrong.

Most of the times this is on sweeping curves or on very sharp curves where it sure looks like the system has a max turning angle limit for a given speed.

They can make very noticeable performance improvements just based off addressing these control algorithm limitations. NN updates aren't the only form of improving AP2.

I will have to check that hill cresting issue. It used to work well on a few hill crests that I encounter regularly but as you noted it start to wig out. It would be nice if that was fixed and I can easily verify because I am very familiar with a couple of spots that used to work then started wigging out.

 

[BigD0g]

Could this be the California maps theory? Recent builds have been making my jaw drop when it comes to local driving. On highways, I routinely make it between San Jose and San Francisco at 70-80mph with zero interventions.

I've set up a batch job to monitor the live map tile date stamps but I haven't seen any updates past the 2017-05 maps.

It's possible, but we're not even sure those maps are used yet in "our" cars, we'd need APE access to figure this mystery out =(.

 

[chillaban]

I will have to check that hill cresting issue. It used to work well on a few hill crests that I encounter regularly but as you noted it start to wig out. It would be nice if that was fixed and I can easily verify because I am very familiar with a couple of spots that used to work then started wigging out.

Yeah I'm curious what you'll find. I'm definitely confident one of the post-2017.50.3 builds made a huge difference for me. On my commute there's many bridges that are poorly joined, so it creates a combination of a bump and a hill. Before these 2018 builds, there was a good 50% chance of the car veering in an unwanted direction shortly after hitting the bump. Since the new builds, I've commuted dozens of times without seeing that behavior at all.

 

[Reciprocity]

Yeah I'm curious what you'll find. I'm definitely confident one of the post-2017.50.3 builds made a huge difference for me. On my commute there's many bridges that are poorly joined, so it creates a combination of a bump and a hill. Before these 2018 builds, there was a good 50% chance of the car veering in an unwanted direction shortly after hitting the bump. Since the new builds, I've commuted dozens of times without seeing that behavior at all.

18.6 and 18.6,1 tested.. Yes, I got the update last night after testing on the way home so I could test on the way to work this morning. The bad news is it was as bad as usual. What is interesting is how consistent it is. Its always exactly the same bad. In general these roads are as bad as they get for AP. Bad road markings, narrow lanes and several crests. On every crest, the car darts back and forth and more then likely would enter the other lane on a very narrow road so I tend to have to stop it as to not freak out the other drivers. So my report is that nothing has changed. Now whats odd is that it did not have these problems last summer. It would handle these roads and crests better if not totally normally without any indecision detected.

 

[croman]

18.6 and 18.6,1 tested.. Yes, I got the update last night after testing on the way home so I could test on the way to work this morning. The bad news is it was as bad as usual. What is interesting is how consistent it is. Its always exactly the same bad. In general these roads are as bad as they get for AP. Bad road markings, narrow lanes and several crests. On every crest, the car darts back and forth and more then likely would enter the other lane on a very narrow road so I tend to have to stop it as to not freak out the other drivers. So my report is that nothing has changed. Now whats odd is that it did not have these problems last summer. It would handle these roads and crests better if not totally normally without any indecision detected.

Last summer wasn't this NN. The control programming doesn't match this NN. 17.17.4 was still the most stable build but it wasn't that great so image recognition... They need to convinc both aspects to have succeed

 

[verygreen]

Interesting link I found: Letter to the editor: NVIDIA Jetson severely limited during object recognition, tracking

Now I know that Jetson TX2 does not have the discrete GPU that Tesla uses, and there's no 4k camera inputs and not quite 13 cameras and the software is somewhat different too, but still interesting.

I wonder what resolution the 7fps was achieved at and with how many cams.