AP2.0 Cameras: Capabilities and Limitations?

[SucreTease]

So it is not sensing rain, but abscense of sky? Luckily does not seem to start in the tunnels I drive.

If you were replying to my post, what you said is contrary to what I was hypothesizing. We know that Tesla is using a camera to detect rain. The effect of a rain drop is that light passing through it is altered by refraction, which alters the amount of light appearing in different directions, like lots of tiny lenses. This means bright spots (at the expense of adjacent darker spots). It is easy to surmise that dust can have a similar effect through its scattering of light by reflection. Scattering of light by dust is most obvious when the light scattered is bright and it is seen against a dark background, which is exactly the "tunnel" scenario presented by @J1mbo and others. I am hypothesizing that the "tunnel" scenario where dust scatters light resembles the pattern that the fisheye NN was trained to recognize.

Disclaimer: I am a physicist

 

[daktari]

If you were replying to my post, what you said is contrary to what I was hypothesizing. We know that Tesla is using a camera to detect rain. The effect of a rain drop is that light passing through it is altered by refraction, which alters the amount of light appearing in different directions, like lots of tiny lenses. This means bright spots (at the expense of adjacent darker spots). It is easy to surmise that dust can have a similar effect through its scattering of light by reflection. Scattering of light by dust is most obvious when the light scattered is bright and it is seen against a dark background, which is exactly the "tunnel" scenario presented by @J1mbo and others. I am hypothesizing that the "tunnel" scenario where dust scatters light resembles the pattern that the fisheye NN was trained to recognize.

Disclaimer: I am a physicist

I was not replying to you, but made a joke based on the two videos posted.
I think your theory is interesting, but maybe to complex. Could of course explain why it tok them a year.

 

[daktari]

I always believed they were gonna train it to recognize different patterns of rain and snow on a windshield. That is why the complete opposite, "absence of sky" is a bit corny.

 

[chillaban]

It's definitely not absence of sky that triggers it. I can drive around quite a bit in an underground parking garage without triggering it. It's sudden brightness changes or driving near textured wall surfaces that sometimes elicits a false wipe or two.

I would guess it is trained against stuff that clings against the windshield (whether it's water or corn flakes), or possibly the movement / buildup of such things on the windshield.

 

[TaoJones]

So..... if it will recognize, say, a bug or bugs splattered upon the windshield whilst meandering through, well, pretty much anywhere outside of SoCal (even the run to Kettleman City produced some bugs on the fascia), how nice it would be to trigger a spray or several of *wiper fluid* so as to have a hope of removing said bug carcasses before visibility is further reduced.

I already go through more wiper fluid manually in these cars than in any other, by far, so wouldn’t mind if it was deployed automagically as well.

And RainX is almost useless for bug mitigation when the bugs have any enthusiasm whatsoever.

Rain in general is great. Don’t get me wrong. But windshields become hard to see through frequently and often due to bugs. And harder to clean. Travel through Texas or the South is this is not immediately apparent.

So... as long as we’re autosensing...

What? A guy can dream.

 

[chillaban]

So..... if it will recognize, say, a bug or bugs splattered upon the windshield whilst meandering through, well, pretty much anywhere outside of SoCal (even the run to Kettleman City produced some bugs on the fascia), how nice it would be to trigger a spray or several of *wiper fluid* so as to have a hope of removing said bug carcasses before visibility is further reduced.

I already go through more wiper fluid manually in these cars than in any other, by far, so wouldn’t mind if it was deployed automagically as well.

And RainX is almost useless for bug mitigation when the bugs have any enthusiasm whatsoever.

Rain in general is great. Don’t get me wrong. But windshields become hard to see through frequently and often due to bugs. And harder to clean. Travel through Texas or the South is this is not immediately apparent.

So... as long as we’re autosensing...

What? A guy can dream.

I'm sure "autospray" is somewhere on their roadmap too.

 

[croman]

I'm sure "autospray" is somewhere on their roadmap too.

I tweeted Andrej Karpathy about this exact thing regarding bugs and salt spray and he just liked my tweet without responding to whether the cNN could be trained to recognize those conditions and spray in addition to wipe.

 

[BigD0g]

I tweeted Andrej Karpathy about this exact thing regarding bugs and salt spray and he just liked my tweet without responding to whether the cNN could be trained to recognize those conditions and spray in addition to wipe.

No reason why not, but the bugs would have to hit where the camera is, you'd have go through a swarm of locusts for that to be effective.

 

[TaoJones]

I'm sure "autospray" is somewhere on their roadmap too.

Well... I dunno... Playing devil's advocate here for a moment, for what will we need clean/clear windshields when the car will drive itself from coast to coast Johnny Cab-style?

Now, for the intervening years between the current reality and that future (sur)reality, I hope you're right. Looking back, bugs have been a much bigger problem than rain, given the poor nature of the wiper system in total to keep the top third of the windshield clean to start with.

If I lived somewhere with normal seasons and rain patterns, things might be different. Conversely, hereabouts at the moment they're expecting an inch of rain (to start with) and the evacuations have already begun.

But there are no bugs on my windshield unless and until I venture inland.

 

[croman]

No reason why not, but the bugs would have to hit where the camera is, you'd have go through a swarm of locusts for that to be effective.

I'm thinking salt spray first since it's a combo of water and contaminant.

 

[Big Earl]

Anyone care to speculate what it is about this scenario which is causing auto wipers to trigger? (Set to position 2)

I thought it might be the reflection of the sun from the road, but they haven't triggered in other, similar, situations. Also, you can see the temporary variation of speed that others have mentioned.

Do the wipers stop afte the gate opens or do they continue until you drive forward?

 

[scottf200]

Anyone care to speculate what it is about this scenario which is causing auto wipers to trigger? (Set to position 2)
<snip>
I thought it might be the reflection of the sun from the road, but they haven't triggered in other, similar, situations. Also, you can see the temporary variation of speed that others have mentioned.

Hopefully you did a "report" on it. Although I wonder how many of those autowiper reports they are getting. May your 'gate' one will cause them to take notice or email it in. They may be interesting in corner cases ... and maybe they can add it to the NN training.

"command_type" : "note",
"description" : "note",
"description" : "report",
"description" : "bug note",
"description" : "bug report",

 

[J1mbo]

Do the wipers stop afte the gate opens or do they continue until you drive forward?

No, they continue until the car has passed the gate.

 

[banned-66611]

LOL

 

[J1mbo]

LOL

Wanna share the joke with everyone else..?

 

[banned-66611]

Wanna share the joke with everyone else..?

The damn auto wipers, the tech even cheap cars have had for over a decade, which Tesla had working on AP1, which normally just requires a €5 sensor... Is just entering beta a year and a half later, barely works when you want it and sometimes works when you don't.

On a car costing the best part of 150k.

And owners are thrilled to beta test this feature for Tesla.

 

[agtdDelirium]

I'm sure "autospray" is somewhere on their roadmap too.

The last thing I need is cresting a hill with a low sun, already limiting my vision, and then the damn auto-wipers swipe and spray totally blinding me for several seconds. If it isn't raining, they should never swipe.

 

[J1mbo]

The damn auto wipers, the tech even cheap cars have had for over a decade, which Tesla had working on AP1, which normally just requires a €5 sensor... Is just entering beta a year and a half later, barely works when you want it and sometimes works when you don't.

On a car costing the best part of 150k.

And owners are thrilled to beta test this feature for Tesla.

Ho ho ho.

Couple of things:

* it's not a €5 sensor, IIRC @lunitiks posted the actual cost earlier in this thread, and it was more like $20
* AP1 solution is not perfect. Pre-AP solution is not perfect
* Tesla don't (yet) make a car that costs 150k for the base spec

 

[thomasmb]

* AP1 solution is not perfect. Pre-AP solution is not perfect

This! It was far from perfect on AP1 if you ask me, same goes for every other rain sensor implementation I've had in my cars – including german cars. Actually surprised this worked as well as it did for a first attempt. But mainly, I am just excited about the possibilities up ahead. Automatically using wiper-fluid to clean the window would be pretty sweet.

 

[chillaban]

Ho ho ho.

Couple of things:

* it's not a €5 sensor, IIRC @lunitiks posted the actual cost earlier in this thread, and it was more like $20
* AP1 solution is not perfect. Pre-AP solution is not perfect
* Tesla don't (yet) make a car that costs 150k for the base spec

Furthermore, $20 sensors do not cost $0 to integrate and test. And when they find issues they're pretty much SOL unless the supplier is willing to play ball (not at all likely for a small fish like Tesla working with German behemoths like HELLA)