AP2.0 Cameras: Capabilities and Limitations?

[lunitiks]

Can we discuss the lack of functioning rain-sensing wipers for a moment?

I think it's on topic because Tesla is obviously trying to accomplish this with their vision (camera) based HW only.

So @verygreen has hinted that there's some traces of SW-code on auto rain sensing wipers, but that's like - what, weeks (months?) ago - and the clock is ticking. I'm really beginning to worry that this seemingly piece of cake feature is a huge problem at Tesla.

So I ask you guys this:

Question 1: Is there any other production vehicle; BMW, Audi, Mercedes or others, that actually uses vision based rain sensing today? Or is this in fact a̶n̶ ̶e̶x̶p̶e̶r̶i̶m̶e̶n̶t̶ unprecedented production tech over at Tesla? I'd really like to know.

Question 2: What's most likely the biggest hurdle they're facing? I've gone through a couple of papers online and the most recent challenge discussed seems to be distinguishing unfocused raindrops. Look at this picture. See the difference between pic (a)/(b) and (c)/(d)? When focus is set on the windscreen, raindrops appear as objects well-separated from their background. Several papers discuss this and appear to have "solved" this. BUT, without a multi-focal camera setup, unfocused raindrops are not separated from the background --- the picture is merely blurred/smudged. How would the computer know it's rain and not some weird lighting condition, a damaged windscreen or some computer error? What's the tolerance for "blurryness" anyway?

This paper from 2011 discusses the above mentioned problem. The authors seem to be on track of something, but the conclusion is not impressive IMO: "The first results we achieved are quite promising since our algorithm was evaluated in highly dynamic scene. However, we believe that our algorithm may suffer from a lack of flexibility and thus need improvements in order to be fully reliable."

 

[AnxietyRanger]

Very good post @lunitiks. Thank you.

I have absolutely zero experise on this particular topic, but I've always considered the AP2 camera-based rain detection from two perspectives:

1) Neural net looking at camera video to determine rain. This I would assume they could get working, but would seem like an unnecessarily massive hog on CPU/GPU power...

2) Some sort of pattern change recognition perhaps looking at how light moves around on the glare shield seen on the fisheye front camera. This would seem like more the traditional rain sensor method where changes in light and rhythm are detected.

I find the latter more likely (though which camera it might use I do not know).

p.s. Can't think of any other car.

 

[AnxietyRanger]

I.e. I don't think seeing rain from a blurry snapshot is so much they way, as in seeing rain in the changes to an image over time... seeing the animation of rain...

 

[Saghost]

Can we discuss the lack of functioning rain-sensing wipers for a moment?

I think it's on topic because Tesla is obviously trying to accomplish this with their vision (camera) based HW only.

So @verygreen has hinted that there's some traces of SW-code on auto rain sensing wipers, but that's like - what, weeks (months?) ago - and the clock is ticking. I'm really beginning to worry that this seemingly piece of cake feature is a huge problem at Tesla.

So I ask you guys this:

Question 1: Is there any other production vehicle; BMW, Audi, Mercedes or others, that actually uses vision based rain sensing today? Or is this in fact a̶n̶ ̶e̶x̶p̶e̶r̶i̶m̶e̶n̶t̶ unprecedented production tech over at Tesla? I'd really like to know.

Question 2: What's most likely the biggest hurdle they're facing? I've gone through a couple of papers online and the most recent challenge discussed seems to be distinguishing unfocused raindrops. Look at this picture. See the difference between pic (a)/(b) and (c)/(d)? When focus is set on the windscreen, raindrops appear as objects well-separated from their background. Several papers discuss this and appear to have "solved" this. BUT, without a multi-focal camera setup, unfocused raindrops are not separated from the background --- the picture is merely blurred/smudged. How would the computer know it's rain and not some weird lighting condition, a damaged windscreen or some computer error? What's the tolerance for "blurryness" anyway?

View attachment 241527

This paper from 2011 discusses the above mentioned problem. The authors seem to be on track of something, but the conclusion is not impressive IMO: "The first results we achieved are quite promising since our algorithm was evaluated in highly dynamic scene. However, we believe that our algorithm may suffer from a lack of flexibility and thus need improvements in order to be fully reliable."

It's certainly a potential problem. If the raindrops aren't in focus, seeing them gets a lot harder. I don't know if you have any background in photography, but one of the critical factors that becomes important when the pictures become more challenging is "depth of field." Conceptually, that's the distance from the nearest object that is in focus with the current settings to the furthest object that's in focus. All other things being equal, smaller sensors make it larger, and so do shorter focal lengths (it's actually a function of the absolute aperture size, not the relative aperture we generally use.)

Remember back here, when we were trying to make sense of the images from the cameras? I believe they have your answer.

This is a shot from the front wide angle camera, and you can see that the dust and anti-reflective coating near the camera (bottom of the image) is blurred from being out of focus - but near the transition line the coating and its reflection at the bottom of the windshield portion are hard, sharp points. With a tiny sensor a a very broad field of view, it looks like the AP wide angle camera has everything from ~3" out to the horizon in focus at all times.

As long as the camera is looking at the bottom quarter or so of the part of the windshield it can see, it should be able to see rain drops as drops, not as a vague blur.

 

[SpotfireY]

I'll step out of the shadows and write my first post for this:
@Saghost raises a very good point. There might actually be an area of the wide angle picture where drops on the windshield might be (partially) in focus... but do we have visible evidence of this? Did @verygreen take any snapshots in rainy/wet conditions yet? I guess if he's up for it he could also grab his garden hose or something and simulate some rain for us

There might actually be other heuristics Tesla can use to determine rain or other bad weather conditions... Maybe something like a sudden, consistent drop in NN confidence from the blurrier picture?

 

[verygreen]

believe it or not every time there's a decent rain my car is in service so is not available.

I guess I just need to spray it from a hose as a decent simulation.

 

[AnxietyRanger]

This is a shot from the front wide angle camera, and you can see that the dust and anti-reflective coating near the camera (bottom of the image) is blurred from being out of focus - but near the transition line the coating and its reflection at the bottom of the windshield portion are hard, sharp points. With a tiny sensor a a very broad field of view, it looks like the AP wide angle camera has everything from ~3" out to the horizon in focus at all times.

While the effect you describe is real, I wonder how well rain actually shows. The stretched image I made back then of course blurs corners also due to stretching, so that needs to be noted as well.

However the main thing to note here is the heating wire running across the top of the picture:

It is never in focus, but you can see it (though to make @lunitiks point you can also see through it!). Rain is even more transparent (however distorting).

It would seem to me based on that, the rain is never in focus on the windshield. But I concede it could be more in focus on the lower edge of the fisheye due to the windshield curving further away from the camera.

Anyway, I do think recognizing rain by some algorithm looking at some of the cameras and the way light etc. moves should be possible.

Still, seems like having a traditional rain sensor would have eliminated one headache...

Minimum FSD capable suite... Minimum FSD capable suite...

 

[SpotfireY]

Its actually possible to make out all the heater wires, even if just very faintly.



I hope it's not just my eyes playing tricks on me, but it still seems like the bottom wire is the most in focus. It Just appears very faintly because of the focal length and the relatively small sensor resolution.

Still, seems like having a traditional rain sensor would have eliminated one headache...

You're definitely not wrong... Rain sensors are proven off the shelf components. It's kinda another case of KISS... or maybe Tesla hubris.

 

[AnxietyRanger]

You are right. I missed the bottom wire. Indeed as speculated due to different windshield distances it is more in focus... but still not properly in focus...

So maybe that bottom wire kind of tells how dirt specles would show in the bottom. It does not quite tell how transparent rain would...

@verygreen ... your turn.

 

[stopcrazypp]

As long as the camera is looking at the bottom quarter or so of the part of the windshield it can see, it should be able to see rain drops as drops, not as a vague blur.

And this approach appears to be exactly what this paper suggests doing (rather than the blurred raindrop detection over the entire frame that @lunitiks' link suggested):
"Hereby, the region at the bottom of the field of view (FOV) of the imager is used to detect raindrops, while the upper part of the image is still usable for other vision-based applications."
http://ieeexplore.ieee.org/document/5164291/?section=abstract

 

[AnxietyRanger]

And this approach appears to be exactly what this paper suggests doing (rather than the blurred raindrop detection over the entire frame that @lunitiks' link suggested):
"Hereby, the region at the bottom of the field of view (FOV) of the imager is used to detect raindrops, while the upper part of the image is still usable for other vision-based applications."
Vision-based rain sensing with an in-vehicle camera - IEEE Xplore Document

Sounds plausible.

Still, rather curious - rain detection was a "will" ship in December 2016 feature.

Here we are in August 2017, eight months later, and nothing released.

I wonder how unreliable the current development version is?

 

[stopcrazypp]

Sounds plausible.

Still, rather curious - rain detection was a "will" ship in December 2016 feature.

Here we are in August 2017, eight months later, and nothing released.

I wonder how unreliable the current development version is?

Who knows. Auto High Beams took a while to come out too and camera based high beams is not new technology either.

 

[lunitiks]

@stopcrazypp that's interesting but seems to be a really old paper... Too bad it's just an abstract. Do you have a link to the actual doucment?

Anyway, I dug some more. At first glance, this patent application by Delphi Technologies Inc. -- published in March 2017 -- seemed very promising. (Let's pray it's not legal issues holding back rain-sensing from our cars ) The application describes an "Integrated camera, ambient light detection, and rain sensor assembly." The general idea being a fisheye lens with an RCCC filter for the central-portion of the FOV, and an RGBN filter for the periphery. The latter serving as rain and ambient-light-intensity detection. (Please read the document in its entirety for details.)

However the application goes on to describe an IR-emission (LED) device to illuminate instances of water:

If the near-infrared sub-pixel in the periphery-portion of the imager-device detects near-infrared light that corresponds to what is being emitted by the light-source, then that may be an indication that the water is present on the windshield. Preferably, the light-source is part of the assembly

So essentially it would rely on light emission the same way as traditional rain sensors (except here the receiving end is a 'multi purpose' video camera).

Side note: Delphi's patent application actually highlights one of the challenges I mentioned earlier: "Enduring obstructions" on the windshield that is not to be interpreted as rain. For example dried bugs splattered on the windshield, dirt, tar and/or damage (stone chips or cracks in the glass). Of course the algos must be set up to classify such obstructions. This could be done by letting the computer know when the windshield-wipers are active, giving them a little time to try to remove the object. If it won't go away, it's usually not rain.

 

[stopcrazypp]

@stopcrazypp that's interesting but seems to be a really old paper... Too bad it's just an abstract. Do you have a link to the actual doucment?

Anyway, I dug some more. At first glance, this patent application by Delphi Technologies Inc. -- published in March 2017 -- seemed very promising. (Let's pray it's not legal issues holding back rain-sensing from our cars ) The application describes an "Integrated camera, ambient light detection, and rain sensor assembly." The general idea being a fisheye lens with an RCCC filter for the central-portion of the FOV, and an RGBN filter for the periphery. The latter serving as rain and ambient-light-intensity detection. (Please read the document in its entirety for details.)

However the application goes on to describe an IR-emission (LED) device to illuminate instances of water:


So essentially it would rely on light emission the same way as traditional rain sensors (except here the receiving end is a 'multi purpose' video camera).

Side note: Delphi's patent application actually highlights one of the challenges I mentioned earlier: "Enduring obstructions" on the windshield that is not to be interpreted as rain. For example dried bugs splattered on the windshield, dirt, tar and/or damage (stone chips or cracks in the glass). Of course the algos must be set up to classify such obstructions. This could be done by letting the computer know when the windshield-wipers are active, giving them a little time to try to remove the object. If it won't go away, it's usually not rain.

Sorry no link, it appears to be behind a paywall, and I'm not paying just to look at it.

I believe there have previously been discussion about possible legal issues with some of the camera based technology (for example auto high beams). Certainly a possibility.

 

[lunitiks]

What if rain sensing won't be done with vision (cameras) at all?

I mean, are we absolutely positive that the "Light/Temperature/Humidity" sensor isn't in fact a rain sensor? To those not familiar with it, I'm talking about this sucker:

It's been discussed before, and the prevailing theory (@bjornb) is that it's not used for rain sensing, but instead for measuring temperature and humidity inside the tri-cam block, or mist/humidity that should engage the heating threads. Plus ambient light detection for MCU/IC screen brightness (hence the outward facing "lens" pictured above).

But I'm not entirely convinced by that theory...

Main reason is of course that vision-based rain sensing seems extremely iffy to me (ref. discussion above).

Now I went through Teslas wiring diagrams and I noticed something else: The sensor (also referred to as "Tesla HVAC Sensor" or "THS") has an amp_9-1718346-1 connector with three wires: Power, ground and LIN. On Model S, the LIN-wire is spliced to a LIN-wire coming from "MSR Headlamp RH", and then goes straight to the Body Control Module (BCM). There's no other signal connections.

Now the interesting bit is that the sensor is referenced on only two pages in the wiring diagram (Model S). The first page is titled "Ext Illum Front 1/2: Lights". On this page, the BCM (which the LTH-sensor is connected to) only has references to headlamp and exterior lighting action ("Corner", "Fog", "Low Beam", "Turn" etc.).

The other page is titled "Wiper, Washer and HVAC Sensor". On this page, the BCM only has references to the wipers ("Wiper Slow", "Wiper Fast", "Wiper Park Pos. Sense", "Wiper Unpark", "Wiper De-Ice"), plus washer pump. If the HVAC-sensor has nothing to do with the windshield wipers, then why is it referenced on the only page in the wiring diagrams that deals with exactly this?

In the wiring diagram for Model X, the HVAC-sensor is exclusively treated on the "Wiper, Washer and HVAC Sensor" page. So no splicing with headlamp-LIN. Simply HVAC-sensor -> BCM -> Wiper functions. That's it.

Furthermore, LIN-wire from HVAC-sensor is designated "WPR" (and then a number) on both S and X diagrams.

Where there's smoke...

 

[AnxietyRanger]

@lunitiks Couldn't being on the Wiper, Washer and HVAC page title be merely a historic relic from the AP1 days - I mean, in this case the sensor relating to HVAC instead of Wiper or Washer - and the page having a rain sensor for wipers earlier during the AP1 days?

HVAC systems commonly use humidity sensors mounted around the rear-view mirror (Audi for the longest time had one on the base of the rear-view mirror for example). They are separate from rain sensors, they are inteded for interior use, not rain sensing (for this purpose HVAC systems also care about light coming down on the car and temperature measurements). Also the external pointing part, clearly the light sensor, is quite different from what rain sensors usually look alike...

I'm not saying your theory is impossible, certainly it is a good theory. But if there was already a sensor for this purpose, why take so long to implement support for it...?

 

[lunitiks]

Couldn't being on the Wiper, Washer and HVAC page title be merely a historic relic from the AP1 days

Thanks @AnxietyRanger for making me look, because now I discovered that the previous wiring diagram (AP1 facelift) had the same page but it was titled "Wiper / Washer and Rain Sensor" So the word "Rain Sensor" has been replaced by "HVAC sensor" in the page title. Also, in the old wiring diagram the "Rain Light Sensor" had 12V, GND and LIN -> BCM -> wiper action, just like the new HVAC sensor

So to me, it looks like the HVAC sensor has replaced the old rain sensor wrt controlling wiper action

HVAC systems commonly use humidity sensors mounted around the rear-view mirror

Sure but I think the old HELLA unit could be measuring in-car humidity as well. Hence the holes in the housing/enclosure - they would provide for some air flow

But if there was already a sensor for this purpose, why take so long to implement support for it...?

That's the big question, isn't it? All I know it that Tesla abandoned the off-the-self HELLA part and went with some custom pcb - probably in house stuff

 

[BigD0g]

I gotta be amazed at all the attention /posts the rain sensing wipes gets..... Folk's really want their rain sensing wipers!

 

[AnxietyRanger]

Interesting research @lunitiks, thanks!

Also, in the old wiring diagram the "Rain Light Sensor" had 12V, GND and LIN -> BCM -> wiper action, just like the new HVAC sensor

Are you saying the new HVAC sensor is also wired to wiper action in the manual?

In your previous post you said it was wired to BCM, but is there some diagram showing a connection to wiper action through there?

(Other than theoretically both being connected to the same BCM of course...)

now I discovered that the previous wiring diagram (AP1 facelift) had the same page but it was titled "Wiper / Washer and Rain Sensor" So the word "Rain Sensor" has been replaced by "HVAC sensor" in the page title

Relabelling Rain Sensor to HVAC Sensor suggests a reason for doing so. Removing rain sensing from the sensor would be one possible reason...

Your theory is still possible, just trying to confirm stuff.

 

[AnxietyRanger]

I gotta be amazed at all the attention /posts the rain sensing wipes gets..... Folk's really want their rain sensing wipers!

I think that's false psychology, though.

Some of us are genuinely interested in the technical details, merits of policies and whatnot. It is a hobby. I know it seems silly for those not so inclined, but rarely is it because e.g. we "want their rain sensing wipers".