FSD 12.4, Vision Attention Monitoring, and IR LEDs

[SourJellybeans]

NotATeslaApp says that 12.4 will rely on the cabin camera for attention monitoring instead of the steering wheel IF several conditions are met:

• the camera is not occluded
• there is sufficient lighting
• the driver is looking forward
• the driver is not wearing sunglasses
• the driver is not wearing a low-brim hat or another object that covers their eyes

Tesla Releases FSD v12.4: New Vision Attention Monitoring, Improved Strike System With Update 2024.9.5

Tesla has released FSD v12.4 to employees with a new vision-based cabin monitoring system.

www.notateslaapp.com

Night driving in remote areas (no oncoming headlights or street lights to illuminate you) will mean the cabin camera can't see your face, so I was wondering if a plug-and-play USB-powered infrared LED somewhere in the cabin could fix that without any actual changes to the car. I had an old security camera with a few IR LEDs so I plugged it in and tested in the garage with no lights... unfortunately, while the cabin camera can see the IR a bit, it's still filtered in some way, because it's not picking up much of it. I had to hold the camera right next to my face for it to pick it up at all, yet the cheap security camera itself sees the cabin as if it's entirely lit up.

I'm under the impression the IR filters are over the lenses of the cameras, so potentially one could retrofit a lens without the IR filter (obviously voiding or outside of warranty), but would be nice 1st party retrofit from Tesla.

 

[gtg465x]

Well, I just discovered that my 2023 Model 3 RWD doesn’t have an IR camera like I thought it did. I guess Tesla only started putting them in Model Y in 2022… not sure if / when they started with Model 3.

 

[JulienW]

.....I had an old security camera with a few IR LEDs so I plugged it in and tested in the garage with no lights... unfortunately, while the cabin camera can see the IR a bit, it's still filtered in some way, because it's not picking up much of it. I had to hold the camera right next to my face for it to pick it up at all, yet the cheap security camera itself sees the cabin as if it's entirely lit up......

And how on earth did you determine if the camera is seeing IR? You do understand that IR is beyond/longer wave length than human vision is capable of seeing. There is no way a person can detect IR without using equipment that shifts IR to a shorter wave length (Red) that falls within human vision. Most cell phone cameras do this when they are exposed to near IR. You can see example of this in pic below where you can see the right IR transmitter shifted to Red. However IR is invisible to the human eye.

Also a quick 101 class in what Tesla is trying to do and what other OEMs do with a detected eye tracking system. A detected eye tracker has IR transmitters aimed directly at your eyes. IR goes through the pupil and reflects of the cornea. this reflection I picked upon by a camera that is narrow field of view and directly in front of the eyes. Pic of the Ford system with IR transmitters (Red Arrow) and Eye tracking camera (Green Arrow). These systems can track your eyes with near 100% accuracy even through the darkest sunglasses.

Tesla has an in cabin camera that is off axis to the driver's eyes since it looks at the entire cabin. Tesla is trying to ad hoc repurpose the camera and added IR transmitters to the cabins. Since it doesn't have line of sight to the pupils, Tesla is trying to use AI to do a combination of visual (visible light) facial cues (main way of taking) combined with the limited IR pupil view. This is why you have limitations and it can be hit or miss.

 

[SourJellybeans]

And how on earth did you determine if the camera is seeing IR? You do understand that IR is beyond/longer wave length than human vision is capable of seeing. There is no way a person can detect IR without using equipment that shifts IR to a shorter wave length (Red) that falls within human vision.

I guess you didn't look at the attachment before putting all that effort into your post. I brought up the Cabin Camera preview in the car and tested it with regular garage lights on, with the garage lights off (and thus nothing is visible), and then plugged in my separate camera with activated IR LEDs and tested pointing it at the Cabin Camera and pointing it at my face to demonstrate how little Infrared the Cabin Camera picked up.

But please continue to educate me more on how Infrared works.

 

[JulienW]

I guess you didn't look at the attachment before putting all that effort into your post. I brought up the Cabin Camera preview in the car and tested it with regular garage lights on, with the garage lights off (and thus nothing is visible), and then plugged in my separate camera with activated IR LEDs and tested pointing it at the Cabin Camera and pointing it at my face to demonstrate how little Infrared the Cabin Camera picked up.

But please continue to educate me more on how Infrared works.

OK so how did you measure the IR signal that the cabin camera received? Did you tap into the camera housing wiring to take the measurement?

Just to add all you show is the camera Preview. That is optical/visible light and offers NO insight to (invisible) IR reception, not to mention that the MCU would not process IR and the LED display could not display IR. LCD displays have a limited color space of Rec. 709 which is even way less than human vision (about 610nm wave length). And even if they could process and display IR you would need a way to detect the IR and convert to visible light.

Here is what the human eye is capable of seeing and the triangle is about the color space that a quality LCD can reproduce. IR has a wave length of 700nm to 1000nm. Even the best eyes can't see longer waves than about 650nm.

 

[JB47394]

OK so how did you measure the IR signal that the cabin camera received? Did you tap into the camera housing wiring to take the measurement?

Here's a prior discussion about infrared lighting as viewed by the cabin camera. The camera feed can be seen through the app. The post includes a clear shot of the cabin entirely as a result of infrared lighting in the car.

I got my new Autopilot defeat device

funny, but I just drove 2 hours with AP running 30.5.1 and hanging my arm/wrist through the steering wheel hole no nags and was not staring straight on the road think they are solely relaying on sensing the arm weight on the steering wheel

teslamotorsclub.com

 

[JulienW]

Here's a prior discussion about infrared lighting as viewed by the cabin camera. The camera feed can be seen through the app. The post includes a clear shot of the cabin entirely as a result of infrared lighting in the car.

There is a simulation of what the camera might see using a FLIR (probably placed in front of the camera with the IR transmitters on) but we have no way of knowing with 100% accuracy what the camera sees in IR and there is no what to see that. All we can see is the cabin camera in visible light through the App or Service menu. The Service menu preview is FAR superior to the App view but wasn't yet available at the time of that thread.

But again it is not really relevant since it is NOT a dedicated IR eye tracking system like other manufactures use and has extreme limitations. I outlined this above.

So in effect to use the windshield wiper analogy Tesla is trying to repurpose a normal optical camera to do a different job using ML/AI just like instead of a Rain Sensor Tesla repurposes the cameras to watch for rain.

 

[APotatoGod]

OK so how did you measure the IR signal that the cabin camera received? Did you tap into the camera housing wiring to take the measurement?

Just to add all you show is the camera Preview. That is optical/visible light and offers NO insight to (invisible) IR reception, not to mention that the MCU would not process IR and the LED display could not display IR. LCD displays have a limited color space of Rec. 709 which is even way less than human vision (about 610nm wave length). And even if they could process and display IR you would need a way to detect the IR and convert to visible light.

Here is what the human eye is capable of seeing and the triangle is about the color space that a quality LCD can reproduce. IR has a wave length of 700nm to 1000nm. Even the best eyes can't see longer waves than about 650nm.

It sounds to me like you have a fundamental misunderstanding of how cameras (and our eyes, for that matter) work. They don't capture every possible wavelength within the visible spectrum (or beyond it), but instead only capture the intensity of 3 specific colors, Red, Green, and Blue. These sensors respond to frequencies at or near the wavelength they're designed for. Most cameras have an infrared filter installed, as without it the red sensor detects infrared light that the human eye can't see. Without this filter, you tend to get inaccurate color reproduction, with more "red" picked up than there really is. That's what this thread is about, trying to intentionally provide more infrared light so it's not disruptive to the driver, but is helpful for the camera.

Looking at a view recorded from the camera wouldn't result in it emitting infrared pixels of light, but instead just more red light, which is visible to us humans.

 

[MidBaz]

Wow . This is a proper 'geek-off'. Once you've come to an agreement and established whether we will/will not retain nag mode - give us a shout.

TVM

 

[KerrySkates]

I don’t see how Tesla can even get away with going to the camera-only detection with or without infrared. NHTSA still isn’t satisfied with the current nagging. If infrared solves the problem Tesla would either need to install new interior cameras in models without support (for free) or find a better solution to detect driver awareness altogether with the existing hardware.

 

[kpanda17]

btw, we have the main FSD thread... just saying

 

[EatsShoots]

This is actually an exciting development if it works and ultimately can work with sunglasses.

Of the several brands of vehicle I've been in that require "touching the wheel" I've been very unsatisfied with 100% of them (toyota, volvo, tesla). It's a terrible approach to the problem.

 

[CRIMSON_DATE]

I think this is going to be a mess. I was already locked out of FSD 12.3 for a few drives already because I was squinting in the morining becase of the sun and it thought I was not watching the road. I believe 12.4 will also put you in FSD jail for a week if you have too many of these lockouts