AP2.0 Cameras: Capabilities and Limitations?

[JeffK]

Could the red channel help them get sky combined with others... and red when used alone or something like that?

The general purpose is to use the red channel for red(ish) objects: stop signs, red lights, brake lights, human skin has a high red response (all nationalities). That plus the HDR monochrome information is fed straight into the computer vision algorithms and/or NNs. They don't need to identify sky as much as signs, signals, cars, pedestrians, and other obstacles within the drivable path.

Although, if you discard the red data sometimes the sky will be totally blown out, it's easy to see how the broadside of a brightly lit white truck might blend into it.

 

[AnxietyRanger]

The general purpose is to use the red channel for red(ish) objects: stop signs, red lights, brake lights, human skin has a high red response (all nationalities). That plus the HDR monochrome information is fed straight into the computer vision algorithms and/or NNs. They don't need to identify sky as much as signs, signals, cars, pedestrians, and other obstacles within the drivable path.

Although, if you discard the red data sometimes the sky will be totally blown out, it's easy to see how the broadside of a brightly lit white truck might blend into it.

Yeah, I always thought it was for red objects too. Seeing what it did to the sky in the first image where ground was not visible, though, kind of made me think. But I think you are right. The blue effect happens on the ground too, so it is not so useful for sky recognition...

 

[verygreen]

Yeah, I always thought it was for red objects too. Seeing what it did to the sky in the first image where ground was not visible, though, kind of made me think. But I think you are right. The blue effect happens on the ground too, so it is not so useful for sky recognition...

The thing with the sky is the sky is not red, this makes the red channel kind of underexposed compared to the clear pixel.

If you paste red data into the red channel, the rest of colors might turn correct or not depending on what the real colors were before.
E.g. I know brown (think shipping boxes) turns into brown because it's kind of darkish in normal sense, but is probably brighter on the red channel.
The sky is not red to any degree so the red channel is supressed compared to green and blue, and green + blue + supressed red gives you kind of this light blueish sky. You an probably just fill the red channel with "black" (or super dark gray) and you'll see a similar effect I suspect.

 

[AnxietyRanger]

@verygreen Thanks! I'm not a complete noob on computer color management or image manipulation, but still clearly out of my depth on this topic. Very educational. Thanks, guys.

 

[lunitiks]

Also if you have the unfused (read debug?) hardware - it writes clips to sata (read any usb) device plugged in. Vision task certainly checks if this dir exists and if it is, I guess writes there.

# if board is not fused, check if a SSD drive is plugged in, automount to
# "/tmp/external_drive/" and start writing clips/RTDVs to it
if detect-ecu-unfused; then
echo "Boot $bootcount: Settling udev $(/sbin/uptime-seconds) s after boot"
# wait for udev to finish so that drive will be mounted if it is present
# finish early if the external drive is already mounted
udevadm settle --exit-if-exists=/dev/sda1 --timeout=1
echo "Boot $bootcount: udev settled $(/sbin/uptime-seconds) s after boot"
# Check if a SSD drive is present and "/tmp/external_drive" is not already mounted
if [ ! -d /tmp/external_drive ] && [ -b "/dev/sda1" ]; then
# Mount the drive and set permissions
mkdir /tmp/external_drive
mount -t ext4 -o data=writeback,barrier=0 /dev/sda1 /tmp/external_drive
if [ $? -ne 0 ]; then
# Can't mount the drive, remove folder
rmdir /tmp/external_drive
echo "Boot $bootcount: External drive detected but not mounted. $(/sbin/uptime-seconds) s after boot"
else
chmod 774 /tmp/external_drive
chown root:autopilot /tmp/external_drive
# Vision will dump logs into /tmp/external_drive/clips
ensure_directory "/tmp/external_drive/clips" "1774" "root:log"
# FIXME - RTDV logs will all go into the same folder across multiple drives.
# only way to differentiate drives will be the time stamps in the
# log files themselves
ensure_directory "/tmp/external_drive/dvlog" "0770" "root:autopilot"
ensure_directory "/tmp/external_drive/dvlog/staging" "0770" "root:autopilot"
# Bind the usb-storage kernel thread and IRQ to the Vision CPUs
for pid in $(pgrep usb-storage); do
taskset -p 6 ${pid}
done
for pid in $(pgrep xhci); do
taskset -p 6 ${pid}
done
echo "Boot $bootcount: External drive mounted $(/sbin/uptime-seconds) s after boot"
fi
fi
fi​

Interesting. Maybe the USB-port you're referring to, is this little bastard? (Passenger/RH side of the ECU.)

04

• lunitiks
• Feb 11, 2017

Maybe your friend is willing to give it a try?

 

[verygreen]

Interesting. Maybe the USB-port you're referring to, is this little bastard? (Passenger/RH side of the ECU.)

04

• lunitiks
• Feb 11, 2017

Maybe your friend is willing to give it a try?

Yes, I think that must be it. Though trim needs to be removed along with some other stuff to get access there, I guess, might be kind of complicated.
It looks like a missing sdcard slot right next to it too.

 

[AnxietyRanger]

To continue on the #101 and #117 on the front fisheye camera, I edited the image a little to get a bit better view of what kind of quality AP2 sees in the wide sense forwards (B pillar cameras pick up from maybe a fifth before the edges of this image). This an RGB greyscale image with red channel included, warped a bit and added brightness/contrast and cropped out the glare shielding.

 

[verygreen]

To continue on the #101 and #117 on the front fisheye camera, I edited the image a little to get a bit better view of what kind of quality AP2 sees in the wide sense forwards (B pillar cameras pick up from maybe a fifth before the edges of this image). This an RGB greyscale image with red channel included, warped a bit and added brightness/contrast and cropped out the glare shielding.

I'd say this is what it might be potentially seeing once wide camera is used.
Right now only Main and Narrow cameras are in any sort of use.

 

[lunitiks]

Fantastic job @AnxietyRanger

 

[lunitiks]

Yes, I think that must be it. Though trim needs to be removed along with some other stuff to get access there, I guess, might be kind of complicated.

I've done it, it's really just a handful of panels -- all mounted with clips. The only exception is a few torx screws and you're done. What needs to be disassembled is the glove box. This video shows the entire procedure in under 4 mins.

From 21.40 to 25.05

 

[AnxietyRanger]

I'd say this is what it might be potentially seeing once wide camera is used.
Right now only Main and Narrow cameras are in any sort of use.

Of course.

 

[bjornb]

@verygreen : Do you perhaps have some more pictures with other scenes? Especially the fisheye camera?
(like the lake and trees image you provided for main and narrow earlier, something from the road or just a parking lot with some cars parked).

In the image provided it can look like the bottom part of the fisheye (marked in red below) is a bit unclear/fuzzy.

It would be interesting to see it in other conditions.

 

[AnxietyRanger]

@verygreen : Do you perhaps have some more pictures with other scenes? Especially the fisheye camera?
(like the lake and trees image you provided for main and narrow earlier, something from the road or just a parking lot with some cars parked).

In the image provided it can look like the bottom part of the fisheye (marked in red below) is a bit unclear/fuzzy.
View attachment 226223
It would be interesting to see it in other conditions.

That fuzziness concerns me a little too. It sees the fisheye is a bit tricky to implement that far behind glass. I guess that is why MobilEye in their patent/prototype (that Tesla also tested) placed the front fisheye much closer to the class so that it wouldn't focus on the fuzz and there'd be less glare shield in the way...

There are more images in previous pages of this thread e.g.: --- Edit: I see you meant other scenes entirely, sorry. Would love those too!

All 7 raw files (plus mono/red separated raws) here:

Box | Simple Online Collaboration: Online File Storage, FTP Replacement, Team Workspaces

 

[verygreen]

@verygreen : Do you perhaps have some more pictures with other scenes? Especially the fisheye camera?
(like the lake and trees image you provided for main and narrow earlier, something from the road or just a parking lot with some cars parked).

In the image provided it can look like the bottom part of the fisheye (marked in red below) is a bit unclear/fuzzy.
View attachment 226223
It would be interesting to see it in other conditions.

Well, the kind of lighting situation made the glass rereflections bad with all the imperfections it accumulated I imagine.

I'll see I can obtain a different set of images. What sort of a different scene would make sense, though?

 

[lunitiks]

There are more images in previous pages of this thread e.g.: --- Edit: I see you meant other scenes entirely, sorry

Yes I think @bjornb meant fisheye stills in other environments (scenes). We've only seen one yet, haven't we?

EDIT: More fisheye in sunny conditions maybe to check if that mall picture is an extreme or normal case

 

[AnxietyRanger]

Yes I think @bjornb meant fisheye stills in other environments (scenes). We've only seen one yet, haven't we?

EDIT: More fisheye in sunny conditions maybe to check if that mall picture is an extreme or normal case

Realized that too, edited to add. I would love more scenes as well!

 

[lunitiks]

It's kind of interesting also, that the glare shield material is different in AP2 and AP1 -- the latter being rubber with grooves (ref my avatar). I wonder if that was a good or bad idea

 

[bjornb]

Well, the kind of lighting situation made the glass rereflections bad with all the imperfections it accumulated I imagine.

Well, they can at least use it to see raindrops and implement rain sensing wipers..

I'll see I can obtain a different set of images. What sort of a different scene would make sense, though?

Thanks. If I can choose I would like to see something that gives us an impression on how helpful it will be in city driving, both under ideal conditions and bad conditions (bad lighting/sun in your face ).

 

[AnxietyRanger]

Well, they can at least use it to see raindrops and implement rain sensing wipers..

That occurred to me too. I was even thinking if the anti-glare material was selected in such a manner that it might be used as a reference area for rain detection. If that solid pattern changes due to rain reflections/shadows... it might make for the largest rain detection sensor in cars... sort of.

 

[lunitiks]

I've never seen a camera based rain sensor before, and I'm not sure if this is how Tesla is going to solve it in AP2... How would that even work? NN rain drop training? Are we sure that there's not an LED and LRD separate from the cameras, facing the windshield?

Rain sensor - Wikipedia
Rain Sensors