A while back I was wondering why the same NN was being run on both the main and narrow cameras. Using one net on two cameras with different focal lengths is going to damage your detection scores, but then
@verygreen said at some point that he believed both main and narrow were being cropped to the same FOV to run through the network. So that fixes the detection score problem, mostly, but it brings up another question. If you're just doing object detection then doing it on two side by side cameras with the same FOV doesn't get you much beyond a bit of redundancy, so why?
@verygreen also suggested that he thought maybe they were doing stereo for the front cameras and I was pretty skeptical because I hadn't seen anyone doing it that way. And I've looked.
Well, looks like I could be wrong about that (again). I seem to be wrong alot, unlike some other folks.
So yesterday
@verygreen shared some output from a profiler that includes names of function calls from inside the vision process, and low an behold we can see "init_undistortmap_for_stereo" among the active functions. The most common methods of doing binocular stereo for computer vision requires that you raster align the left and right image, which in turn requires that you do some preprocessing to remove lens distortion and make both the right and left images completely 'flat'. Stereo processing then just measures the left right offset of pixels at various places in the image to create a depth map. (incidentally that LR offset is called disparity by the stereo vision folks) There's a bunch of information on that stuff here if you're interested:
The KITTI Vision Benchmark Suite
Basically, you have to "undistort" both the main and narrow before you run a raster-aligned algorithm on them to generate a stereo depth map. Which got me to wondering if main and narrow were far enough apart to generate any decent amount of binocular vision for a car application. They're pretty close together after all, and cars and stuff are pretty far so maybe you don't get much disparity from them. And once again the valiant
@verygreen comes to the rescue with a set of images that
@bjornb turned into this mosaic:
https://teslamotorsclub.com/tmc/attachments/merged_text-png.245225/
Which tells us two interesting things. One is that the fisheye is in the middle of the 3 forward facing cameras, so main and narrow have the largest separation, which is good if you want to use them for stereo vision. And the other thing is apparent when you look at the part of the mosaic where narrow is embedded in main. It's pretty obvious that the box around main and the box around narrow are not concentric, and that offset is the 'disparity' of the stuff near the boundary of the 'narrow' frame - which is mostly a couple of car lengths away. So yes, there is definitely plenty of disparity for things that are a few car lengths away, especially when you consider that these are high def images so there are a lot of raw pixels in there. So you can definitely get useful distance measurements using main and narrow as a pair of stereo cameras. And that undistort_for_stereo function implies that this is probably happening in AP2.
Though whether they use it to make driving decisions or it's one of those 'shadow mode' things is still unknown.
