Sounds suspect, I suspect it's all grayscale as that's what the NN is trained on, I believe the service portal is shut out in terms of M3 parts and such so I don't even think @lunitiks can answer this one for us. I suspect in the next few months we'll know all, because the flood gates will open and geeks will get there hands on the M3 and start tearing it apart.
-
Want to remove ads? Register an account and login to see fewer ads, and become a Supporting Member to remove almost all ads.
HW2.5 capabilities
- Thread starter verygreen
- Start date
-
- Tags
- Autonomous Vehicles
Sounds suspect, I suspect it's all grayscale as that's what the NN is trained on, I believe the service portal is shut out in terms of M3 parts and such so I don't even think @lunitiks can answer this one for us. I suspect in the next few months we'll know all, because the flood gates will open and geeks will get there hands on the M3 and start tearing it apart.
For 360 degree view on the new Model 3/Intel central screen?
AP2 rear camera appeared on AP1 cars beforehand. As did AP1 stalk arrangement on pre-AP cars... so a new part could signal an upcoming larger change.
@jimmy_d would the NN require modification if the inputs vary across the platform? In other words, does it make sense to have some grayscale and some color cameras?
I just think they could have color cameras but chuck out the color information. That seems easiest to maintain continuity.
Anyway this is what Tesla service told this user (wrong camera type, and they specified that it is black and white, should be colour).
Of course it is possible, that the colour vs. black and white was service personel’s own interpretation.
Of course it is possible, that the colour vs. black and white was service personel’s own interpretation.
I think that the advantage of grayscale is that with no color filter on most pixels it lets in more light so you can have better low light performance and/or higher frame rates (less motion blur).
(Edited for typo)
jimmy_d
Deep Learning Dork
The NNs that I've seen have two input channels, which I interpreted as matching the red+gray camera specification that someone else posted here a while back. If you were to switch to a color camera you could preprocess it into red/grey and continue using the original network. That would be really easy. You might see some variation in behavior but generally it should work ok. (You wouldn't throw away the other color channels, you'd combine all 3 to make a grey channel and then feed that to the NN along with just the red as a second channel).
As background: The optics of a red/grey camera give you a red channel with nominal dynamic range and a grey channel with higher dynamic range (the grey channel is generated from multiple combined filter channels to get higher SNR at the same camera resolution, but it comes at the cost of the other colors). So you get say a 10 bit red channel with a 20 bit grey channel instead of 10 bits each for RGB. The extra dynamic range for the grey can be used to see detail in shadow during the day or to see details at night while still resolving the shape and location of light sources.
Of course, when you give up the full color range you lose some other stuff that might be useful too. For one thing the images from those cameras look strange to people - they are sort of but not really color and end up looking kind of like a picture from a poor quality color camera. If you cared about being able to use those cameras to generate images that people would look at then that might matter. An ideal camera would have plenty of dynamic range and also give you full color but the automotive grade cameras that I've seen generally require you to choose one or the other.
It's an interesting question whether, to an NN, an enhanced dynamic range red/grey is better than full color. The tradeoffs aren't totally clear but simplistic analysis suggests the enhanced dynamic range of the red/grey is probably really useful for car applications, which may be why it seems to be what people are using in cars (aside from for backup cameras). But I wouldn't be surprised if full color turned out to be better on further analysis. There are an awful lot of tricks that an NN can do, and NNs are so new that it's unlikely that a definitive answer is known today. It's also possible that there's a new camera that provides the desired dynamic range without sacrificing full color capability. Cameras and sensors get better every year. Eventually you won't have to choose.
The 'perfect' camera for an automotive NN is probably not going to use the same RGB channels that the human eye employs. It would probably have more color channels, it might have polarized filters, and would probably employ at least one near IR channel to allow better application of IR illuminators at night. But high volume camera sensors today are mainly produced for human eye compatible color representation or a simple variation of that (which is what the red/grey cameras are). You can expect that self driving car cameras will change as the application becomes important enough to drive it's own camera industry.
But if you "reprocess" it by just picking a random other color then it does not matter if the camera is BW or not. This whole "color" camera is quite a bit suspect and it could be SC was misinformed or misunderstood (color camera = different color code as an example).
jimmy_d
Deep Learning Dork
I suspect you're right about the purported camera change being not quite the same as what we are hearing.
And yes - picking a different color to substitute for grey could cause problems. An appropriately weighted combination of the RGB components can provide a good approximation to grey, but it won't be exactly the same. However it is pretty easy to implement, the computational overhead is trivial, and it wouldn't be hard to validate the substitution if Tesla had some reason to need to make the camera change without a commensurate change in the network. My guess, and it's only a guess, is that it's likely to allow you to continue using a network that was trained on red/grey, with some small performance degradation.
So the supposed extra overhead from HW2.5 processing would be used to normalize HW2.5's cameras to the 2.0 cameras? That seems short sighted since 2.0 cars are limited and will only become even more marginalized over time.
I am of the camp that this is erroneous information, as usual, supplied by Tesla employees given very limited information by an organization that prides itself on feeding information to customers like North Koreans (both actual food and information).
btw the other thing is: at least on my car on hw2.0 only main and narrow cams are participating in any sort of calibration. I somehow doubt hw2.5 made a huge leap there (but it's relatively easy to verify on a rooted hw2.5 car, do we have any of those yet?)
Regarding winter driving, I was testing my AP2 (2017.46.8) today on a highway in snowing weather - first time really with this car in this weather - just a pretty short trip and it seemed to be a bit less sure than usual, but OK. Until after maybe 3-4 minutes of driving the two red hands appear on the screen with an audio signal and both TACC/AP turn off.
My first thought is "did I miss hands on wheel nags" (both hands were on wheel but sometimes it misses that), but no, it reported Front Radar Visibility Reduced and kept TACC/AP off for the rest of the trip.
The front of the car looked pretty much like Bjorn's when I got home, less bad, but similarly covered... The lack of a heated front really seeming to "pay dividend"... The weather was nothing special, this really does not bode well for winter use? I've used adaptive cruise since 2006 or so and usually only one or two issues every winter... never on the first day of snow driving...
Bjørn Nyland on Twitter
My first thought is "did I miss hands on wheel nags" (both hands were on wheel but sometimes it misses that), but no, it reported Front Radar Visibility Reduced and kept TACC/AP off for the rest of the trip.
The front of the car looked pretty much like Bjorn's when I got home, less bad, but similarly covered... The lack of a heated front really seeming to "pay dividend"... The weather was nothing special, this really does not bode well for winter use? I've used adaptive cruise since 2006 or so and usually only one or two issues every winter... never on the first day of snow driving...
Bjørn Nyland on Twitter
lunitiks
Cool James & Black Teacher
The cameras in 2.0 are purpose-built for this application,
You can clear that unhappy snowface by opening the trunk from the MCU
You can clear that unhappy snowface by opening the trunk from the MCU
Similar threads
- Article
