Tesla replacing ultrasonic sensors with Tesla Vision

[MR F]

Anyone taken delivery of a vehicle without the USS?

 

[orion2001]

Judging by their success so far (e.g. auto wipers), It will take a while until they solve it so that it is comparable to USS.
Even if they solve it, there is a sw maintenance cost that increases exponentially with complexity. I thing we could agree that measuring distance with USS is way less complex than measuring using TV.
Then, there is the opportunity cost. From an average customer perspective (who does not really care about USS vs TV until that becomes an issue) there are higher priority problems that need solving - auto wipers, auto high beams, phantom braking, etc. It is clear that Tesla is more interested in cutting cost than serving the customer.

People also seem to have forgotten that "Vision-only" is yet to achieve parity with the Vision + Radar feature set (90mph limit and 1-"car" follow distance setting). "Coming soon" . And that was technically an easier solution seeing as their cameras are much better positioned for regular AP/FSD sensing than for what the USS are positioned to do. I wouldn't hold my breath on Tesla Vision achieving parity with USS any time soon.

 

[stopcrazypp]

Technically you could detect individual photons with a photomultiplier tube, which can cost hundreds or thousands of dollars depending on what you're looking for... But yeah, they're not using those in Teslas, either. Bundling enough of them together to make even a low-resolution "camera" would probably bring the cost and size to at least the scales you mentioned. And it would be practically useless for the application anyway. Basically just a fancy saturated photodiode at normal light levels.

The photon count bit is only related to 10-bit RAW images which give significantly higher dynamic range than the 8-bit processed images they were using previously.

"Upgraded generalized static object network to use 10-bit photon count streams rather than 8-bit ISP tonemapped images by adding 10-bit inference support in the Al compiler stack. Improved overall recall by 3.9% and precision by 1.7%."

FSD Beta 10.9 (2021.44.30.10) Official Tesla Release Notes - Software Updates

The reason why that "photon count" terminology is used in relation to RAW images, is because the signal the image pixel gets is generally directly proportional to the amount of photons captured by the given pixel, which is then amplified and converted to a digital number that is saved into a RAW file.

Information Transfer – The ISO Invariant Case

The pixels in the 8-bit images used previously however have no direct relation to the number of photons that hit that pixel.

They aren't talking however about a device that actually counts each individual photon (as you point out, it doesn't really add any advantage to this application).

 

[LorentzTorque]

The photon count bit is only related to 10-bit RAW images which give significantly higher dynamic range than the 8-bit processed images they were using previously.

"Upgraded generalized static object network to use 10-bit photon count streams rather than 8-bit ISP tonemapped images by adding 10-bit inference support in the Al compiler stack. Improved overall recall by 3.9% and precision by 1.7%."

FSD Beta 10.9 (2021.44.30.10) Official Tesla Release Notes - Software Updates

The reason why that "photon count" terminology is used in relation to RAW images, is because the signal the image pixel gets is generally directly proportional to the amount of photons captured by the given pixel, which is then amplified and converted to a digital number that is saved into a RAW file.

Information Transfer – The ISO Invariant Case

The pixels in the 8-bit images used previously however have no direct relation to the number of photons that hit that pixel.

They aren't talking however about a device that actually counts each individual photon (as you point out, it doesn't really add any advantage to this application).

Yes, the signal is proportional, but considering the number of photons involved in a change of 1 part in 2^10 in signal intensity, that's like saying you're counting water molecules with a pipette. It's all a bit beside the point of the thread, but I can see why the terminology irritates some people, or at least makes them roll their eyes.

 

[Boza]

The photon count bit is only related to 10-bit RAW images which give significantly higher dynamic range than the 8-bit processed images they were using previously.

"Upgraded generalized static object network to use 10-bit photon count streams rather than 8-bit ISP tonemapped images by adding 10-bit inference support in the Al compiler stack. Improved overall recall by 3.9% and precision by 1.7%."

FSD Beta 10.9 (2021.44.30.10) Official Tesla Release Notes - Software Updates

The reason why that "photon count" terminology is used in relation to RAW images, is because the signal the image pixel gets is generally directly proportional to the amount of photons captured by the given pixel, which is then amplified and converted to a digital number that is saved into a RAW file.

Information Transfer – The ISO Invariant Case

The pixels in the 8-bit images used previously however have no direct relation to the number of photons that hit that pixel.

They aren't talking however about a device that actually counts each individual photon (as you point out, it doesn't really add any advantage to this application).

It is a BS. They are not “counting photons”; they are just using RAW HDR instead of compressed image - something they should have been doing from day one, anyway.
It is almost childish how they come up with “sophisticated” terms for everyday things, just to sound cool:
“Photon counting” for RAW HDR
Auto Pilot for Adaptive Cruise Control
Full Self Driving for … driver assistance lane keeping?

 

[stopcrazypp]

Yes, the signal is proportional, but considering the number of photons involved in a change of 1 part in 2^10 in signal intensity, that's like saying you're counting water molecules with a pipette. It's all a bit beside the point of the thread, but I can see why the terminology irritates some people, or at least makes them roll their eyes.

Well there's not another really good term that really fits that well to describe what the values in the RAW files ultimately measure as opposed to tone-mapped images which typically have a boatload of other processing done to it (to try to match how humans perceive images).

If talking about measuring water, you can talk about the "volume" of water (for example one gallon) or you can talk about the "mass" of water (for example 1 kg of water). You can get an approximation of how many molecules there are either way. However photons have neither volume nor mass, so can't really be described in such terms. I'm guessing that is why they described it as "count" (even though it is only an approximation).

 

[stopcrazypp]

It is a BS. They are not “counting photons”; they are just using RAW HDR instead of compressed image - something they should have been doing from day one, anyway.
It is almost childish how they come up with “sophisticated” terms for everyday things, just to sound cool:
“Photon counting” for RAW HDR
Auto Pilot for Adaptive Cruise Control
Full Self Driving for … driver assistance lane keeping?

Given this was mentioned in the context of FSD Beta release notes that generally look written by engineers instead of polished by marketing, I find it more likely they are using a description that more closely describes what I mention in the comment above instead of trying to purposefully sound "sophisticated".

 

[daktari]

The Kia uses HDA 2 just like the Hyundai above, presumably they still have HDA (which keeps the front radar)?

That sounds like an upstream chip that controls both. Are they bundled in a package? Do they still offer features that still use the radar (like FCW and AEB)? The BMWs seem to do that. Do you know if they actually removed the radar unit?

The Kia offered a "special edition". Equipment removed was:

Parking collision avoidance assist (when backing out, 2x rear radars?)

Highway Driving assist 2 (auto lane change, 2 rear radars?) Car should still have smart cruise control

FCA crossing (collision avoidance in unprotected turns) car would still have EAB

Remote smart parking assist (Summon like thing)

360 surround view

Blind spot monitor view (side cam in display when blinker on)

Sunroof

Vegan seats

So could be 4x corner radars, 4x surround cams missing or a computer unit or cables. Seems nose radar still was there and working.
They also offer3d SE2 and SE3 versions later but I don't know what they skipped.

 

[LorentzTorque]

Well there's not another really good term that really fits that well to describe what the values in the RAW files ultimately measure as opposed to tone-mapped images which typically have a boatload of other processing done to it (to try to match how humans perceive images).

If talking about measuring water, you can talk about the "volume" of water (for example one gallon) or you can talk about the "mass" of water (for example 1 kg of water). You can get an approximation of how many molecules there are either way. However photons have neither volume nor mass, so can't really be described in such terms. I'm guessing that is why they described it as "count" (even though it is only an approximation).

If you do want to get into the physics... The best justification for calling it a "count" would be that the number of electron-hole pairs created in the semiconductor lattice of the CCD pixel would be proportional to the number of incident photons over a certain energy threshold (but reduced by the finite cross-section of the reaction). This is in contrast to, for example, the total absorbed energy or the strength of the EM wave (macroscopic quantities that would be more analogous to the water-measurement example I gave). So in that sense, it's counting, but it's counting such an enormous number that in practical terms you might as well be dealing with a continuous variable when you actually go to design the circuits and write the software. It's also not unheard-of for engineers to frame things in a way that they know will sound buzzier for the management & marketing teams, so when there's a loose justification for calling it "photon counting" they very well could choose to do so. Even if it's not how they would look at the problem on a practical level.

Side note: I suppose it is remotely possible that they could be dealing with such an attenuated signal that the discrete nature of the interaction would show up, but I doubt they would operate in this regime. They'd probably have a terrible signal-to-noise ratio if they did it this way, unless they kept the CCD very, very cold.

All that being said, I think CCD physics is actually beside the point, because the human-friendly image form of the data is just another representation of the same data that could be obtained as a flat array or 2D array directly from the sensor. It's no more or less of a photon count. Wasn't the original point something about the ML algorithms being trained on the raw feed from the sensor, thereby eliminating the need for some real-time image-processing steps? Seems like a separate issue from whether it is or isn't counting photons.

 

[JHCCAZ]

I also think that "photon count" is essentially correct but an unnecessary descriptor, possibly trying to sound Innovative or sophisticated, but ends up sounding forced. Kind of like saying "Infrasonic Rythmic Perception" instead of "Feeling the Bass".

The established and well-understood term here is simply "raw video". I can imagine that the "photon count" term might have been used in a meeting of the engineers, just to clarify to Elon or someone else who is technical but perhaps not particularly versed in the jargon of camera image processing. Then Elon liked it, picked it up and used it in one or two interviews and now it's all over the Tesla user community

Along similar lines @daktari just reminded me of "Vegan Leather" which seems to have become a popular term for synthetic leather upholstery inTeslas and all sorts of other products. To me, this is also an unnecessary name, a trendy but imprecise term for something that has other long-established names. I don't see anything "vegan" about it, other than it may contain some plant cellulose byproducts for body and texture, mixed in with the the petroleum-based thermoplastics, pigments, stabilizers and so on.

But hey, I can get used to Vegan Leather, Photon Counts and Ego Yaw-Control Planning if it gets me around the corner in comfort and safety.

 

[LorentzTorque]

I also think that "photon count" is essentially correct but an unnecessary descriptor, possibly trying to sound Innovative or sophisticated, but ends up sounding forced. Kind of like saying "Infrasonic Rythmic Perception" instead of "Feeling the Bass".

The established and well-understood term here is simply "raw video". I can imagine that the "photon count" term might have been used in a meeting of the engineers, just to clarify to Elon or someone else who is technical but perhaps not particularly versed in the jargon of camera image processing. Then Elon liked it, picked it up and used it in one or two interviews and now it's all over the Tesla user community

Along similar lines @daktari just reminded me of "Vegan Leather" which seems to have become a popular term for synthetic leather upholstery inTeslas and all sorts of other products. To me, this is also an unnecessary name, a trendy but imprecise term for something that has other long-established names. I don't see anything "vegan" about it, other than it may contain some plant cellulose byproducts for body and texture, mixed in with the the petroleum-based thermoplastics, pigments, stabilizers and so on.

But hey, I can get used to Vegan Leather, Photon Counts and Ego Yaw-Control Planning if it gets me around the corner in comfort and safety.

I'd pat them on the back for counting vegan photons in batches of 10 million if they'd just install ultrasonics on my car.

 

[Watts_Up]

I wonder if the previous Tesla with utra sonic sensors will get a new software update
to replace the use of the sensors with cameras, as this happen for the radar?

 

[JulienW]

My worthless <2¢ #2

Anecdote with loose analogy:

I’m a runner and suffer cold hands. This time of year I debate whether to carry gloves or not. Then I always remember that if I don’t need them just put them in my pocket. However if I don’t bring them and need them then I’m SOL.

Same with the USS. Until Tesla is 100% sure they are no longer needed and ready to flip the switch on “Deep Ultra” they should continue installing them. That way if it takes a couple of years to develop “Deep Ultra” all new buyers will be covered rather than taking delivery of cars with features removed. After all we know how well “Deep ” and “Deep Bright” replaced detected sensors.

Now a photon and an electromagnetic radiation wave (quantum duality buddies) walk into a bar. Whose ID did they check?

 

[JHCCAZ]

I'd pat them on the back for counting vegan photons in batches of 10 million if they'd just install ultrasonics on my car.

A hopeless wish, I'm afraid. But do not feel alone. It will not be long before all of our Ultrasonics will have chirped their last.

---

The hour grows late, and as I close my eyes I imagine that I can see into the future. (it is a gift of prophecy and a burden I bear). Yes... I foresee... that when the time is nigh, the Grim Reaper of Deprecated Subsystems shall come, gliding through the night, in the form of a major Firmware Update (also known as a "big FU") to bring order and conformity to all Teslas, and to chastize and humble all the unbelievers who yet lurk in this forum.

He shall carry the mark of 2022.666.0.0.0.0.

He is surely Coming Soon, it hath been written.

And as ye await in dread of this calamity, pray and pray that ye may be saved by the miracle of the Vision! Only this can defend that which ye hold most precious, thy bumpers and thy fenders. Cling ye must to this hope, for indeed it shall be at this appointed Hour of Update that our poor beloved family of innocent transducers shall be sacrificed, silenced forever.

But, before you affirm the fateful Install:
Park thee Well, one last time. Gather thy dogs and thy cats, two of each, to witness that last mournful whistle, as only they can hear. That final music, the Ultrasonic Swan Song, fading and fading into the Immeasurable Distance of the Tesla future...

 

[mongo]

Within some accuracy and precision:

Amps are electron count
Moles are molecule count
CCD values are photon count

Outside a highly funded lab, are any close to exact? No.

CCD values can be raw or processed.
It is a raw (unprocessed), but linearly scaled and offset by sensor physics, photon count. They never claim it to be an exact photon count.

All that being said, I think CCD physics is actually beside the point, because the human-friendly image form of the data is just another representation of the same data that could be obtained as a flat array or 2D array directly from the sensor. It's no more or less of a photon count. Wasn't the original point something about the ML algorithms being trained on the raw feed from the sensor, thereby eliminating the need for some real-time image-processing steps? Seems like a separate issue from whether it is or isn't counting photons.

Human friendly version is a lessor representation because the transformation from raw CCD values to RGB colorspace loses data/ precision.
Black and white or greyscale (via processing) are also a representation, but you would not say they contain the same amount of information as color.

Raw: 4 pixels with 10 bits of data (more if HDR)
Processed: 1 pixel with 3 channels of 8 bit data (more if HDR)

Upgraded generalized static object network to use 10-bit photon count streams rather than 8-bit ISP tonemapped images by adding 10-bit inference support in the Al compiler stack.

Coursework (not mime) on it showing transfer curves.
http://graphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture12.pdf

The established and well-understood term here is simply "raw video". I can imagine that the "photon count" term might have been used in a meeting of the engineers, just to clarify to Elon or someone else who is technical but perhaps not particularly versed in the jargon of camera image processing. Then Elon liked it, picked it up and used it in one or two interviews and now it's all over the Tesla user community

Except that (to the public) raw video can mean unedited footage, versus the industry meaning of unprocessed. In that sphere, saying raw video adds nothing to their understanding.

 

[DrGriz]

I wonder if the previous Tesla with utra sonic sensors will get a new software update
to replace the use of the sensors with cameras, as this happen for the radar?

They said they wouldn't disable the sensors on those cars. Yet.

 

[Dewg]

Like many here, I think this is a global supply issue. The question really comes down to: How long do you want to wait for your new car? 6 months? 12 months? 18 months? Could be significant depending on supply.

 

[Boza]

Given this was mentioned in the context of FSD Beta release notes that generally look written by engineers instead of polished by marketing, I find it more likely they are using a description that more closely describes what I mention in the comment above instead of trying to purposefully sound "sophisticated".

That makes it even scarier - engineers using inaccurate terms.

 

[Boza]

The automated dihydro-oxygen removal tool exhibits highly randomized oscillations due to susceptibility of the photon counting apparatus to external interference. The confidence interval of photon counting is apparently too wide because it causes random behavior in the front wide spectrum, non-coherent photon emitters.

 

[Skotty]

My concern is that USS is a proven technology that works extremely well. Absolute best case scenario with using vision instead -- is that it turns out to not be worse. For consumers there will be zero benefit. If there is any savings related to this, Tesla will be the one pocketing the profit.