Welcome to Tesla Motors Club
Discuss Tesla's Model S, Model 3, Model X, Model Y, Cybertruck, Roadster and More.
Register

Tesla replacing ultrasonic sensors with Tesla Vision

This site may earn commission on affiliate links.
I really hope they do not use wheel rotation to measure distance. Wheel circumference is affected by many factors, aside from the wheel size you select in the UX.
Doubtful. They have to use accelerometers and do intertial navigation.

I really hope they do not use wheel rotation to measure distance. Wheel circumference is affected by many factors, aside from the wheel size you select in the UX.

The idea with static/dynamic objects is interesting but I am not sure if it is practical. There is a similar concept in navigation but we use predetermined static objects; the robot does not have to make that determination - just to orient itself towards them.

Then there have to be objects there where it knows it's sizes so I doubt it.

Speed + how much features shrink / grow can determine the distance. The slower the features grow, the further away you are. If they grow at a faster rate you are closer. So my guess is that they have to look for the "line" feature where the floor meets the wall etc.
 
Doubtful. They have to use accelerometers and do intertial navigation.
[disclaimer that I don't think there's any published material on how the TV USS replacement will work, so this is all guesswork extrapolating from the FSD presentations]

I don't think either inertial navigation nor wheel rotation is sufficiently accurate for the sort of tolerances needed in parking movements.

Rather than thinking that they need to find a way to use a camera instead of an ultrasonic sensor I think we have to consider what we know about TV works. TV doesn't work by doing image recognition off individual cameras (the current Autopilot etc does do this). With TV the output of all cameras are put together cumulatively and a 3D model of the environment is built. After that, the entities in that environment are identified and classified and their potential future motion is predicted. This is refreshed 10s (or hundreds, can't remember) of times every second, but the refreshes take in to account the existing model - it's not throwing away its conclusions from the last time the model ran and as a result it is capable of modelling entities that it has lost sight of due to obstructions etc (the motion of those objects will be based on guesswork until they reenter field of view, obviously).

Given that foundation, the parking system challenge isn't about looking at cameras and working out what's going on - it's about taking the vehicle which exists in the 3D space the system has modelled and reporting on its location vs nearby obstacles as it moves. This must be possible because it's required for the stuff above to work; you can't have object permanence unless you're capable of understanding that a thing is still the same thing despite having appeared to move relative to you.

This is why a raindrop on a camera is not necessarily a problem - given that the car moves any partial obstruction only obscures some of the image some of the time. It's also why the blindspot at the front of the car nose is not a massive issue. In order to get in to that space an object has to pass through space that the car is modelling - or in the case of a static object, the system should understand that the object continues to exist even after it's passed out of view of the camera. There is an edge case for objects that move in to that space while the car is off and not modelling it's surroundings (the hypothetical 'child lies down in front of your car while it's parked' scenario), but aside from that the car shouldn't get surprised by objects in that space.

This is also why I think this function is blocked behind the supposedly imminent single stack release - it's 100% built on it.
 
Last edited:
Doubtful. They have to use accelerometers and do intertial navigation.
I would assume the Tesla auto pilot system does use inertial navigation if I had to guess. We know it has accelerometers since it is able to show acceleration magnitude and direction in track mode. It seems they would be giving up free information if they didn't use this data in their trackers.
 
This is refreshed 10s (or hundreds, can't remember) of times every second, but the refreshes take in to account the existing model - it's not throwing away its conclusions from the last time the model ran and as a result it is capable of modelling entities that it has lost sight of due to obstructions etc (the motion of those objects will be based on guesswork until they reenter field of view, obviously).

This is the goal of Tesla Vision, but I do not believe the software in our cars is working that way yet. Maybe with V11 we will finally get object persistence and prediction in our cars.
 
I would assume the Tesla auto pilot system does use inertial navigation if I had to guess. We know it has accelerometers since it is able to show acceleration magnitude and direction in track mode. It seems they would be giving up free information if they didn't use this data in their trackers.

I think tesla vision in auto pilot is trained with the radar as ground truth and has just learned to memorize the typical size of a car and calculate the distance based on how many pixel it occupies in the image.
 
  • Like
Reactions: MrBadger and enemji
Hi,
We have a 2023 MY Shanghai built Model 3 Performance - with USS.
Interestingly - I reverse the car into my garage and the front of the car is just a few feet from the automatic roller door.
When I get in the car and put on my seatbelt and put it into drive - the screen is showing an object In front of the car (the garage door)
I then open the door with the remote control and wait for it to fully raise.
When I then move forward - the display is still showing the object in front of the car (‘which is no longer there) and even shows me getting nearer to it!
So - the existing USS already has some sort of “persistence of vision” function!
Cheers
Steve
 
I have the latest FSD Beta software and I haven't seen any indication that the object persistence algorithms are running on that version yet.
There is a variation of it running today. You can see it when at a stop light, watching cross traffic. If there is a truck or some other large vehicle in front of you, you'll see cars blink in and out of visualization or wobble/wiggle as the cameras lose sight of them. There is about a 0.5-1 second persistence, where the computer is projecting the path of the object even after the cameras have lost sight of it. Usually you'll see the car wiggle around in its lane until it snaps back into proper position after the cameras pick it up again on the other side of the truck.
 
  • Like
Reactions: Mullermn
There is a variation of it running today. You can see it when at a stop light, watching cross traffic. If there is a truck or some other large vehicle in front of you, you'll see cars blink in and out of visualization or wobble/wiggle as the cameras lose sight of them. There is about a 0.5-1 second persistence, where the computer is projecting the path of the object even after the cameras have lost sight of it. Usually you'll see the car wiggle around in its lane until it snaps back into proper position after the cameras pick it up again on the other side of the truck.

I'll have to take a closer look next time I am in my car. The last time I paid attention to it, the traffic passing behind obstructions was immediately disappearing, and then reappearing once it got past the obstruction.

I will say, though, that 0.5 to 1 second of persistence will not be enough!
 
Is it possible that it is just a delay in processing?
Good call Boza, I thought that too. But when I paid attention to the objects, their trajectory wobbles when they're out of camera view and then snap back to smooth trajectory and pinpoint positioning when the cameras pick it back up. The wobble must be the system predicting their trajectory based on last observed position and speed. And it's only for a second or less, so if the cameras don't pick them back up, they disappear from visualization - either permanently (if they're no longer visible), or temporarily (they blink back in when a camera spots them again).
 
  • Informative
Reactions: Boza
Is it possible that it is just a delay in processing?
This was discussed in the ON keynote, you can see it does a good job of predicting object trajectory through occlusions, as well as through multicam views (versions before Tesla Vision even have cars blinking in and out when they straddle two camera views, you see none of that happening here). See starting 18 minutes for an example:

This point was addressed also when they first introduced Tesla Vision and talked about RNN (Recurrent Neural Network). Before this, they used to use a single frames to do perception, and when objects get occluded, the object may disappear or have very wrong position predictions. See 1:09:

Blue is the video based perception with persistence. Orange are based on single frames. You can see when the pickup truck is not blocking the two cars, the single frame does ok (although even so, because the camera views were not synced, the orange results still have quite a bit of variation):
AIDay1.jpg


When the pickup truck blocks the two cars, the two completely disappear from single frame (while they still exist in the video based one):
AIDay2.jpg


When partially occluded, the cars still hold their positions in the video based, while for the single-frame, they are all over the place, even the orientation is wrong:
AIDay3.jpg


Given my car personally is still on a non-Tesla Vision version, I see a lot of the single frame artifacts in the visualization (cars blink in and out of visualization when straddling camera views, when partially occluded I see also the weird orientation changes with the car rotating like in picture above).

ON should further improve on above given it gives general 3D recognition of occupancy, while previous NNs are based on recognizing a certain object type, like a car or a person. From above you can see it gives a 2D bounding box based recognition and a top down recognition, but nothing like the 3D model of the occupancy network (which even has the rough shape of the object, which is achieve for "free").

And also ON works at much higher rates (100fps) and is very memory efficient (allowing Tesla to have longer persistence if they desire).
A Look at Tesla's Occupancy Networks

My guess is when it switches to parking assist mode (the mode where it shows a zoomed in top down view and the USS pings and distance measurement), they can reduce the viewing distance of the ON (ignore objects far away which USS can't detect anyways given they have 2-5 meter max range), which can allow them to boost persistence and resolution (using smaller size voxels). This may come at a later version however if they want to get something out working quickly first.
 
Last edited:
I use my USS twice a day, once to wedge myself in between my motorcycle and my husband’s car at home and again at work to squeeze into very cramped charging spaces with inconsistent bollard placements.

My USS have been dead accurate for five years now. Come to think of it, the USS are the only reliably accurate sensors on my Tesla. Clearly USS had to go so as to match the performance of FSDb, auto wipers, auto lights, auto mirror dimming and auto seat heaters.

I’m capable of parking on my own, of course, but for seventy grand I want this functionality to be present and reliable. I can crank a window (and did for a long time) but for these prices I expect the windows to be power. Same with locks and lights and on and on. These are expensive cars.

I wouldn’t even consider the purchase of a Tesla without these sensors and, based on five years’ worth of Elon’s software misrepresentation, I wouldn’t consider a car with a vision-only solution until there were many tens of thousands of happy voices singing its praises.

No wonder inventory is piling up.
 
I use my USS twice a day, once to wedge myself in between my motorcycle and my husband’s car at home and again at work to squeeze into very cramped charging spaces with inconsistent bollard placements.

My USS have been dead accurate for five years now. Come to think of it, the USS are the only reliably accurate sensors on my Tesla. Clearly USS had to go so as to match the performance of FSDb, auto wipers, auto lights, auto mirror dimming and auto seat heaters.

I’m capable of parking on my own, of course, but for seventy grand I want this functionality to be present and reliable. I can crank a window (and did for a long time) but for these prices I expect the windows to be power. Same with locks and lights and on and on. These are expensive cars.

I wouldn’t even consider the purchase of a Tesla without these sensors and, based on five years’ worth of Elon’s software misrepresentation, I wouldn’t consider a car with a vision-only solution until there were many tens of thousands of happy voices singing its praises.

No wonder inventory is piling up.
If that's true, the other CEOs of the other car mfgs should be whipping their workers with horse whips to get new EVs out with ADAS features on par with Tesla yesterday. They can't afford to miss this shot, or history will judge them poorly, as will the stock market.
 
If that's true, the other CEOs of the other car mfgs should be whipping their workers with horse whips to get new EVs out with ADAS features on par with Tesla yesterday. They can't afford to miss this shot, or history will judge them poorly, as will the stock market.

Let me know when BMW or Honda start promising that their existing products on the lot will do something in the future that they are incapable of when a customer takes delivery.

If my 15 months with FSDb are any indication, Tesla is a still a long, long way from delivering what they sold me in 2018. There is no way I would trust their software promises at this point without proof.
 
Let me know when BMW or Honda start promising that their existing products on the lot will do something in the future that they are incapable of when a customer takes delivery.


They've already been doing that for years.

See Fords Blues Clues ADAS system for example- where it wasn't even on the car (software anyway, HW was) initially-- they they OTAed the software the next year... but still have added promised features not yet delivered (and the delivered ones kinda suck)
 
They've already been doing that for years.

See Fords Blues Clues ADAS system for example- where it wasn't even on the car (software anyway, HW was) initially-- they they OTAed the software the next year... but still have added promised features not yet delivered (and the delivered ones kinda suck)

That’s a good example and Cadillac is in the same boat with the Lyriq and supercruise at the moment.

I think all bluecruise features have been delivered now (and I’m also under the impression they suck) but maybe not.

The sentiment stands though, I wouldn’t consider one of those products either until they were proven to me. Those software suites also don’t cost $15k and promising software to go with the hardware is fundamentally different than promising that software can replace hardware when it’s never been done before. Supercruise works very well on the exact same hardware on other GM vehicles, for example.

I’ve been burned by Tesla promises enough that I wouldn’t even consider replacing mine with another until they prove that Vision can replace my USS. I’m done with the blind trust.