Tesla replacing ultrasonic sensors with Tesla Vision

[Battpower]

Maybe a shadow and a sideways truck look similar to cameras (but quite different to a radar sensor!)

Although radar has problems with completely stationary objects ... which is part of the problem still being addressed.

I'm not sure that radar's place / role is similar enough (or even at all) to USS deletion to merit how much we have ended up discussing it. Imo the connection is basically that Tesla feels VO is capable of displacing both radar and USS.

 

[stopcrazypp]

That's kind of my point. That NN confidence is not like human confidence. I doubt anyone knows how a given NN is 'NN confident' about anything (IE: what characteristics it finally used to determine something), but based on vision only it has a sensorial depravation compared to humans.

It makes sense with NN having such a limitation to restrict your objective to 'locating clumps of stuff' / 'volumetric driveable space' / ON.

So I agree ON could be the basis for solving a lot of issues.

But the myopic vision only approach does not make as much sense as other multi-sensory approaches sense imo since cameras get obscured and often - by sun, by rain / road spray / fog, by dirt / mud splashes / bird excrement etc especially for AEB for example.

Someone beat me to it, but I was just using AP this weekend when the sun was shining directly into the camera (it was shining into my eyes too, so I had to put down the visor) and it was able to lane keep just fine even on curvy roads. The ability to keep within a lane is entirely based on vision, so if vision is obscured, the system can't be expected to work properly anyways. That's why there is so much focus on vision.

If however the goal is only to run ACC, this perhaps is not needed, but Tesla has never prioritized that (nor do they even offer a dumb CC mode except for cars not equipped with AP).

 

[Battpower]

I hadn't seen that aidriver video of fsd with cameras and radar blocked off one at a time etc. That is pretty impressive.

We all seem to focus on different combinations of observations. While your recent observations and aidriver's video show how capable the vision approach can be for fsd, I have been focusing more in my mind on radar for aeb. Now if radar version requires fsd (or at least the ai NN behind it) to be running to trigger aeb then I think that changes my view somewhat at least related to fsd beta.

If single stack is here soon and ends up available to my UK car, I feel more ready to let go of radar once I've heard enough positive evidence that VO is at least no worse.

 

[stopcrazypp]

Hi,
That video shows that even when he covered all the forward facing cameras - the car was still able to drive on FSD.

Your description is very misleading. I watched the video. Other than leaving only the main camera uncovered, as soon as he covered two front, including the main, FSD Beta refused to activate.

When he covered all the cameras, the visualization was able to pick up the road, but it still refused to start.

If his tape is electrical tape, I'm guessing it's slightly translucent, so with all cameras taped, the system may have maxed out the gain, and combined with it being night and headlights shining, the road lines may still have enough contrast through the translucent tape for one or more cameras to pick up some faint patterns.

Did he forget that the car has GPS and therefore should know where it is on a map?
This means it should know where the roads are - but not be able to see objects, pedestrians, cars etc.

AFAIK, from Tesla presentations, Tesla does not rely on a detailed GPS map to determine curbs. The curbs are all generated based on the cameras. Very early on (from memory around 2016) Tesla did dabble a bit on generating lane maps, but by the time FSD Beta was out they have abandoned that approach.

Regarding phantom braking under bridges - you would imagine that the system knows from GPS that a bridge is present.
It also knows the time of day, direction and position of the sun - relative to the front of the car, outside temperature (an indication whether it is potentially a sunny day), external light level sensor - again indicates sunny day.
With all the above data and the different camera views - you would imagine that the likelihood of a shadow being present under a bridge is quite predictable.

The braking for bridges or overpasses is mostly a problem affecting radar. Tesla did use a GPS based map to whitelist those areas to try to reduce the chances of phantom braking:

Tesla releases the details of its new radar processing technology: point cloud, 2-car ahead tracking & more

Tesla and Elon Musk released the much-anticipated blog post about Autopilot improvements under its upcoming v8.0 software update. They confirmed a...

electrek.co

However - I remember an early bad accident that somebody had in a Tesla (model S) which was on AP and it ploughed through the side of a truck that was across the road in front of it.
Maybe a shadow and a sideways truck look similar to cameras (but quite different to a radar sensor!)
Cheers
Steve

That was one of the earliest accidents on AP1 and it was when the car had radar. The radar tuned out what looked like a stationary object and matches a profile of a roadsign (a large flat surface with empty space below it, keep in mind semi trucks are quite high).

Understanding the fatal Tesla accident on Autopilot and the NHTSA probe

The first reported death in a Tesla Model S crash while the Autopilot was activated has been shaking the Tesla...

electrek.co

The camera didn't catch it either (though white surface was sky).

 

[Boza]

Someone beat me to it, but I was just using AP this weekend when the sun was shining directly into the camera (it was shining into my eyes too, so I had to put down the visor) and it was able to lane keep just fine even on curvy roads. The ability to keep within a lane is entirely based on vision, so if vision is obscured, the system can't be expected to work properly anyways. That's why there is so much focus on vision.

If however the goal is only to run ACC, this perhaps is not needed, but Tesla has never prioritized that (nor do they even offer a dumb CC mode except for cars not equipped with AP).

Try it in a snowstorm. The FSD refuses to activate but AP works. The funny part is that I cannot see s%*t but this thing wanted to drive with the speed limit. When I overwrote it to a sensible speed it was doing reasonably well. Granted, there were guardrails on the road and it could see the car in front of me (which I could barely recognize from time to time).

 

[stopcrazypp]

Try it in a snowstorm. The FSD refuses to activate but AP works.

I presume you mean FSD Beta? Being end to end L2, it makes sense for it to be more cautious. I don't think any of the L4 systems operate in snowstorms for the same reason.

The funny part is that I cannot see s%*t but this thing wanted to drive with the speed limit. When I overwrote it to a sensible speed it was doing reasonably well. Granted, there were guardrails on the road and it could see the car in front of me (which I could barely recognize from time to time).

If you have the setting Controls > Autopilot > Set Speed to "Speed Limit" it will try to keep the higher number between the detected speed limit and your set speed. I believe that is the default. You can change it to "current speed" to ignore speed limits and keep at your set speed. The manual specifically warns that you need to judge yourself if your speeds are safe for the conditions.

Model 3 Owner's Manual | Tesla

Learn about Traffic-Aware Cruise Control and its specifications.

www.tesla.com

Note above applies to highway driving, in cities there are different limits (it will try to do speed limit +5).
See restricted speed section in manual.

Model 3 Owner's Manual | Tesla

Learn about Autosteer and its specifications.

www.tesla.com

Radar can maintain ACC by following the car in front, but it's definitely dangerous if vision is obscured to the point you can't see the target vehicle. If another stationary truck situation came up it may be disaster. The front bumper can also accumulate snow which blocks the radar and may disable AP (I'm pretty sure I saw complaints of this although it may affect Model S more).

 

[Boza]

It is the regular FSD, not beta. I would not have let it drive anyway - the conditions were too bad for me to trust it. Frankly, using the AP was more of an experiment, I was curious at the time but I will not repeat it.

Even if they call it FSD it will take decades before it becomes comparable to a human driver. It is a great assistance feature on a clean highway but nothing more.

There are a lot of postings about how beta is much more refined. It may be so but I am holding off any future upgrades before there is solid, long term evidence - been burned too many times. In a strange way, I would gladly give up the AP and FSD (and all the games, farts, etc.) for a “good ol’” dumb cruise control, regular auto wipers and better, dumb headlights. If they throw in a better UI I may even send them a thank you letter

 

[stopcrazypp]

It is the regular FSD, not beta. I would not have let it drive anyway - the conditions were too bad for me to trust it. Frankly, using the AP was more of an experiment, I was curious at the time but I will not repeat it.

You said FSD did not activate, but AP did. There is no FSD mode except FSD Beta. Did you mean instead Autosteer (double press of stalk) did not activate, but TACC (single press) did?

If so, that is entirely expected. ACC definitely can work just on radar because most cars in fact just use radar to do it. The system doesn't need to keep in the lane, so the cameras are not really necessary. As mentioned, however if you can't see the target vehicle you are taking a risk yourself.

Even if they call it FSD it will take decades before it becomes comparable to a human driver. It is a great assistance feature on a clean highway but nothing more.

There are a lot of postings about how beta is much more refined. It may be so but I am holding off any future upgrades before there is solid, long term evidence - been burned too many times. In a strange way, I would gladly give up the AP and FSD (and all the games, farts, etc.) for a “good ol’” dumb cruise control, regular auto wipers and better, dumb headlights. If they throw in a better UI I may even send them a thank you letter

You don't need Beta yourself, you can just look at videos that show the latest versions. It should be immediately obvious it's much more advanced than what we see using production code.

 

[Boza]

One click - AP; two clicks - FSD (according to the manual). There are plenty of cases when AP is available but FSD (blue wheel) is not.

Videos are impressive, I agree. However, those are carefully selected cases. I would like to see how it performs in the real world. I think I have the right to be skeptical given Tesla’s track record (the wipers performance, UI attempts, etc. ). Great EV but pretty spotty on the AI side.

 

[arbitrage000]

If you don't have FSD beta (autosteer on city streets) then isn't FSD just the same as AP except for the ability to change lanes by itself, navigate on and off ramps and stop at stop signs and traffic lights?

And even if you have FSD beta it is still just using AP when on highways until the stacks are merged.

 

[Supcom]

If you don't have FSD beta (autosteer on city streets) then isn't FSD just the same as AP except for the ability to change lanes by itself, navigate on and off ramps and stop at stop signs and traffic lights?

And even if you have FSD beta it is still just using AP when on highways until the stacks are merged.

You can look up all the capabilities of FSD on Tesla's web site.

 

[Boza]

If you don't have FSD beta (autosteer on city streets) then isn't FSD just the same as AP except for the ability to change lanes by itself, navigate on and off ramps and stop at stop signs and traffic lights?

And even if you have FSD beta it is still just using AP when on highways until the stacks are merged.

I am afraid that you are incorrect, at least according to Tesla. My 2020 bill of sale includes the FSD option. Additionally, under Software, I see “Autopilot: Full self-driving computer” and “Full Self Driving Capability: Included Package”.

BTW, the FSD _does_ steer on (some) city streets but I would never let it do that.

 

[stopcrazypp]

One click - AP; two clicks - FSD (according to the manual). There are plenty of cases when AP is available but FSD (blue wheel) is not.

I get the confusion now and why your terminology is not consistent with what others are discussing. What you refer to as "FSD" is AP, more accurately Autosteer:
"To initiate Autosteer, move the drive stalk fully down twice in quick succession."
Model 3 Owner's Manual | Tesla
One click is TACC (Traffic-Aware Cruise Control):
"To engage Traffic-Aware Cruise Control when it is available (the car status area of the touchscreen displays the gray cruising speed icon), move the drive stalk down once, then release the accelerator pedal to allow Traffic-Aware Cruise Control to maintain the cruising speed."
Model 3 Owner's Manual | Tesla

When people refer to FSD, they usually are referring to FSD Beta. Simply having the FSD option means you have the same visualizations as someone with regular AP, nothing more advanced. You need FSD Beta to have the advanced features.
To toggle FSD Beta on or off, you toggle this option in the menu. Most people just set two profiles to make switching much faster.

Videos are impressive, I agree. However, those are carefully selected cases. I would like to see how it performs in the real world. I think I have the right to be skeptical given Tesla’s track record (the wipers performance, UI attempts, etc. ). Great EV but pretty spotty on the AI side.

 

[stopcrazypp]

I am afraid that you are incorrect, at least according to Tesla. My 2020 bill of sale includes the FSD option. Additionally, under Software, I see “Autopilot: Full self-driving computer” and “Full Self Driving Capability: Included Package”.

As mentioned in other comments, just having the FSD option means the visualizations and base code is the same as AP. You only get a few more features like NOA, but the base code is no more advanced than what regular AP has.

You need FSD Beta to actually get the most advanced code. You can see the video here that shows the difference when they toggle the two. If you don't see the red curb visualizations that can zoom out, you aren't running the most advanced code.

BTW, the FSD _does_ steer on (some) city streets but I would never let it do that.

AP also autosteers on some city roads, that is not a feature unique to the FSD package. FSD Beta however has the car be able to make right and left turns at intersections, something neither AP nor the regular FSD package can do. If instead you are saying your car does make right/left turns at intersections, you are the first to report it without FSD Beta.

 

[Boza]

I think I see where the confusion is coming from. I have the FSD package (hence, it is incorrect to say that I do not have FSD) that delivers Autosteer capability. FSD beta (which some people incorrectly shorten to “FSD”) is a very different package that tests more advanced capabilities than Autosteer.
So called Autopilot includes the TACC (via built-in functionality) and, optionally, Autosteer (via the optional FSD).

Source: Model S Owner's Manual | Tesla

Autosteer (FSD) does steer on few city streets but it gives up at intersections (even though it recognizes traffic lights colors), has no idea about turning lanes (even though I “sees” the arrows), etc. Basically, it is unusable (but not disabled) after it exits the highway.

 

[Battpower]

I'm surprised there is any confusion between FSD and FSD Beta.......

Yet at the same time I'm not! In discussions on TMC what you yourself (probably) know is often more a function of the other person telling you what they believe is the case than what you actually know yourself!

To someone who is a casual observer, how can you tell what the chronology, functionality or hierarchy is from the following terms:

AP, AP1, AP2, AP2.5, EAP, FSD Capable, FSD, FSD Beta, HW2, HW3, ..... ?HW4, MCU, MCU2, MCU3.....

Then superimpose v8, v9 ...... V11, and what the significance of those references could be in different contexts...

And if you tried to correlate the above with Tesla's marketing material you'd probably have just as hard of a time. And when 'the same car variant' driven in a different country on different road layouts in different weather can behave so differently from another instance....

I've never owned a car (or anything come to think of it) that has required so much ongoing effort just to try and keep up my understanding of what I can maybe expect from the product.

And I nearly forgot the issue of discussions sometimes appearing to forget or ignore which vehicle model the poster is referring to as they write ..

Ostensibly many of us purchase / subscribe to the same thing yet what we actually have or might end up with could vary considerably. You can say 'buyer beware' and 'Tesla made it perfectly clear' all you like, but I don't see that at all. The way Tesla used loyal owners, referral codes and a highly motivated social media following to promote products based on highly dubious and nebulous claims hardly makes anything clear. These are still the foundations on which many current owners' reasonable (or unreasonable!) expectations are based.

In what system does xyz-beta go into an extensive public test phase months after buyers were being sold 'xyz-non beta' which again by normal use of terms would suggest 'xyz-final'?

What sense is there in including features like traffic light warnings / control in a system intended to be used on road types that rarely if ever have traffic lights?!

[Not FSD related, but in a similar way, who can make realistic efficiency claims of cars that then require endless tweeks to energy hogging battery heating (and cooling) to 'optimise battery life'? And to what real purpose?]

At the time I purchased my MS towards the end of 2019, it didn't take much digging to work out that if you purchased the FSD option at that time, the only tangible difference from the lower spec EAP at that time was that you had a line item on an invoice saying Full Self Driving option.

The removal of USS, ongoing changes in behaviour of auto-headlights and wipers, optimization based on landscape screens, removal of radar etc all paint a chequered development path with very little clarity at all.

Even now, the continued use of 'coming soon' is at best meaningless when taken in context. IMO something this poorly defined is usually called Snake Oil. Any normal interpretation of 'coming soon' other than in a cosmic sense would usually at least mean 'before your car becomes obsolete', 'before your warranty expires' or 'before your finance deal ends'.

 

[Battpower]

I have EAP, which is basically FSD except for city streets,

From a completely unrelated post, but it links with my point above about product names etc

 

[Boza]

In your reference to buying the car at the end of 2019 when things were clearer: Something happened about two years ago and EM lost touch with reality. That shows everywhere: The obsession with AI, super confusing and almost dictatorship approach to design, ignoring obvious flaws in favor of ideology, weird public statements, Twitter fiasco… Almost like “how the world must be” vs. the “reality of the world”.

 

[aronth5]

I'm surprised there is any confusion between FSD and FSD Beta.......

Yet at the same time I'm not! In discussions on TMC what you yourself (probably) know is often more a function of the other person telling you what they believe is the case than what you actually know yourself!

To someone who is a casual observer, how can you tell what the chronology, functionality or hierarchy is from the following terms:

AP, AP1, AP2, AP2.5, EAP, FSD Capable, FSD, FSD Beta, HW2, HW3, ..... ?HW4, MCU, MCU2, MCU3.....

Then superimpose v8, v9 ...... V11, and what the significance of those references could be in different contexts...

Of course if Tesla moved at the glacial pace of most of car companies it would be relatively easy to keep track. None of what you mentioned bothers me in the slightest since that is typical for a tech company. Or maybe I've gotten used to Tesla "confusion" having followed them every day since 2008.

 

[Battpower]

Just to be clear, personally I'm not unhappy with my car. But I am and have always been dubious to highly dubious about Tesla since it took 4 attempts over a few months for them to find me a car (UK) that they could actually raise an invoice for. Like you @aronth5 I've followed Tesla for years since Roadster launch. I really wanted a Roadster (living in San Diego at the time) but the timing was wrong. Especially from a UK perspective I feel things have just got too far off the rails as far as how much work it is to try and understand what's going on with your car. Sure, if you're just happy with the kool-aid, no problem. And if you are getting SOMETHING, (FSD beta) I guess that's a consolation.

When I bought my Kona EV, I didn't need to follow Hyundai for 10+ years. I saw they had battery issues. I saw how they appeared to deal with that. Made my choice, and the rest is (good) history. No rollercoasters.

My MS purchase was my choice rather than putting down a deposit for the new Roadster. I reckon I'm better of with the choice I made!

Thinking back, I agree @Boza, things did seem more settled back in 2019 (on many fronts!) and that's why I decided to give Tesla a try. I agree with the point that the motor industry traditionally moved slower than snails pace, and no disruptor on Tesla's scale is going to have an easy time of it. But by now, shouldn't we be seeing a bit more maturity?