Elon tweets "pure vision" will solve phantom braking

[mikes_fsd]

What he does not address, however, is the vertical blue lines before the car starts to slow down. In the previous slide, he says that the vertical lines on the radar signal leads to phantom braking. Does the vertical lines on this blue line indicate phantom braking caused by the vision system?

He covers that: Go back and listen at the 26 minute mark.

The previous vision was still doing depth/velocity predictions but the bar was set at the radar, i.e. the radar was the standard.
The problem is no matter WHAT radar gives, you still have to correlate it (fuse it) to vision to make sure you care about that object.

The point he is making in that scenario is that vision saw it way earlier and locked on to it earlier than radar did. And since it is vision that is doing the detection and depth/velocity predictions, there is no fusion to worry about.

Link to the timestamp.

https://youtu.be/NSDTZQdo6H8?t=1569

I don't think we (audience members of his talk) have enough information to be able to determine the answer to that question, but I do find it interesting that the abrupt changes to the position signal in the radar version that are a "problem" are apparently not one in the vision version. Or maybe they are and he just failed to mention it.

I would have to disagree, as shown above.

 

[powertoold]

What he does not address, however, is the vertical blue lines before the car starts to slow down. In the previous slide, he says that the vertical lines on the radar signal leads to phantom braking. Does the vertical lines on this blue line indicate phantom braking caused by the vision system?

I don't think we (audience members of his talk) have enough information to be able to determine the answer to that question, but I do find it interesting that the abrupt changes to the position signal in the radar version that are a "problem" are apparently not one in the vision version. Or maybe they are and he just failed to mention it. Or, if it is not a problem in the vision system, how come those same techniques couldn't be used in the radar vision?

In the hard braking example, the "flickering" of the yellow radar graph was to demonstrate that the lead car disappeared and reappeared several times for the radar. This led to an unsmooth braking with the radar. This wasn't an example of phantom braking (in the talk).

In the stationary white trailer example, the flickering of the blue vision graph was due to the vision system's lack of confidence in the trailer ahead (since it was so far). The flickering could also be due to the vision-system not being sure that the trailer was in the lane of travel since only ~15% of the trailer was in the lane. The car began to slow down once it was confident that there was a trailer ahead.

But you may be right. If the vision system has false positives (detects a car closer than it actually is or wrongly detects a non-car as a car ahead), it could lead to phantom braking.

Also, Tesla has advertised the vision-only system's 250m "seeing" ability, but in the white trailer example, it saw it at around 150m instead of 250m (assuming the graph's Y-axis is in meters). Perhaps the system can see 250m in the best circumstances (large truck fully within the lane of travel at 250m).

 

[mikes_fsd]

Also, Tesla has advertised the vision-only system's 250m "seeing" ability, but in the white trailer example, it saw it at around 150m instead of 250m (assuming the graph's Y-axis is in meters). Perhaps the system can see 250m in the best circumstances (large truck fully within the lane of travel at 250m).

The info is in the slide:

250m is the max distance. you still need time to determine what you saw at 250m is useful/needed to be tracked and reacted to.

Initial reaction in this test was at 180m and consistent from 145m
vs the alternative of 110m with the radar.

 

[FloridaJohn]

In the hard braking example, the "flickering" of the yellow radar graph was to demonstrate that the lead car disappeared and reappeared several times for the radar. This led to an unsmooth braking with the radar. This wasn't an example of phantom braking (in the talk).

Yes, you are correct. I made the jump to phantom braking. What I should have said was that the radar had low confidence in the emergency braking car it was approaching.

Also, Tesla has advertised the vision-only system's 250m "seeing" ability, but in the white trailer example, it saw it at around 150m instead of 250m (assuming the graph's Y-axis is in meters). Perhaps the system can see 250m in the best circumstances (large truck fully within the lane of travel at 250m).

Good observation, I did not notice that. I wonder if that is why the vision-only cars are limited to 75 mph; the vision algorithm can't reliably detect obstructions far enough away to drive at a higher speed?

 

[powertoold]

The info is in the slide: View attachment 676757

250m is the max distance. you still need time to determine what you saw at 250m is useful/needed to be tracked and reacted to.

Initial reaction in this test was at 180m and consistent from 145m
vs the alternative of 110m with the radar.

Just based on the blue line graph, it doesn't look like there was a confident "lock" and slow down until ~150m. But ya, it seems the car did see the trailer before 150m. Again, this all could be because the trailer wasn't fully within the lane, so the confidence is lower as to whether it's relevant for slowing down.

 

[powertoold]

Good observation, I did not notice that. I wonder if that is why the vision-only cars are limited to 75 mph; the vision algorithm can't reliably detect obstructions far enough away to drive at a higher speed?

There aren't many vision data examples of cars going 90mph and then slowing to a complete stop for a stationary vehicle within the lane of travel. It's possible Tesla is still collecting and/or validating this rare case. Granted, even with my radar car, using AP to go from 90mph to a stop wouldn't be fun.

 

[mikes_fsd]

Just based on the blue line graph, it doesn't look like there was a confident "lock" and slow down until ~150m. But ya, it seems the car did see the trailer before 150m. Again, this all could be because the trailer wasn't fully within the lane, so the confidence is lower as to whether it's relevant for slowing down.

Agreed, it still needs to get better,
But this is the acid test, white trailer in pure sunlight (looks like ~noon) and Vision only still did better than the radar stack.

 

[greenmore]

Just to repeat myself. At no point did Karpathy say that it would be impossible for radar+vision to work well with each other.
He said that the version of the radar stack that they have is not good enough to do it, but they could make it better by doing a proper hyper parameter sweep. However that would take a lot of resources so they decided to just go with full vision in anticipation that it will eventually work more robustly than their current radar+vision approach (considering their recent improvements in vision only ranging).
He's trusting his gut in this and I don't mind that. He's a smart guy and I'm sure his intuition is better than anyone's.
However, even with a perfect vision system radar still would provide information that vision can't. If you really want a superhuman autopilot, you might want to cover those edge cases with radar that vision can't figure out for you.

Edit: I kinda have a felling they never really invested much resources in radar signal processing and their current methods are just not working very well.

 

[dgatwood]

How convenient... this is what excites engineers... seeing your solution making a difference in real life!

Here's hoping none of the engineers are secretly homicidal.

 

[mikes_fsd]

I kinda have a felling they never really invested much resources in radar signal processing and their current methods are just not working very well.

Disagree for two important reasons.

First - the Florida AP1 death of Brown and the subsequent - "seeing the world in radar" post.

Second - green had posted evidence of them not only testing with high def - 4D - radar, but also the fact that they have their own interval radar hardware developed. You don't develop radar hardware without investing resources into signal processing.

For some reason people want to assume that Tesla is lazy / skipping steps without any proof to back up that assumption.

 

[rxlawdude]

For some reason people want to assume that Tesla is lazy / skipping steps without any proof to back up that assumption.

I wonder what that reason might be... With a 10% jump in two days, I bet the boo birds will be singing their tunes of doom and gloom in 3... 2... 1.

 

[greenmore]

For some reason people want to assume that Tesla is lazy / skipping steps without any proof to back up that assumption.

Direct quote from Karpathy:

"And of course, you could go into the radar stack and you could adjust the hyper parameters of the tracker, like why is it dropping tracks and so on, but then you are spending engineering efforts and focus on a stack that is not really barking up the right tree."

His justification for not investigating fully: it's "not really barking up the right tree".

He had no such concerns about the vision-based ranging method dropping tracks and so on, although it was clear from his figures that vision was doing that too. The difference is that they are deeply committed to improving vision (which I'm all for) and so engineering resources will be easier to find for those efforts.

 

[mikes_fsd]

Direct quote from Karpathy:

"And of course, you could go into the radar stack and you could adjust the hyper parameters of the tracker, like why is it dropping tracks and so on, but then you are spending engineering efforts and focus on a stack that is not really barking up the right tree."

His justification for not investigating fully: it's "not really barking up the right tree".

He had no such concerns about the vision-based ranging method dropping tracks and so on, although it was clear from his figures that vision was doing that too. The difference is that they are deeply committed to improving vision (which I'm all for) and so engineering resources will be easier to find for those efforts.

You're missing the forest for the trees.

The vision one, is shown to be beating the radar by 35 meters (115 ft) sooner. (with the consistent track for the object in question)

That extra time, gives the car options (slow down more gently, check if you can pass safely --- as an alternative to slamming on the brakes).
The interrupted/"dropped" tracking in blue is even earlier at 180 m (that is 70 meters - 230ft - sooner than the radar) having it be consistent from 180 m down would be awesome, but to dismiss the better consistent performance as "it's just like radar was shown to lose track in previous slide" is disingenuous... here is why:

The radar tracking loss on the previous slide of the presentation... note how the tracking is lost through the braking curve?
So, if vision as show above beats radar in distance/time of detection and consistent tracking of objects and their depth/distance and velocity, then you would still win on using vision.

Can you see the difference? at 180 m the radar didn't even have a lock on the object (i.e. it didn't even know something was there.)

 

[mikes_fsd]

He had no such concerns about the vision-based ranging method dropping tracks and so on,

Another way to put it.
Karpathy (and Tesla) is trying to show that vision is better than radar, those are the parameters.
They are not aiming to prove some other hypothetical parameter (i.e. at 180 m vision loses tracking) well the radar is only rated at 160m and only picked up the object/trailer at 110m in the real life example we are shown.

 

[gearchruncher]

I wonder what that reason might be... With a 10% jump in two days, I bet the boo birds will be singing their tunes of doom and gloom in 3... 2... 1.

If every "boo bird" is a short, I'll go ahead and assume every blind optimist is a shill as well, and that all of us here are just arguing because we have financial reasons to and facts don't matter.

 

[rxlawdude]

If every "boo bird" is a short, I'll go ahead and assume every blind optimist is a shill as well, and that all of us here are just arguing because we have financial reasons to and facts don't matter.

Again, if one touts one company's product (not the one this site is about), one's objectivity must be questioned. You're the Tesla "guilty unless proven innocent type," while touting Waymo as "innocent unless proven guilty." You can't even accept the fact that Waymo can't negotiate construction cones.

Not symmetric at all. I get the frustration over Tesla's incessant delays. I don't get the spewing of supposed internals and test status of Tesla's FSD as evidence. Especially while one spews happy time thoughts about another technology (Waymo) that we know Tesla will not be employing. Yet it gets compared. Over and over and over.

 

[gearchruncher]

if one touts one company's product (not the one this site is about), one's objectivity must be questioned. You're the Tesla "guilty unless proven innocent type," while touting Waymo as "innocent unless proven guilty." You can't even accept the fact that Waymo can't negotiate construction cones.

As has happened multiple times, you are much more interested in the messenger than the message. Because someone discusses another leader in autonomy while we're having discussions about autonomy in general, their motives must be questioned!? You're looking for any excuse to ignore a message you don't like. You're additionally all over the place- first it's Tesla shorts, referencing Tesla share price, and now it's anyone that discusses Waymo?

You're the Tesla "guilty unless proven innocent type," while touting Waymo as "innocent unless proven guilty." You can't even accept the fact that Waymo can't negotiate construction cones.

My post history says no such thing. I personally consider Waymo when people discuss how well Tesla is doing and how their process is clearly going to be superior, while there are examples of other companies doing things differently and being ahead in certain ways (like already being at L4), as a counterpoint to Tesla's obvious superiority. But overall, I have no idea who will "win" and get to generalized L4 first:

Of course, all of this can change with one software update, as it can with any software program. Tesla can escape vaporware status at any instant. Until they do though, this is pretty classic vaporware (with the added "pay in advance" multiplier). Some companies exit vaporware with a great product and still dominate. Some never deliver and fade away. Nobody knows which one Tesla will be any more than Mobileye, Zoox, Cruise, Waymo, Huawei....

You can't even accept the fact that Waymo can't negotiate construction cones.

I never said that, and you bring this up while also saying comparisons to Waymo are irrelevant and it's others that constantly compare to Waymo. But while we're throwing things out, Tesla can't negotiate construction zones either, and many Tesla optimists can't accept the fact that Tesla removed Radar for money / supply chain reasons, not because they are sure that vision will work better and have already proved that. That seems pretty biased too, their objectivity must be questioned like you said, and it is easy to just assume financial bias like you did, so I'll go with paid shills.

 

[emmz0r]

While they have been experimenting with this, they should have employed a team dedicated to correct wrong speed limits, etc, and really make the best of what they have on the old stack. Autopilot and NoA could have been such a good experience on the road to FSD, if they had a dedicated team to maintain and fix issues

 

[ZeApelido]

Tesla wants to currently ignore radar because they want to "move fast and break things". This is not some dumb approach that is reckless - it is an attempt to end up at the optimal solution in less time than they would with the pragmatic approach. (Ironically it is the seem approach as gradient descent used in neural nets - move fast (aggressive learning rate, but not too much) and break things (sometimes error goes up)). Having to stop and "fix" the radar contribution at each iteration before moving on to the next vision iteration may significantly slow down the development cycle. It may be more beneficial to iterate 100 times on vision and then update the radar contribution, because the vision updates could happen much faster and thus the net error reduction / month would actually be more significant.

After some time, radar signals could be reintroduced to the perception stack to see how much they contribute. Though it seems Tesla is betting that the camera vision stack will get to a point where radar contribution is insignificant.

 

[alsetym]

Tesla wants to currently ignore radar because they want to "move fast and break things". This is not some dumb approach that is reckless - it is an attempt to end up at the optimal solution in less time than they would with the pragmatic approach. (Ironically it is the seem approach as gradient descent used in neural nets - move fast (aggressive learning rate, but not too much) and break things (sometimes error goes up)). Having to stop and "fix" the radar contribution at each iteration before moving on to the next vision iteration may significantly slow down the development cycle. It may be more beneficial to iterate 100 times on vision and then update the radar contribution, because the vision updates could happen much faster and thus the net error reduction / month would actually be more significant.

After some time, radar signals could be reintroduced to the perception stack to see how much they contribute. Though it seems Tesla is betting that the camera vision stack will get to a point where radar contribution is insignificant.

The only thing I might disagree with there is they are unlikely to add radar back, simply to save face and the cost of installing radar in radardless cars. Their language is pretty clear, radar is gone. For better or worse. Much more likely would be replacing a forward looking camera to fill any gap they may have, something with better low light performance or further reach or whatever.