New feature: Autosteer Stop Light Warning

[diplomat33]

Cool the fanboy spin. It's not close to reliable but it's nice to see that they are actually working on it and could deliver something in like 3 months maybe, 6 months definitely.

Yeah, I am at least pleased that we are finally seeing some progress on this feature.

 

[Bladerskb]

Yes, I watched the video. Let me walk you through it:

Again, not perfect. Get's 12 out of 15 right to my count. That's 80% right. Not good enough for full public release which is why it is still in early access. But for an alpha, early access, feature, it looks pretty good.

That's your idea of reliable?

By the way you missed:
0:07 mark: misidentifies traffic light as green
0:10 mark: misidentifies traffic light as green AGAIN
3:55 - 4:50 : misidentifies red traffic light as green SEVEN TIMES in a row
6:32: misidentifies traffic light as red momentarily
7:28 very late green light identification

RELIABLE?

 

[mongo]

That's your idea of reliable?

By the way you missed:
0:7 mark: misidentifies traffic light as green
0:10 mark: misidentifies traffic light as green AGAIN
3:55 - 4:50 : misidentifies red traffic light as green SEVEN TIMES in a row
6:32: misidentifies traffic light as red momentarily
7:28 very late green light identification

RELIABLE?

Wow, first release of beta feature is not 100% perfect, whoda thunk it???
So eleven more events yields 12 out of 26 (though 7 issues with the same light maybe should count as one). Slightly under 50% and still infinity percent better than other cars...

 

[croman]

Well, I wouldn't make comparisons. Doing something really badly might be worse than not doing it at all. That being said, its dev so its not even worth doing much other than tracking its improvement and not absolute strength. Its not worth actual testing until its 99.9%+.

Also, its not worth mentioning EyeQ4 for the same reason (its currently doing nothing, so maybe it could be better but why even mention it like its available and commercially sound).

 

[diplomat33]

That's your idea of reliable?

By the way you missed:
0:07 mark: misidentifies traffic light as green
0:10 mark: misidentifies traffic light as green AGAIN
3:55 - 4:50 : misidentifies red traffic light as green SEVEN TIMES in a row
6:32: misidentifies traffic light as red momentarily
7:28 very late green light identification

RELIABLE?

I missed 0:07 or 0:10 but they don't matter because the car was too far away. Misidentifying a traffic light way before you need to stop does not matter. what matters is identifying the traffic light when it is time to stop at either a red or go through on green. You will notice that the car DID correctly identify the light as red 1 second later (0:11) when it got closer and it mattered.

3:55-4:50 that only counts once since it was clearly the same bug, not 7 different bugs. Also, the car DID correctly identify the light as green when it was time to go again.

Yes, I missed 6:32 and 7:28. Although 7:28 is not a big deal since it DID correctly identity the light as green when it needed to.

So if we look at every single instance of identifying a light, the success rate is about 50-60%. If we only look at the instances that actually matter for driving, the success rate is closer to 70-80%.

Again, I am not arguing that the software is perfect. It is clearly far from it. But I think you are missing the point that we are seeing the feature in development mode. All development software is buggy. That's why Tesla released "autosteer stop light warning" the way they did as a L2 driver assist that just warns about running a red light, instead of releasing a full fledged "car will stop at red lights" feature. Tesla knows that it is not reliable yet to be used as a "FSD" feature. But Tesla will of course work out the bugs as they collect data from "autosteer stop light warning".

 

[S4WRXTTCS]

That's your idea of reliable?

By the way you missed:
0:07 mark: misidentifies traffic light as green
0:10 mark: misidentifies traffic light as green AGAIN
3:55 - 4:50 : misidentifies red traffic light as green SEVEN TIMES in a row
6:32: misidentifies traffic light as red momentarily
7:28 very late green light identification

RELIABLE?

Why are we talking about reliability of an unreleased function on hardware we know is obsolete, and isn't intended for this?

We know they have to greatly reduce the image size before it goes to the Neural network inference. Obviously this will impact the accuracy of it. We also know the neural network isn't the same one that's intended for HW3.

Now I do see validity in talking about the accuracy of the RELEASED red light warning. Something where it appears like it uses the same detection, but they only take the most probable of occurrences. One could argue that the false negatives are so high that it defeats the purpose of even releasing it. I personally think it shouldn't have been released the way they did. It wasn't clear that it wouldn't be immediately active, and there was no ability to adjust it's warning time. Or at least have it follow the FCW time.

 

[Bladerskb]

Why are we talking about reliability of an unreleased function on hardware we know is obsolete, and isn't intended for this?

Because @diplomat33 said it was "pretty reliable already"....
Haven't you thought about it though, after the whole drums of "billions of miles of data", this is it?

We know they have to greatly reduce the image size before it goes to the Neural network inference. Obviously this will impact the accuracy of it. We also know the neural network isn't the same one that's intended for HW3.

According to @verygreen its taking full resolution now.

We also know the neural network isn't the same one that's intended for HW3.

HW3 NN is a myth. All HW3 cars come with the same AP2.5 NN. If there were some secretive uber level NN. You would think they would be pumping it out right out the gate.

 

[Bladerskb]

So if we look at every single instance of identifying a light, the success rate is about 50-60%. If we only look at the instances that actually matter for driving, the success rate is closer to 70-80%.

Again, I am not arguing that the software is perfect. It is clearly far from it. But I think you are missing the point that we are seeing the feature in development mode. All development software is buggy.

I already applauded Tesla for bringing industry standard tech to production. The underlying tech behind enhanced summon and their traffic light/sign feature has existed for a long time. Startups are leading the charge. Automakers are slow so you won't see it in production anytime soon even though traffic light/sign recognition for L3/L4 cars has been ready since late 2017 (EyeQ4). But don't act like Tesla are trailblazers here. They are forerunners in deployment.

This begs the question. After the "billions of miles of data" drum was beat so hard i had to get surgery on my ear drums... yet all we get is this?

 

[diplomat33]

HW3 NN is a myth. All HW3 cars come with the same AP2.5 NN. If there were some secretive uber level NN. You would think they would be pumping it out right out the gate.

This is pure FUD.

This begs the question. After the "billions of miles of data" drum was beat so hard i had to get surgery on my ear drums and yet all we get is this?

Uh?! The billlions of miles of data were used for more than just traffic light and stop detection. Tesla used the billion of miles collected so far to give us all of EAP and NOA. All of EAP and all of NOA and traffic light and stop sign detection is all thanks to the billions of miles of data. So I would say all that data has given us a lot actually. And as Tesla collects more data, the current and future features will continue to get better.

 

[mongo]

HW3 NN is a myth. All HW3 cars come with the same AP2.5 NN. If there were some secretive uber level NN. You would think they would be pumping it out right out the gate.

??? Really, you think Tesla doesn't actually have a NN that requires the processing power of HW3? And you are basing that on not releasing it the moment that a few customers get HW3 cars? Do you comprehend the floodgate of upgrade requests that will be unleashed once a larger than 2.x HW NN is released to the public?

According to @verygreen its taking full resolution now.

Where'd they say that? Very impressive if true, or is that only a subset of cameras?

 

[S4WRXTTCS]

Because @diplomat33 said it was "pretty reliable already"....
Haven't you thought about it though, after the whole drums of "billions of miles of data", this is it?

But, you know that there is no relevancy of those billions of miles.

You know that it requires labeled datasets, and the quality of it is going to be highly dependent on the capability of the hardware doing the inference. It sets how large/complex the deep neural network can be. What kind of input image sizes it can deal with, and what kind of speed it can run at.

You also know that there is a difference between a casual statement like it seems pretty reliable versus the necessary 99.99% (or whatever it might be) to actually release it.

It was certainly worthy of a joke reply. Something like "That's a a race of the 9's? More like a race of the .1's".

"According to @verygreen its taking full resolution now."

Really? This is what he just said early today or last night.
Red light detection, stop sign detection, stopping!
Post #24

As to the HW3 Neural Net. I don't believe it was reasonable to expect that the it would immediately ship out of the gate with the much bigger neural network. There was a crap ton of work to do in just getting all the existing features running on it. It wasn't just adding a custom neural net accelerator, but also the SOC's that bring the images into the system.

I'm amazed that they shipped it as well functional as it is. Where they didn't disable AEB for "qualification" like they did when AP2 first shipped. To my knowledge it's only missing things like dashcam, and Sentry mode.

 

[S4WRXTTCS]

But don't act like Tesla are trailblazers here. They are forerunners in deployment.

I can't speak for anyone else, but that's exactly why I'm here.

Various vendors can make all the claims in the world, but the real proof is it running on my car.

To use it on a daily basis, and to see how well it does.

It's what really matters because I absolutely know automotive companies will ruin it by some asinine decision. Like the BMW X5 was exciting until I learned that the hands free eye tracking thing was only good up to 37mph.

Not that I would have gotten it, but it would have set precedence. Cadillac isn't a strong enough player to push the industry.

 

[mongo]

But, you know that there is no relevancy of those billions of miles.

Unless Tesla has a way to capture them and use them as test/ training data. If HW3 has dual processors for safety in FSD mode, but only uses one in driver monitored AP, then the other could be doing some interesting things.
Just adding a campaign of AP disengagements (with or without running in background) would allow rapid validation testing.

Just talking potential here based on the number of miles/ day.

 

[diplomat33]

HW3 NN is a myth. All HW3 cars come with the same AP2.5 NN. If there were some secretive uber level NN. You would think they would be pumping it out right out the gate.

Looks like Elon has answered your question. HW3 NN is not a myth. Tesla is just not there yet. When the software gets there, then Tesla will retrofit cars with AP3.

 

[Bladerskb]

Looks like Elon has answered your question. HW3 NN is not a myth. Tesla is just not there yet. When the software gets there, then Tesla will retrofit cars with AP3.


View attachment 391724

No you just proved my point that the *magical* HW3 NN doesn't exist and won't for a while. Even when it does exist we don't know how much "meanful advantage" it will have over AP2.

 

[diplomat33]

No you just proved my point that the *magical* HW3 NN doesn't exist and won't for a while. Even when it does exist we don't know how much "meanful advantage" it will have over AP2.

I am confident HW3 software will come. It's like the base Model 3. People said it was a myth because it did not exist yet but it eventually came.

 

[iTech]

I am confident HW3 software will come. It's like the base Model 3. People said it was a myth because it did not exist yet but it eventually came.

Unlimited time isn't a thing for public companies

 

[electronblue]

Why are we talking about reliability of an unreleased function on hardware we know is obsolete, and isn't intended for this?

Because historically speaking it probably is pretty representative of what we’re going to get?

We have heard your line before. Too many times.

I think with Tesla Autopilot 2+ an important thing to remember is history has shown development versions with their issues are pretty close to what will get released. There has been no significant giant leap forward that would have surprised us thusfar, instead a slower than expected pace with two steps forward and more steps backwards than expected.

Tesla tends to release their stuff very early in their development cycle compared to the rest of the industry. There has been no magical second codebase brewing in a secret basement so far, no matter how much it has been speculated over and wished for.

Already Musk is saying HW3 will initially be at a slight disadvantage over HW2. That would be very much in line with history where AP2 and AP2.5 (people tend to forget this one) were regressions over what came first initially and then over time things develop from there.

I have no doubt Tesla can run higher resolutions and better versions of their networks on HW3 over time. But I do think it is wishful thinking if we are expecting that to cause a giant leap in performance. More likely it will just set the stage for the development to continue to bigger and better things over longer periods of time.

 

[S4WRXTTCS]

Because historically speaking it probably is pretty representative of what we’re going to get?

We have heard your line before. Too many times.

I think with Tesla Autopilot 2+ an important thing to remember is history has shown development versions with their issues are pretty close to what will get released. There has been no significant giant leap forward that would have surprised us thusfar, instead a slower than expected pace with two steps forward and more steps backwards than expected.

Tesla tends to release their stuff very early in their development cycle compared to the rest of the industry. There has been no magical second codebase brewing in a secret basement so far, no matter how much it has been speculated over and wished for.

Already Musk is saying HW3 will initially be at a slight disadvantage over HW2. That would be very much in line with history where AP2 and AP2.5 (people tend to forget this one) were regressions over what came first initially and then over time things develop from there.

I have no doubt Tesla can run higher resolutions and better versions of their networks on HW3 over time. But I do think it is wishful thinking if we are expecting that to cause a giant leap in performance. More likely it will just set the stage for the development to continue to bigger and better things over longer periods of time.

What disadvantages are we aware of with HW3 versus HW2?

Dashcam mode? No, HW2 doesn't have it
Sentry Mode? No, HW2 doesn't have it

Musky also said HW2+ cars would get Sentry mode.

I also fail to see why a much better neural network would be a significant leap forward. nI'm not expecting a significant leap forwards for NoA, Enhanced Summons, etc.

There are very real sensory limitations that prevent the car from doing a whole lot. I certainly expect incremental improvements over time. Where there is HW3.5, HW4, etc.

I did get a chuckle out of how Elon is trying to rename HW3 as the FSD computer.

But, stop light detection? REALLY? That's a giant leap?

If Karpathy can't do Stop light detection then he really needs to find a different career. Maybe take up knitting.

 

[electronblue]

@S4WRXTTCS

My only point is that historically we have been able to judge Tesla by what we can glimpse from their development versions — there is no giant leap between that and what eventually gets released, in fact there is often a disappointment.

Hence I didn’t consider your implication accurate that judging stop light performance on AP2/2.5 would not be usefully representative of how it will fare on HW3 in the short term as well.

Why are we talking about reliability of an unreleased function on hardware we know is obsolete, and isn't intended for this?

Because it is our best insight into what Tesla has cooking — and likely is quite representative of the quality what they have released and will release.

In fact I’m hardpressed to think of any instance where Tesla has actually positively surprised us with a new release or product. The Roadster is the exception to that rule as it pretty much came out of nowhere. Other than that, we pretty much always know the level of which Tesla releases will be and our hopes and dreams tend to be way too optimistic.

With Tesla what you see is what you get, probably a bit worse than that. This will develop over time, but not with a giant leap.