Seeing the world in autopilot, part deux

[Joel]

The amount of False Positives and False Negatives in Telsa's NN at this stage of the game is alarming!

By the way you should do a three camera setup with visual overlays similar to what Zoox did.

Left Repeater | Fisheye | Right Repeater
or
Left Pillar | Main | Right Pillar

If you really want to collab shoot me a pm and i can maybe generate a 3d view using the vector info you have.

Not quite, it recognized it as the edge of the road and then as the edge of the lane. it however doesn't recognize that there is a obstacle. That's not good. What semantic free space does is color every pixel that looks like the road. I don't want a car resorting to using me as a boundary of a lane/road.

@verygreen - Are you able to display the three camera overlay as @Bladerskb suggests. I'm curious as well.

@Bladerskb - To be fair - Zoox missed a few scenarios. Would you classify any of these scenarios as "alarming?" If not, I can photo-shop the Zoox logo in the bottom right and replace it with the Tesla logo. If this was Zoox demonstration footage, I'd be interested to view the non-demo equivalent to what @verygreen is posting.

Missed Pedestrian - Ideal driving condition:

Missed Pedestrians Night:

Missed Pedestrian Unpacking Car:

Missed Pedestrian Crossing in Rain:

Two Pedestrians One Bounding Box (Missed One Pedestrian):

Missed Label:

 

[Bladerskb]

To be fair - Zoox missed a few scenarios. Would you classify any of these scenarios as "alarming?" If not, I can photo-shop the Zoox logo in the bottom right and replace it with the Tesla logo. If this was Zoox demonstration footage, I'd be interested to view the non-demo equivalent to what @verygreen is posting.

Not at all, first of all, I didn't post the Zoox vid to compare their detection but simply for their overlay format.
secondly the false negatives you listed were actually classified a split second later than the picture you took or befoe. For example the one from 3:09 and others. Again I didn't post Zoox vid because of their classification ability. The Zoox vid is poor compared to mobileye and i have addressed it before. The tesla video is even worse compared to the Zoox vid even if you try to clean it up by not displaying... say sub 15% detection.

 

[verygreen]

@verygreen - Are you able to display the three camera overlay as @Bladerskb suggests. I'm curious as well.

it's tricky for a variety of reasons I don't want to get into right now.

But I am working on it time permitting.

 

[Joel]

Not at all, first of all, I didn't post the Zoox vid to compare their detection but simply for their overlay format.
secondly the false negatives you listed were actually classified a split second later than the picture you took or befoe. For example the one from 3:09 and others. Again I didn't post Zoox vid because of their classification ability. The Zoox vid is poor compared to mobileye and i have addressed it before. The tesla video is even worse compared to the Zoox vid even if you try to clean it up by not displaying... say sub 15% detection.

You are right. The pedestrian had to be in the street for vision to sense it. I'd argue a system should track pedestrians when they enter the field of view (i.e., before they are in front of the car). Tesla and Zoox are the same in this regard. It's interesting to note that both Zoox and Tesla eventually sensed the pedestrians. Zoox even lost track of one pedestrian, but tracked the pedestrian once it moved into the intersection.

@Bladerskb - I know Tesla only identified the pedestrian as 6%, so don't yell at me. That it is a low confidence interval. But to be fair, we do not know the Zoox confidence interval.

We can agree 6% confidence interval is better than never detecting a pedestrian, right? The final example is worse than both Zoox and Tesla, right? Never identifying a pedestrian is less than 15%, right? And it's less than 6%, right? It is 0.00%

Zoox:

This is ALARMING (watch with caution):

 

[Bladerskb]

You are right. The pedestrian had to be in the street for vision to sense it. I'd argue a system should track pedestrians when they enter the field of view (i.e., before they are in front of the car). Tesla and Zoox are the same in this regard. It's interesting to note that both Zoox and Tesla eventually sensed the pedestrians. Zoox even lost track of one pedestrian, but tracked the pedestrian once it moved into the intersection.

@Bladerskb - I know Tesla only identified the pedestrian as 6%, so don't yell at me. That it is a low confidence interval. But to be fair, we do not know the Zoox confidence interval.

We can agree 6% confidence interval is better than never detecting a pedestrian, right? The final example is worse than both Zoox and Tesla, right? Never identifying a pedestrian is less than 15%, right? And it's less than 6%, right? It is 0.00%

No Zoox and Tesla are not the same. Tesla from the videos @verygreen provided has hundreds of false positives and false negative. its not even comparable and i'm talking about pedestrian in FULL VIEW.

But your rebuttal is to post an old mobileye video of pedestrians being 90% occluded most of the time in response to someone's statement that Tesla misses pedestrian in PLAIN SIGHT (with zero occlusion) during multiple frames?

Can Tesla ever do wrong according to you?

 

[Joel]

No Zoox and Tesla are not the same. Tesla from the videos @verygreen provided has hundreds of false positives and false negative. its not even comparable and i'm talking about pedestrian in FULL VIEW.

But your rebuttal is to post an old mobileye video of pedestrians being 90% occluded most of the time in response to someone's statement that Tesla misses pedestrian in PLAIN SIGHT (with zero occlusion) during multiple frames?

Can Tesla ever do wrong according to you?

That Mobileye Video was from September 2017.

Tesla can do wrong.

Mobileye can do wrong.

And @Bladerskb can do wrong. You once claimed you were never wrong. Well, today you can no longer claim that. That Mobileye video from September 2017 proves you wrong.

Unlike you, I repeatedly said all technologies have positives and negatives. None are perfect. Even Mobileye completely misses pedestrians.

Sorry, Mobileye’s technology did not identify those pedestrians. Period. Sorry to point out you were wrong and will be wrong in the future.

Do you want to explain why Mobileye completely missed those pedestrians, full stop? Or do you want to deflect to Tesla’s technology?

 

[Bladerskb]

That Mobileye Video was from September 2017.

Nope that video is from 2016 or late 2015, i remember seeing it in a presentation (Which i can't recall at the moment. Its atleast 2 years old.
Infact the video uploaded right after it is similar to it aswell. That video is from 2015 aswell but was uploaded October 2017.

And @Bladerskb can do wrong. You once claimed you were never wrong. Well, today you can no longer claim that. That Mobileye video from September 2017 proves you wrong.

I have never been wrong and my streak continues. Suck it!

Unlike you, I repeatedly said all technologies have positives and negatives. None are perfect. Even Mobileye completely misses pedestrians.

Sorry, Mobileye’s technology did not identify those pedestrians. Period. Sorry to point out you were wrong and will be wrong in the future.

Do you want to explain why Mobileye completely missed those pedestrians, full stop? Or do you want to deflect to Tesla’s technology?

Its an old video from 2015/2016 of 90% occluded pedestrian while the so called Tesla Vision from the most advanced team in the world with billions of mile data misses 0% occluded pedestrians, clear as day, in plain sight pedestrians.

 

[Joel]

Nope that video is from 2016 or late 2015, i remember seeing it in a presentation (Which i can't recall at the moment. Its atleast 2 years old.
Infact the video uploaded right after it is similar to it aswell. That video is from 2015 aswell but was uploaded October 2017.

I have never been wrong and my streak continues. Suck it!



Its an old video from 2015/2016 of 90% occluded pedestrian while the so called Tesla Vision misses 0% occluded pedestrian, clear as day, in plain sight pedestrians.

Wrong twice in one night:

 

[Bladerskb]

@Joel

Lol The video is from 2015/2016 just like the Driving Scene video is from 2015. you don't understand the concept of uploading old videos?
Infact all three videos uploaded in that span are old.

Mobileye REM Mapping Technology
Sensing the Driving Scene
Mobileye Pedestrian Detection

They are all 2-3 years old.

Exclusive: The Tesla AutoPilot - An In-Depth Look At The Technology Behind the Engineering Marvel - Page 4 of 6

You are talking to the resident Mobileye historian. lmao come on bro, try harder!

 

[Joel]

@Bladerskb Mobilye has a link to their 2017 CES presentation, which was January 2017. And they uploaded that video September 17, 2017. I know you can’t Admit you’re wrong. Let’s agree you are the opposite of right.

Just answer the question: why did Mobileye’s technology miss those pedestrians? All of them except one? Why?

 

[Bladerskb]

@Bladerskb Mobilye has a link to their 2017 CES presentation, which was January 2017. And they uploaded that video September 17, 2017. I know you can’t Admit you’re wrong. Let’s agree you are the opposite of right.

You mean to tell me you don't know how to use Youtube? And that you don't know that you can put any description you want including linking to a 2014 demonstration video called "Autonomous Car Driving with Prof..." when you upload an old video?

Why won't you admit that i'm always right and that all 3 videos are 2-3 years old? Sad!

Exclusive: The Tesla AutoPilot - An In-Depth Look At The Technology Behind the Engineering Marvel - Page 4 of 6

Tell me this, why does a neural network made by the affirmed most talent team in the world with access to billions of miles data miss non-occluded pedestrians in plain sight?

 

[Joel]

@Joel

Lol The video is from 2015/2016 just like the Driving Scene video is from 2015. you don't understand the concept of uploading old videos?
Infact all three videos uploaded in that span are old.

Mobileye REM Mapping Technology
Sensing the Driving Scene
Mobileye Pedestrian Detection

They are all 2-3 years old.

Exclusive: The Tesla AutoPilot - An In-Depth Look At The Technology Behind the Engineering Marvel - Page 4 of 6

You are talking to the resident Mobileye historian. lmao come on bro, try harder!

I need you to be wrong a third time tonight, so I’ll ask again: Why does Mobileye state “Note how Mobileye Vision Technolgy detects EVERY pedestrian as well as the direction they are walking.” That, like you, is wrong.

Please, stay calm and take a deep breath, and answer why you and that statement are both wrong. Your anger demonstrates you are wrong. We all understand that. But I would appreciate you sticking to the content as to why you are wrong. You’re smart, you can articulate it, if you want I can swap the logos to make it easier for you.

 

[Bladerskb]

@Joel

All 3 videos are 2-3 years old as evident by this 2015 article.

Exclusive: The Tesla AutoPilot - An In-Depth Look At The Technology Behind the Engineering Marvel - Page 4 of 6

I also have footage of those same videos in 2015 and 2016 presentations but i don't have time to dig them up now.
Don't feel bad Joel, you are not the only one who have attempted to fault me and failed.

 

[Cirrus MS100D]

As usual, thank you @verygreen !

It sucks Tesla feels they need to waste time trying to “protect” things from people like you and in your circle of tinkerers and I would rather they just their time doing something more productive. Hope it doesn’t create an annoying situation for you and others, but, as always, we all appreciate what you share!

 

[Joel]

@Joel

All 3 videos are 2-3 years old as evident by this 2015 article.

Exclusive: The Tesla AutoPilot - An In-Depth Look At The Technology Behind the Engineering Marvel - Page 4 of 6

I also have footage of those same videos in 2015 and 2016 presentations but i don't have time to dig them up now.
Don't feel bad Joel, you are not the only one who have attempted to fault me and failed.

None of the footage from that video is in the article you sent.

Thank you for conceding you are wrong.

Your inability to explain Mobileye’s failure to detect “EVERY pedestrian and the the direction they are walking” is very telling.

I understand this was a difficult night for you to reconcile and admit you were wrong and Mobileye’s technology is not perfect. On behalf of everyone on the forums, thank you.

 

[FluxCap]

Why is bladerskb not banned?

Bob Lutz, is that you?

 

[S4WRXTTCS]

Why is bladerskb not banned?

Bob Lutz, is that you?

He occasionally posts useful information. He did offer to help out so it shows that he might be warming up a bit to the idea of maybe not taking this so seriously, and instead using it to have fun with.

I've tried to mostly avoid this whole false positive, or false negative bit because this isn't released software. Obviously if it was released the car would be using it for something, but it's not.

Who knows what it's been tuned for or what the priority was for it.

I don't think pedestrians off to the side is really something they're concerned with. It wasn't even until V9 that they even showed a pedestrian on the IC.

With that being said I do wish Tesla would give us the option of embedding detection on the dash cam video.

This issue with detection is nothing compared to the false braking events that happen with AP2 on a regular basis.

Embedding detection or some meta data from the car on the video would be really helpful in trying to discover why the car put on the brakes.

Bladerskb would die of laughter if these false breaking events were being caused by an NN, and not by radar.

 

[Bladerskb]

Just answer the question: why did Mobileye’s technology miss those pedestrians? All of them except one? Why?

Easy if you actually watch the video in slow-mo you will know its the visual tagger threshold that is set to only visibly tag peds that are completely in full view. Those peds weren't missed even though that's an old vid. the same threshold doesn't exist for cars.

This is all visual preference.

 

[scottf200]

But right now we are going to explore the depth of their visual detection set that seems to be not without corner cases some of which are surprising to see. For example they know about pedestrians, but what if a pedestrian is doing something? Like say they are pushing a shopping cart? Well, if you are doing something like that, Tesla does not see you. They even get the radar return from the shopping card and proceed to ignore it due to no corresponding visual match. See the video below (the other part of it is a regular pedestrian on a late evening to show it's not just lack of light that plays tricks with us).

Recognized a Ped(estrian) when they were close even with a cart at 3/4 angle it seems. Also isn't green the drivable freespace and it seemed early that that ped/cart was excluded from that area.

 

[J1mbo]

Easy if you actually watch the video in slow-mo you will know its the visual tagger threshold that is set to only visibly tag peds that are completely in full view. Those peds weren't missed even though that's an old vid. the same threshold doesn't exist for cars.

This is all visual preference.

Wrong... again?

Is there someone changing the rear roadside tyre on the truck on the right?

Yes, there is!

Screencaps are from the Mobileye Sept 2017 video.

Surely only peds that are completely in full view should be tagged....?

How did Mobileye spot this pedestrian before it could see them? Suggests to me that there is enough of a time lag between the raw video feed and the classification that they have had to frig the boxing overlay to make it appear like detection is happening sooner than it really is.

Tut tut.