Welcome to Tesla Motors Club
Discuss Tesla's Model S, Model 3, Model X, Model Y, Cybertruck, Roadster and More.
Register

Seeing the world in Autopilot V9 (part three?)

verygreen

Curious member
Jan 16, 2017
2,928
11,352
TN
Well, as the great (I think) v8.1 autopilot footage was released it was a bit marred by a quickly followed v9 update where Tesla improved their offering quite a bit (took away some of the new thingies since then).

More importantly, they started using all the cameras and that created quite some problems in how to capture those esp. without Tesla actually cooperating.

The solution I chose was to just limit framerate on all the cameras but main one (the storage driver on ape can handle only at most 80MB/sec). 9fps is used for the side cameras and 6 fps for fisheye (the car actually gets 36fps feeds from all cameras bu backup where it gets 30fps) and even that tends to overwhelm the drive from time to time and this is visible in still frames here and there.
One of the problem is the backup camera is actually quite a bit unreliable and in many runs there's no output captured from it. As such I decided not to collect it at all for now. (you know it's not working on your cam when on a trip the cars come behind your car real close at a traffic light and nothing shows on the IC, surprisingly the CID backup cam display still works, so Tesla decided to just paper over the old "freeze frame" issue but not hte autopilot problem of the same).

It's also notable that different cameras get different detection rates it appears and since I cannot predict which ones are which sometimes detections seem to be a little bit off - know it's most likely a sampling artifact.

@DamianXVI was hard at work meanwhile improving visualizations - we now get some basic car state, 3D bounding boxes and other smaller things.

The video combines 6 cams and the narrow while captured is not shown simply because there's no space for it (as is we are already almost at 4K with native resolutions of all cams used and one more row would bring us over). If you have any cool ideas about how to visualize this differently and fit more cams in - let me know and I can provide individual (time-synced) streams of the 7 cameras.
Keep in mind all detections are independently run on every camera with little regards to other cams so you cannot just stitch them all together into 360-like video unfortunately.

Anyway here's the video, it includes different traffic modes in a North-American megapolis, though luckily no traffic jams, try not to go too cross-eyed when trying to watch 6 video-feeds at the same time:


Also another important question that I am sure is on everybody's mind is "how does the pothole/animals/othe road hazards" are handled by autopilot. Well the bad news is it's still nto handled at all as is shown in the video below. The quality is somewhat worse than the premier video above because I did not get my capture scripts to a good enough shape by then, but I don't run into tires in the middle of highways every day, so cannot really choose from multiple options.



I have a couple more of processed clips, but they are similar in nature to the first video and I am not sure if there's any interest in publishing that?

Post whatever interesting timelines you see.
 

verygreen

Curious member
Jan 16, 2017
2,928
11,352
TN
well, there's certainly a big improvement in pedestrian detections on the side cams! Would come handy since those pedestrians can approach you from any direction!

btw pedestrians also have 3D bounding box, but it's not being displayed for now. Details are being worked out by the usual suspect.
 

J1mbo

Active Member
Aug 20, 2013
1,593
1,386
UK
well, there's certainly a big improvement in pedestrian detections on the side cams! Would come handy since those pedestrians can approach you from any direction!

Excellent work! Seems that the pillars have issues with drivable space determination which we don't see in the other cams.

Also, we can add white jackets & chinos to the "avoid wearing this if you don't want to be run over" list (from 0:57s)! :)
 

Bladerskb

Senior Software Engineer
Oct 24, 2016
2,279
2,582
Michigan
Semantic Free Space detection on everything other than the main cameras is quite bad, so is the car detection.

Atleast they have 3DVD, although not as accurate as i would like.

If funny how when i mentioned 3DVD all Tesla fans jumped down my throat and said if Tesla doesn't have it it means its irrelevant "marketing gimmick" as though Tesla is the judge and jury on SDC. Now Tesla has it, its "omg its amazing". Shows you how alot of people don't have a barometer of truth, same as the Lidar situation.

I'm starting to actually believe i'm never ever gonna be wrong about anything tesla lol

giphy.webp


In conclusion. They are finally catching up, but still got a long long long way to go before even sniffing Mobileye.
 
Last edited:

Mo City

Active Member
Jul 17, 2016
1,910
11,670
near Houston
Question for you @verygreen

Do the views of the rear facing side cameras intersect behind the vehicle? If so, how far behind? I ask because whenever it rains, the rear camera becomes virtually worthless. Seems like the side cameras must intersect at some point to ensure a 360-view is maintained at all times.

There should be a rear facing radar IMO.
 

verygreen

Curious member
Jan 16, 2017
2,928
11,352
TN
Question for you @verygreen

Do the views of the rear facing side cameras intersect behind the vehicle? If so, how far behind? I ask because whenever it rains, the rear camera becomes virtually worthless. Seems like the side cameras must intersect at some point to ensure a 360-view is maintained at all times.

There should be a rear facing radar IMO.
I don't think repeater cams intersect as is evidenced by the video.

the rear cam is a lot more problematic even without rain since it often just stops working (possibly fixable in software, but nobody bothered yet for whatever reason).
 

pkodali

Member
Oct 22, 2017
824
554
Sacramento
I don't think repeater cams intersect as is evidenced by the video.

the rear cam is a lot more problematic even without rain since it often just stops working (possibly fixable in software, but nobody bothered yet for whatever reason).
Is the backup camera problem also present in AP 2.5?
 

verygreen

Curious member
Jan 16, 2017
2,928
11,352
TN
Atleast they have 3DVD, although not as accurate as i would like.
I believe the 3DVD is just a bolt-on thing to appease critics and they track this stuff internally by other means.

This is evidenced by the thing only appearing in 18.39.6 where as 18.39.0.1 has the same NN and was not exposing anything like that.
 
  • Like
Reactions: Anner J. Bonilla

verygreen

Curious member
Jan 16, 2017
2,928
11,352
TN
Is the backup camera problem also present in AP 2.5?
Yes, this is hw2.5. It's less visible on hw2.5 because there's a hardware splitter so cid can get direct feed even when autopilot loses the signal. On hw2.0 autopilot repeats the signal to cid in software it looks like so it's a lot more visible when ther are problems.

But even then there's disconnect since sometimes software repeating works and yet internally no feed is received. Tesla programmers are amazing like that!
 

pkodali

Member
Oct 22, 2017
824
554
Sacramento
Yes, this is hw2.5. It's less visible on hw2.5 because there's a hardware splitter so cid can get direct feed even when autopilot loses the signal. On hw2.0 autopilot repeats the signal to cid in software it looks like so it's a lot more visible when ther are problems.

But even then there's disconnect since sometimes software repeating works and yet internally no feed is received. Tesla programmers are amazing like that!
Any idea why it happens?
 

Bladerskb

Senior Software Engineer
Oct 24, 2016
2,279
2,582
Michigan
I believe the 3DVD is just a bolt-on thing to appease critics and they track this stuff internally by other means.

This is evidenced by the thing only appearing in 18.39.6 where as 18.39.0.1 has the same NN and was not exposing anything like that.


3DVD is only useful for FSD related tasks so it makes sense. Tasks mobileye's amon already detailed and I already outlined.

Also another important question that I am sure is on everybody's mind is "how does the pothole/animals/othe road hazards" are handled by autopilot. Well the bad news is it's still nto handled at all as is shown in the video below. The quality is somewhat worse than the premier video above because I did not get my capture scripts to a good enough shape by then, but I don't run into tires in the middle of highways every day, so cannot really choose from multiple options.

This is pretty interesting and disappointing if true, looks like they are farther back when i thought. Even eyeq3 had a network that detected potholes, animals and road debris. Infact Audi is using it for Automatic Active Suspension adjustment.

Also is road barriers (railings, etc) also not detected? wow. that's really far behind.
 

verygreen

Curious member
Jan 16, 2017
2,928
11,352
TN
Also is road barriers (railings, etc) also not detected? wow. that's really far behind.
well, things like trafffic cones and such are definitely limiting the driveable space. But debris apparently don't. I saw bigger dogs detected as motorcycles, but not smaller ones like that tire where it's only "noticed" when it's too late.

I also have interesting bit from walmart parking lot where the car is somewhat confused by the long daisychained shopping carts "train".
 

Mo City

Active Member
Jul 17, 2016
1,910
11,670
near Houston
I don't think repeater cams intersect as is evidenced by the video.

the rear cam is a lot more problematic even without rain since it often just stops working (possibly fixable in software, but nobody bothered yet for whatever reason).
More than anything else, the lack of rear radar and the poor rear camera is why I believe Level 5 autonomy is not achievable with current sensor hardware.
 

About Us

Formed in 2006, Tesla Motors Club (TMC) was the first independent online Tesla community. Today it remains the largest and most dynamic community of Tesla enthusiasts. Learn more.

Do you value your experience at TMC? Consider becoming a Supporting Member of Tesla Motors Club. As a thank you for your contribution, you'll get nearly no ads in the Community and Groups sections. Additional perks are available depending on the level of contribution. Please visit the Account Upgrades page for more details.


SUPPORT TMC
Top