TMC is an independent, primarily volunteer organization that relies on ad revenue to cover its operating costs. Please consider whitelisting TMC on your ad blocker and becoming a Supporting Member. For more info: Support TMC

Seeing the world in Autopilot V9 (part three?)

Discussion in 'Autonomous Vehicles' started by verygreen, Oct 23, 2018.

  1. verygreen

    verygreen Curious member

    Joined:
    Jan 16, 2017
    Messages:
    2,031
    Location:
    TN
    Well, as the great (I think) v8.1 autopilot footage was released it was a bit marred by a quickly followed v9 update where Tesla improved their offering quite a bit (took away some of the new thingies since then).

    More importantly, they started using all the cameras and that created quite some problems in how to capture those esp. without Tesla actually cooperating.

    The solution I chose was to just limit framerate on all the cameras but main one (the storage driver on ape can handle only at most 80MB/sec). 9fps is used for the side cameras and 6 fps for fisheye (the car actually gets 36fps feeds from all cameras bu backup where it gets 30fps) and even that tends to overwhelm the drive from time to time and this is visible in still frames here and there.
    One of the problem is the backup camera is actually quite a bit unreliable and in many runs there's no output captured from it. As such I decided not to collect it at all for now. (you know it's not working on your cam when on a trip the cars come behind your car real close at a traffic light and nothing shows on the IC, surprisingly the CID backup cam display still works, so Tesla decided to just paper over the old "freeze frame" issue but not hte autopilot problem of the same).

    It's also notable that different cameras get different detection rates it appears and since I cannot predict which ones are which sometimes detections seem to be a little bit off - know it's most likely a sampling artifact.

    @DamianXVI was hard at work meanwhile improving visualizations - we now get some basic car state, 3D bounding boxes and other smaller things.

    The video combines 6 cams and the narrow while captured is not shown simply because there's no space for it (as is we are already almost at 4K with native resolutions of all cams used and one more row would bring us over). If you have any cool ideas about how to visualize this differently and fit more cams in - let me know and I can provide individual (time-synced) streams of the 7 cameras.
    Keep in mind all detections are independently run on every camera with little regards to other cams so you cannot just stitch them all together into 360-like video unfortunately.

    Anyway here's the video, it includes different traffic modes in a North-American megapolis, though luckily no traffic jams, try not to go too cross-eyed when trying to watch 6 video-feeds at the same time:



    Also another important question that I am sure is on everybody's mind is "how does the pothole/animals/othe road hazards" are handled by autopilot. Well the bad news is it's still nto handled at all as is shown in the video below. The quality is somewhat worse than the premier video above because I did not get my capture scripts to a good enough shape by then, but I don't run into tires in the middle of highways every day, so cannot really choose from multiple options.




    I have a couple more of processed clips, but they are similar in nature to the first video and I am not sure if there's any interest in publishing that?

    Post whatever interesting timelines you see.
     
    • Love x 21
    • Informative x 13
    • Like x 8
  2. pkodali

    pkodali Banned

    Joined:
    Oct 22, 2017
    Messages:
    817
    Location:
    Sacramento
    I've been waiting on this for so long. Thank you so much for posting this!
     
  3. verygreen

    verygreen Curious member

    Joined:
    Jan 16, 2017
    Messages:
    2,031
    Location:
    TN
    #3 verygreen, Oct 23, 2018
    Last edited: Oct 23, 2018
    Do watch the videos in 4K if you can, btw. (damn th 10 minute edit limit!)
     
  4. Mo City

    Mo City Member

    Joined:
    Jul 17, 2016
    Messages:
    228
    Location:
    USA
    Spectacular!
     
  5. BigD0g

    BigD0g Active Member

    Joined:
    Jan 12, 2017
    Messages:
    1,653
    Location:
    Somewhere
    Please don't watch if your prone to seizures.... Epic video @verygreen do you notice any improvements in the pedestrian detection? I see in the epic video that a few poor souls would be ran over like Jarvis going for a squirrel
     
    • Like x 1
    • Funny x 1
  6. verygreen

    verygreen Curious member

    Joined:
    Jan 16, 2017
    Messages:
    2,031
    Location:
    TN
    well, there's certainly a big improvement in pedestrian detections on the side cams! Would come handy since those pedestrians can approach you from any direction!

    btw pedestrians also have 3D bounding box, but it's not being displayed for now. Details are being worked out by the usual suspect.
     
    • Love x 3
    • Like x 1
  7. J1mbo

    J1mbo Active Member

    Joined:
    Aug 20, 2013
    Messages:
    1,314
    Location:
    UK
    Excellent work! Seems that the pillars have issues with drivable space determination which we don't see in the other cams.

    Also, we can add white jackets & chinos to the "avoid wearing this if you don't want to be run over" list (from 0:57s)! :)
     
    • Funny x 4
  8. gluu

    gluu Member

    Joined:
    Jan 18, 2018
    Messages:
    49
    Location:
    Chicago
    Awesome work! Do you have any idea if v9 is generating maps like Mobileye REM?
     
    • Like x 1
  9. EvanLin

    EvanLin Member

    Joined:
    Oct 9, 2016
    Messages:
    65
    Location:
    Asia
    Awesome work!
    @3:36, AP seems to draw a left lane in main camera even before camera can see it. Any idea?
     
  10. Bladerskb

    Bladerskb Like how many times do i have to be right?

    Joined:
    Oct 24, 2016
    Messages:
    1,254
    Location:
    Michigan
    #10 Bladerskb, Oct 23, 2018
    Last edited: Oct 23, 2018
    Semantic Free Space detection on everything other than the main cameras is quite bad, so is the car detection.

    Atleast they have 3DVD, although not as accurate as i would like.

    If funny how when i mentioned 3DVD all Tesla fans jumped down my throat and said if Tesla doesn't have it it means its irrelevant "marketing gimmick" as though Tesla is the judge and jury on SDC. Now Tesla has it, its "omg its amazing". Shows you how alot of people don't have a barometer of truth, same as the Lidar situation.

    I'm starting to actually believe i'm never ever gonna be wrong about anything tesla lol

    [​IMG]

    In conclusion. They are finally catching up, but still got a long long long way to go before even sniffing Mobileye.
     
    • Disagree x 2
  11. 1375mlm

    1375mlm Member

    Joined:
    Jun 29, 2017
    Messages:
    118
    Location:
    Texas, Houston
    Do you think the new Tesla Chip (HW3) would be required for completion of EAP?
     
  12. Mo City

    Mo City Member

    Joined:
    Jul 17, 2016
    Messages:
    228
    Location:
    USA
    Question for you @verygreen

    Do the views of the rear facing side cameras intersect behind the vehicle? If so, how far behind? I ask because whenever it rains, the rear camera becomes virtually worthless. Seems like the side cameras must intersect at some point to ensure a 360-view is maintained at all times.

    There should be a rear facing radar IMO.
     
  13. verygreen

    verygreen Curious member

    Joined:
    Jan 16, 2017
    Messages:
    2,031
    Location:
    TN
    I don't think repeater cams intersect as is evidenced by the video.

    the rear cam is a lot more problematic even without rain since it often just stops working (possibly fixable in software, but nobody bothered yet for whatever reason).
     
    • Informative x 2
    • Like x 1
  14. pkodali

    pkodali Banned

    Joined:
    Oct 22, 2017
    Messages:
    817
    Location:
    Sacramento
    Is the backup camera problem also present in AP 2.5?
     
  15. verygreen

    verygreen Curious member

    Joined:
    Jan 16, 2017
    Messages:
    2,031
    Location:
    TN
    I believe the 3DVD is just a bolt-on thing to appease critics and they track this stuff internally by other means.

    This is evidenced by the thing only appearing in 18.39.6 where as 18.39.0.1 has the same NN and was not exposing anything like that.
     
    • Like x 1
  16. verygreen

    verygreen Curious member

    Joined:
    Jan 16, 2017
    Messages:
    2,031
    Location:
    TN
    Yes, this is hw2.5. It's less visible on hw2.5 because there's a hardware splitter so cid can get direct feed even when autopilot loses the signal. On hw2.0 autopilot repeats the signal to cid in software it looks like so it's a lot more visible when ther are problems.

    But even then there's disconnect since sometimes software repeating works and yet internally no feed is received. Tesla programmers are amazing like that!
     
    • Funny x 2
    • Like x 1
  17. pkodali

    pkodali Banned

    Joined:
    Oct 22, 2017
    Messages:
    817
    Location:
    Sacramento
    Any idea why it happens?
     
  18. Bladerskb

    Bladerskb Like how many times do i have to be right?

    Joined:
    Oct 24, 2016
    Messages:
    1,254
    Location:
    Michigan

    3DVD is only useful for FSD related tasks so it makes sense. Tasks mobileye's amon already detailed and I already outlined.

    This is pretty interesting and disappointing if true, looks like they are farther back when i thought. Even eyeq3 had a network that detected potholes, animals and road debris. Infact Audi is using it for Automatic Active Suspension adjustment.

    Also is road barriers (railings, etc) also not detected? wow. that's really far behind.
     
  19. verygreen

    verygreen Curious member

    Joined:
    Jan 16, 2017
    Messages:
    2,031
    Location:
    TN
    well, things like trafffic cones and such are definitely limiting the driveable space. But debris apparently don't. I saw bigger dogs detected as motorcycles, but not smaller ones like that tire where it's only "noticed" when it's too late.

    I also have interesting bit from walmart parking lot where the car is somewhat confused by the long daisychained shopping carts "train".
     
  20. Mo City

    Mo City Member

    Joined:
    Jul 17, 2016
    Messages:
    228
    Location:
    USA
    More than anything else, the lack of rear radar and the poor rear camera is why I believe Level 5 autonomy is not achievable with current sensor hardware.
     
    • Disagree x 5

Share This Page

  • About Us

    Formed in 2006, Tesla Motors Club (TMC) was the first independent online Tesla community. Today it remains the largest and most dynamic community of Tesla enthusiasts. Learn more.
  • Do you value your experience at TMC? Consider becoming a Supporting Member of Tesla Motors Club. As a thank you for your contribution, you'll get nearly no ads in the Community and Groups sections. Additional perks are available depending on the level of contribution. Please visit the Account Upgrades page for more details.


    SUPPORT TMC