TMC is an independent, primarily volunteer organization that relies on ad revenue to cover its operating costs. Please consider whitelisting TMC on your ad blocker and becoming a Supporting Member. For more info: Support TMC
  1. TMC is currently READ ONLY.
    Click here for more info.

Neural Networks

Discussion in 'Autopilot & Autonomous/FSD' started by lunitiks, Nov 5, 2017.

  1. RobIII

    RobIII Member

    Joined:
    Apr 3, 2017
    Messages:
    151
    Location:
    Gemert
    • Funny x 4
    • Like x 1
  2. J1mbo

    J1mbo Active Member

    Joined:
    Aug 20, 2013
    Messages:
    1,565
    Location:
    UK
    If I understand this right, the implication here is that some of the FSD decision-making will be done by a NN which we haven't seen yet.

    If true, then we have been correct on assuming that FSD development has been going on in parallel to EAP. Very cool post, thanks!
     
  3. jimmy_d

    jimmy_d Deep Learning Dork

    Joined:
    Jan 26, 2016
    Messages:
    416
    Location:
    San Francisco, CA
    They have to be using vision. It could be an NN, but it might be a different NN than the one that I'm seeing, or it could be the same one.

    I haven't seen any functions that seem particularly tailored to dealing with the high beams but it wouldn't be hard to mix that into the other capabilities of the vision NN in a way that would be invisible to me. For instance, the super_lanes output is a high dimensional vector that could be used to capture the gestalt of what the camera sees. Embedded into that vector could be all kinds of things like "night, country road, parked cars, tail lights" and so forth. To extract that information a downstream process could pull the vector apart and evaluate it for some criteria - like "are there headlights coming towards me within 300 ft". If they use that approach then I'd never see evidence of it in the stuff I have access to.

    OTOH it's been around for a while (high beams) so it's conceivable that they used a simple heuristic approach to get it out the door and maybe they haven't revisited it since then. It's the kind of thing that's easy to get working at a basic level but hard to get it to work really well. In the long run they'll want it in the NN but maybe right now they have more pressing concerns on the vision front? Hard to say.
     
    • Informative x 7
    • Helpful x 1
  4. jimmy_d

    jimmy_d Deep Learning Dork

    Joined:
    Jan 26, 2016
    Messages:
    416
    Location:
    San Francisco, CA
    The decision making is not getting done in these networks. These are pure perception networks that only look at an instant of time and only process the input from a camera.

    I can't come up with a compelling argument that either 1) FSD will be an extension of EAP or alternately that 2) FSD is not being developed in common with EAP. There's a ton of stuff that must be present in their FSD vision code which isn't showing up in what I've seen so far. For instance, visual flow needs to be processed from full frame analysis of sequential frames. The FSD demo video that they released clearly shows visual flow being extracted, but the vision networks I've seen so far are not capable of comparing sequential frames and downstream processing of the output of these networks is very unlikely to be capable of extracting flow. That means that there are, at a minimum, different vision networks are being used for FSD from what I see here for AP.
     
    • Informative x 15
    • Helpful x 2
    • Like x 2
  5. scottrobertson

    Joined:
    Jul 4, 2017
    Messages:
    130
    Location:
    United Kingdom
    Thanks for the reply.

    That makes total sense as to why you would not be able to see. I just wanted to check if they were using a specific NN for it or not.
     
  6. mongo

    mongo Well-Known Member

    Joined:
    May 3, 2017
    Messages:
    12,857
    Location:
    Michigan
    If they get object recognition working well with EAP and they are using a ConvNet, then can they use the heat map output from each frame to track object movements for the higher level control routines to use?

    Hypothetical: lanes are a set of path vectors to group objects and their movements. For example head on (parallel), cross (perpendicular), or other angles. Mapping would be more dense closer and sparse for object further where the objects don't matter as soon/ path is less predictable.
     
  7. J1mbo

    J1mbo Active Member

    Joined:
    Aug 20, 2013
    Messages:
    1,565
    Location:
    UK
    And I guess it would be easier to do this and get predictable and repeatable results based on a series of "sterialised" snapshots from super_lanes than from the raw feed.
     
    • Helpful x 1
    • Informative x 1
  8. jimmy_d

    jimmy_d Deep Learning Dork

    Joined:
    Jan 26, 2016
    Messages:
    416
    Location:
    San Francisco, CA
    Quite true - that can be done. I don't mean to imply that there's no way to understand the movement of objects in the world from the output of the current AP vision. The method you suggest should work for that and I wouldn't be surprised if Tesla is already using that sort of thing to predict the trajectories of moving objects.

    My comment about flow is referring to this particular technique: [1504.06852] FlowNet: Learning Optical Flow with Convolutional Networks. In the FSD video you can see the side view cameras are showing motion flow as green line overlays most noticeable when passing foliage and so forth. This isn't the detection of motion in identified objects, it's the detection of background motion through texture translation. The linked paper is an example of how this is done using NNs today. Of course there are other methods but they are computationally intense enough that they will be run on a discrete GPU. And the requisite infrastructure isn't present in the NNs that I have seen so far.

    Sorry if I wasn't clear.
     
    • Informative x 7
    • Like x 1
  9. mongo

    mongo Well-Known Member

    Joined:
    May 3, 2017
    Messages:
    12,857
    Location:
    Michigan
    Oh no, I was the unclear one. I was just posing an idea, it was not a push back against your post.

    I've messed around a bit with the motion detection examples from Nvidia. It's seemed to me that blobafying objects then tracking them and/or identifying pavement and applying a perspective based distance mapping might be effective for preemptive collision detection.
     
    • Like x 1
  10. Snuffysasa

    Snuffysasa Member

    Joined:
    Feb 24, 2017
    Messages:
    487
    Location:
    Minneapolis
    @jimmy_d

    Just awesome! Really appreciated.

    Can anyone confirm the repeater cameras are being used? I understand the NN are there, but I am wondering if anyone can tell if they are being used for autopilot with this current firmware.
     
  11. Snuffysasa

    Snuffysasa Member

    Joined:
    Feb 24, 2017
    Messages:
    487
    Location:
    Minneapolis
    #171 Snuffysasa, Mar 15, 2018
    Last edited: Mar 15, 2018
    So is super_lanes the only vector that is an output from the main/narrow NN? Are there other output vectors too? Do they have names?

    Wait a moment,...
    you are saying the only change in the main/narrow neural network is they increased the data flow, and changed the name of an output vector?

    I feel I missed something? The rest of the network architecture, layer size/type/order etc is all still the same?
     
  12. Snuffysasa

    Snuffysasa Member

    Joined:
    Feb 24, 2017
    Messages:
    487
    Location:
    Minneapolis
    Ooooh. Sorry if this has been mentioned before... does the main/narrow network also output a six class segmentation map and bounding boxes??

    Oooo! Good catch. I figured the wide angle camera would eventually have to be used for scanning a wider FOV for cars/pedestrians... but figured they are aways off from that point in development.
     
  13. Tjhappel

    Tjhappel Member

    Joined:
    Dec 12, 2014
    Messages:
    254
    Location:
    San Clemente, California
    Thank you for the awesome detailed info... I can’t wait to see what 2-3 large updates look like when they actually use all cameras together. I feel like we are so close to being chauffeured (at least on the freeway).

    Thank you for your insightful work again, please keep it up
     
    • Like x 1
  14. Tjhappel

    Tjhappel Member

    Joined:
    Dec 12, 2014
    Messages:
    254
    Location:
    San Clemente, California
    Like when jay z told Kanye you a genius (homie) substituted wording.

    Seriously good post bruh
     
    • Like x 2
  15. Cirrus MS100D

    Cirrus MS100D Supporting Member

    Joined:
    Jul 6, 2017
    Messages:
    681
    Location:
    Pennsylvania, USA
    You all convinced me to take a drive last night to see how improved things really were (they are; I won’t go into detail here since so many others have already commented and demonstrated and posted video...) but I did want to mention that I felt the auto high beams were working phenomenally last night.

    Could totally be placebo, but it was something I noticed.

    I also believe some adjustments had to be made to the car-control side of things. Previously, the car seemed aware it wasn’t in the lane - it KNEW it was over the lane lines, but last night, it actually steered correctly to stay IN the lanes that it knew were there. No doubt the NN updates probably play a very large role, but actual car control seems vastly improved too. (Maybe just a case of “less garbage in / less garbage out?”)
     
    • Like x 7
  16. bhakan

    bhakan Member

    Joined:
    Jan 2, 2018
    Messages:
    224
    Location:
    Bethesda, MD
    jimmy_d- great review. Your last paragraph about "combining these camera outputs with the perception of sensors and making driving directions" will be the most critical step if TESLA has to roll our FSD in near future..
     
    • Like x 1
  17. jimmy_d

    jimmy_d Deep Learning Dork

    Joined:
    Jan 26, 2016
    Messages:
    416
    Location:
    San Francisco, CA
    The output branches of main/narrow are more complicated than just generic categories, but it has enough 'object category' type outputs that it could be a superset of what the other cameras are looking for.

    Yeah, in FSD the fisheye will have to do a lot of things besides operate the wipers.
     
    • Like x 1
  18. jimmy_d

    jimmy_d Deep Learning Dork

    Joined:
    Jan 26, 2016
    Messages:
    416
    Location:
    San Francisco, CA
    Yes. In recent builds the car clearly knew when it was out of it's lane because it would almost always have correct lane lines showing in the display even as it wandered across the lane boundaries. There had to have been some kind of control limitation that was preventing the vehicle from moving back to the center of the lane even when it knew it was in the 'wrong' place. Another poster relayed information from Tesla tech support to the effect that the car was only 'allowed' to turn a certain amount depending on speed, which nicely explains this weird mismatch between it's perception and it's actions. If lane holding is substantially improved now (I haven't received the update, myself) then Tesla must be confident enough in the new lane detection and steering planning processes that they are comfortable with relaxing those steering limitations.
     
    • Informative x 4
  19. mitchellh3

    mitchellh3 Member

    Joined:
    Nov 18, 2016
    Messages:
    162
    Location:
    Los Angeles, CA
    I have a local winding road I test AP updates on and I can say that at high enough speed it still veers out of its lane across the center line on turns. It does correct back into the middle but if there were oncoming cars it would be quite dangerous.

    And this isn't a freaky high speed, its a 35mph road where folks typically drive 40 to 45 and I set it at 40mph (due to local road Autosteer restrictions) and it will still veer.

    At 35mph though the turn is negotiated perfectly, which it never did before. So it is improving, but its still not perfect there.

    What I'm interested to test is freeway curves at 90mph Autosteer with no trailing car. That's always been incredibly sketchy as it gets really close to the wall. It seems to be a combination of detecting the curve too late and not turning enough. I haven't tried with 10.4 though!
     
    • Informative x 3
    • Like x 1
  20. Legolad

    Legolad Member

    Joined:
    Mar 10, 2018
    Messages:
    22
    Location:
    Atlanta
    So does this mean I should have taken the $3k deal on AP? I chose not to when I ordered, but I think I can still add it before I take delivery.
     
    • Like x 2

Share This Page

  • About Us

    Formed in 2006, Tesla Motors Club (TMC) was the first independent online Tesla community. Today it remains the largest and most dynamic community of Tesla enthusiasts. Learn more.
  • Do you value your experience at TMC? Consider becoming a Supporting Member of Tesla Motors Club. As a thank you for your contribution, you'll get nearly no ads in the Community and Groups sections. Additional perks are available depending on the level of contribution. Please visit the Account Upgrades page for more details.


    SUPPORT TMC