TMC is an independent, primarily volunteer organization that relies on ad revenue to cover its operating costs. Please consider whitelisting TMC on your ad blocker and becoming a Supporting Member. For more info: Support TMC
  1. TMC is currently READ ONLY.
    Click here for more info.

Andrej Karpathy - AI for Full-Self Driving (2020)

Discussion in 'Autopilot & Autonomous/FSD' started by Bladerskb, Apr 21, 2020.

  1. diplomat33

    diplomat33 Well-Known Member

    Joined:
    Aug 3, 2017
    Messages:
    6,832
    Location:
    Terre Haute, IN USA
    Thanks for sharing.
     
  2. scaesare

    scaesare Well-Known Member

    Joined:
    Mar 14, 2013
    Messages:
    8,178
    Location:
    NoVA
    I thought you were done here? ;)

    The problem is not that the additive input of all of the sensors doesn't provide a superset of any one sensor alone. It does.

    The point that has been made repeatedly, is that, even though not completely done yet, there are a core set of data that MUST be provided: edge detection, range, object classification, color/position recognition, etc... If the removal of any one sensor does not provided that needed data in the resulting subset, then the rest of the sensor suite is useless.

    So, nobody is arguing that lidar doesn't give you object and edge detection in the dark. But CV plus headlights also do

    Nobody is saying HD maps don't provide path planning. But so does CV, with the added benefit of dynamic response.

    Nobody is saying radar doesn't give range/speed data, but CV appear to be capable of that sufficiently.

    Nobody is saying ultrasonics don't provide close-proximity data. But CV could as well.


    In short, CV is the only sensor that has a roadmap to be able to provide all the needed driving data to a sufficient degree. Which isn't surprising, as that's how billions of cars are driven today.

    So while getting to that point is a work in progress, you have to decide where you want to point your architectural development and balance cost/benefit until CV is sufficient,

    Do you to plan on needing lidar knowing it can't suffice alone? Not a great development plan if focusing on CV can eliminate it. Do you want to support it in the interim? Not at 50-100x the cost of a camera, with perhaps multiple units needed.

    Radar in the mean time? It's cheap, so maybe so. Same with ultrasonics. But I wouldn't be surprised to see them eventually get deprecated as well.

    Bottom line: lidar is a crutch. An expensive one.
     
    • Like x 3
    • Disagree x 1
    • Love x 1
  3. diplomat33

    diplomat33 Well-Known Member

    Joined:
    Aug 3, 2017
    Messages:
    6,832
    Location:
    Terre Haute, IN USA
    I thought I would give you the courtesy of a reply to your post. :)
     
  4. scaesare

    scaesare Well-Known Member

    Joined:
    Mar 14, 2013
    Messages:
    8,178
    Location:
    NoVA
    As can CV
     
    • Like x 1
  5. Soda Popinski

    Soda Popinski Member

    Joined:
    May 9, 2018
    Messages:
    351
    Location:
    Los Angeles, CA
    Definitely expensive in 2016 when AP2 came out. Less so today and in the future. I can see a world where Tesla "solves" CV to 99.9999%, but maybe others need the "crutch" of Lidar to achieve that margin of safety under normal conditions (but say drop to 99% if either CV or Lidar fails, so it can stop safely). Or maybe Tesla will fail to hit that level with CV alone.
     
  6. mspisars

    mspisars Active Member

    Joined:
    May 23, 2014
    Messages:
    2,030
    Location:
    Charlotte, NC
    It is still expensive today.
    You are looking at a solid state LiDAR provider with a headline "$500 SS Lidar to come out in 2022".
    That is just the sensor itself, and one that only gives a limited field of view, so you'll need multiple sensors to get full coverage.

    This does not take into account the actual processing power and the Lidar-Vision software that needs to be written and validated.

    In Tesla's case, for example, you would probably need a second HW3 computer just for processing 360 view of all the lidar sensors (assuming your going with solid state, you will need multiple sensors to have a 360 view all around the car)
    Then, think of all the power consumption and the wiring harness redesign that would need to go into the cars with that new hardware.

    So, while Lidar sensors might get cheaper over time, the jump to include Lidar is not a simple slapping on of a few Lidar sensors but a much bigger overhaul of the entire suite.

    Not worth the headache since you MUST have CV solved anyway.
     
    • Like x 3
    • Disagree x 1
  7. diplomat33

    diplomat33 Well-Known Member

    Joined:
    Aug 3, 2017
    Messages:
    6,832
    Location:
    Terre Haute, IN USA
    If Tesla is not able to "solve vision" to enough 9's with cameras, then they will need additional sensors to get those extra 9's.
     
    • Disagree x 1
  8. Soda Popinski

    Soda Popinski Member

    Joined:
    May 9, 2018
    Messages:
    351
    Location:
    Los Angeles, CA
    I didn't say not expensive. I said less expensive. Even 8 solid state Lidars at $1000 is cheaper than the $40,000 Lidar systems available from Velodyne in 2016.

    I'd expect the lower resolution of Lidar to require less processing power than CV. The chip running the Lidar would have to do the distance measurement, since it's based on time of flight of photons. Everything else to generate the point cloud is simple trigonometry, with no NN required.

    Karpathy showed off Pseudo-Lidar, which produces a 3D point-cloud compatible with Lidar databases from camera data. If that's going in a future FSD system (HW3 re-write or even HW4 based), then the required NN is already baked in. Not that I expect Tesla to start adding in Lidar into the sensor suite.


    Depending on the application, say for dedicated robotaxies (not from Tesla), the extra expense could be well worth it to get to market sooner. I wrote it in another thread, but I can see a situation where you have CV + Lidar at 99.9999%, with CV being "solved enough" to say 99% alone in case Lidar fails, so the car pull over.[/quote][/QUOTE]
     
    • Like x 1
  9. diplomat33

    diplomat33 Well-Known Member

    Joined:
    Aug 3, 2017
    Messages:
    6,832
    Location:
    Terre Haute, IN USA
    Yes. According to this article, lidar actually requires less computing power than camera vision:

    "Finally, LiDAR saves computing power. LiDAR can immediately tell the distance to an object and direction of that object, whereas a camera-based system must first ingest the images and then analyze those images to determine the distance and speed of objects, requiring far more computational power."
    LiDAR vs. Cameras for Self Driving Cars - What's Best? - AutoPilot Review
     
  10. scaesare

    scaesare Well-Known Member

    Joined:
    Mar 14, 2013
    Messages:
    8,178
    Location:
    NoVA
    Still handing out courtesy replies? :D
     
    • Love x 2
    • Like x 1
  11. Bladerskb

    Bladerskb Senior Software Engineer

    Joined:
    Oct 24, 2016
    Messages:
    2,068
    Location:
    Michigan
    That chart is hilariously wrong and should not be used. It was created by Lex Fridman (Elon superfan) . Comparing radar resolution to Lidar is like saying being completely blind and only seeing random dots is only slightly worse than having 20/20 vision.



    That's their 5th gen radar. Tesla uses their 4th gen radar from 2010 (MRR (AP.20) and the ARS410 (AP 2.5+) and that one definitely CAN'T do object classification and has massive problems with differentiating stationary objects or large objects from small objects.

    Even that isn't adequate enough which is why Waymo created their own imaging radars.
     
    • Informative x 1
    • Like x 1
  12. scaesare

    scaesare Well-Known Member

    Joined:
    Mar 14, 2013
    Messages:
    8,178
    Location:
    NoVA
    Everything is relative. So that's 5x less than a Velodyne system. It's also 500x the cost of cameras.

    You know what else has a bill of material cost of $8000 in an mid-priced car? Nothing, except maybe the battery pack in an EV, and that has been ID'd as the biggest area where cost has to be reduced.


    Moore's law says the computing capability for a size/cost will double every 18 months... which is faster than non-computer hardware typically declines. It'll be cheaper to add more compute than try to add expensive but less data-intensive sensors.

    Heck, Tesla could double the amount of AI computing power in their cars today for less than the price of 1 lidar sensor, much less 8.


    This is why many of us see it as a crutch... sort term gains at the cost of dead-end long term architecture.
     
    • Like x 3
  13. mspisars

    mspisars Active Member

    Joined:
    May 23, 2014
    Messages:
    2,030
    Location:
    Charlotte, NC
    Rp = Radar Power consumption
    CVp = camera power consumption
    Lp = Lidar power consumption
    CPUp = in Tesla's case this is HW3 computer, but more generally the FSD computer.

    Rp + CVp + CPUp << Rp + CVp + CPUp + Lp

    Same concept for processing/compute power.
    Just because individually Lidar is less processing intensive, does not mean there is no processing required.
    And if there is processing required, then it is consuming more processing capacity of the CPU.
    Since we can't get rid of computer vision from any FSD solution any additional processing required for lidar is overhead.
     
    • Like x 3
  14. diplomat33

    diplomat33 Well-Known Member

    Joined:
    Aug 3, 2017
    Messages:
    6,832
    Location:
    Terre Haute, IN USA
    Except that you don't need as much CV computing power if you are using lidar. For example, you don't need CV to do "pseudo-lidar" if you are using lidar. So we don't know what the actual numbers will be for each variable in your equation.
     
    • Disagree x 4
  15. mspisars

    mspisars Active Member

    Joined:
    May 23, 2014
    Messages:
    2,030
    Location:
    Charlotte, NC
    Absolutely false. Remember when a few posts up you agreed that vision must be solved... well that vision solution will still end up using up compute power.

    You will still need to recreate an accurate virtual reality of your actual surroundings.
    That will all take the inputs from all the sensors and create the "birds-eye view" that Karpathy was talking about.
     
  16. Soda Popinski

    Soda Popinski Member

    Joined:
    May 9, 2018
    Messages:
    351
    Location:
    Los Angeles, CA
    I should clarify, I wasn't talking exclusively about Tesla's FSD solution. For other companies, that don't have access to Tesla's data, or constrained to selling cars private owners can afford, then I see a Lidar + CV system to be a viable path to FSD if you can't solve CV to 99.9999%. A short term gain could be the difference of going out of business / getting shut down or not for a Waymo or Cruise.
     
    • Helpful x 1
    • Disagree x 1
  17. diplomat33

    diplomat33 Well-Known Member

    Joined:
    Aug 3, 2017
    Messages:
    6,832
    Location:
    Terre Haute, IN USA
    But like I tried to explain to you earlier, you don't need to "solve vision" to the same number of 9's with lidar than without lidar. With camera-only, you need to "solve vision" to more 9's than with cameras + lidar. I am thinking that camera-only, since it needs to be solved to more 9's will probably use more computing power than CV that only needs to be solved to less 9's.
     
    • Disagree x 3
  18. mtndrew1

    mtndrew1 Active Member

    Joined:
    May 12, 2015
    Messages:
    1,311
    Location:
    Gardena, CA
    We’re really getting into the weeds here trying to justify LIDAR compute + vision using fewer cycles than just vision.

    An autonomous car can safely drive on vision alone.

    An autonomous car cannot safely drive on LIDAR alone.

    This is all anyone needs to know for years to come. If LIDAR is useful at any point in the future it will ADD to a solved vision system. You’ve agreed with this up thread.

    Honestly you’re looking for every little LIDAR crevice into which you can push string. Your argument is noted and nobody will know for some time whether adding it to a solved vision system will provide enough value to justify its cost, unsightliness, and range hit.
     
    • Like x 4
    • Disagree x 1
  19. MP3Mike

    MP3Mike Well-Known Member

    Joined:
    Feb 1, 2016
    Messages:
    14,925
    Location:
    Oregon
    You still have to solve to the same number of 9s, but you might not have to solve as many vision issues. You still have to have the same number of 9s for reading signs/stoplights but maybe you can rely exclusively on LiDAR for lane lines so vision doesn't need to handle that.
     
  20. scaesare

    scaesare Well-Known Member

    Joined:
    Mar 14, 2013
    Messages:
    8,178
    Location:
    NoVA
    Then lidar is going to reliably get a lot of cars up to the intersection only to have them sit there at green lights or blow through red ones significantly more often...
     
    • Like x 1
    • Funny x 1
    • Love x 1

Share This Page

  • About Us

    Formed in 2006, Tesla Motors Club (TMC) was the first independent online Tesla community. Today it remains the largest and most dynamic community of Tesla enthusiasts. Learn more.
  • Do you value your experience at TMC? Consider becoming a Supporting Member of Tesla Motors Club. As a thank you for your contribution, you'll get nearly no ads in the Community and Groups sections. Additional perks are available depending on the level of contribution. Please visit the Account Upgrades page for more details.


    SUPPORT TMC