TMC is an independent, primarily volunteer organization that relies on ad revenue to cover its operating costs. Please consider whitelisting TMC on your ad blocker and becoming a Supporting Member. For more info: Support TMC
  1. TMC is currently READ ONLY.
    Click here for more info.

Dojo

Discussion in 'Autopilot & Autonomous/FSD' started by Lasairfion, Dec 6, 2019.

  1. heltok

    heltok Active Member

    Joined:
    Aug 12, 2014
    Messages:
    1,135
    Location:
    Sweden
    Dojo will bring down the cost for Tesla, but it will not do anything that writing a big check to AWS will not do. It is Karpathy, his team, the data and HW3 that will bring in the improvements. Right now the bottle neck is validation of HW3 specific software. Once that is out we should see a big improvement in performance. How big is just speculation at this point.
     
    • Like x 2
    • Helpful x 1
  2. GrimRe

    GrimRe Member

    Joined:
    Aug 18, 2019
    Messages:
    161
    Location:
    Sydney, Australia
    It’s not just cost dude. AWS currently use the NVIDIA GPU acceleration for their deep learning instances. Whilst this is at an enormous scale there is a fundamental use case difference in that NVIDIA have many different types of workload to think about. This means certain a chip design that doesn’t suit Tesla specifically. You can hear Andrej talk about in his pyTorch talk to do with the GPU and Layer 1/2 RAM not being suited to video frames.

    So even if Tesla cut AWS a $500m cheque, it wouldn’t help them achieve FSD any faster. The idea is similar to sharpening the saw philosophy. Give Tesla a year to cut down a tree and they will spend 90% of it sharpening the saw.
     
    • Like x 1
    • Disagree x 1
  3. heltok

    heltok Active Member

    Joined:
    Aug 12, 2014
    Messages:
    1,135
    Location:
    Sweden
    Tesla has various needs, inference, training etc. All of these can be run on GPUs. Some task will require a lot of memory, like the pytorch example in the video. This can be done on GPUs with less memory, but it will be less efficient, thus cost more.

    Giving AWS a $500M check would decrease time to achieve FSD, it would allow developers to compile more frequently. Elon will do it better and get long term benefits, but it would help.
     
    • Like x 1
  4. GrimRe

    GrimRe Member

    Joined:
    Aug 18, 2019
    Messages:
    161
    Location:
    Sydney, Australia
    Respectfully, I disagree. Of course Tesla will use cloud providers like AWS for the vast majority of their IT workloads. However Dojo is different.

    Let’s assume the design goal of Dojo is roughly the same as inference optimised FSD chips. 10x performance increase at 40% price reduction.

    Now let’s say you have a training job on AWS that takes 100hrs to complete and costs $100,000. If you want AWS to now perform at the same level as Dojo the cost to train that model in 10hrs would be $1m. That or $60k on Dojo. For the same cost of $1m you could run almost 17 parallel runs of the Dojo training. Tesla running at the scale they are, $500m would be burned very quickly on AWS with not much to show for it. $500m worth of Dojo chips would result in a multi year improvement in FSD readiness.
     
    • Like x 2
  5. Pilot7478

    Pilot7478 Member

    Joined:
    Mar 13, 2018
    Messages:
    182
    Location:
    Toronto, ON Canada
    I think, once HW3 became available to Tesla developers, they have started using it exclusively for all new development.
    Once all this HW3 only work is released, we should see the massive improvement in Autopilot.

    What we see now (these few new features) is just a small part of what can be EASILY deployed in HW2+.
    It just does not make sense for them to continue extensive development for HW2+.
    And most of the new work just cannot be released to HW2+.

    However, HW2+ still needs to be supported for existing functionality (freeway driving), but I guess it's almost complete and will be just fine tuned in future releases (this is what we see in 40.2.1 - improvements to lane changes and NoA).
     
    • Helpful x 2
  6. strangecosmos2

    strangecosmos2 Koopa Troopa

    Joined:
    Nov 24, 2019
    Messages:
    177
    Location:
    New Donk City
    I think some version of this must be true. Going back to October 2018, Karpathy was discussing that his team has developed and trained new neural nets designed specifically for HW3/FSD Computer that are too compute hungry to run on HW2. He said: “we are currently at a place where we trained large neural networks that work very well, but we are not able to deploy them to the fleet due to computational constraints.” (full quote)

    I don't think anyone has found definitive proof that these new networks haven't been deployed to HW3 cars or that they have been deployed. Elon's tweets would suggest not yet. So, my hunch is that the “feature complete” FSD update will include the new nets. (Or if there are multiple updates adding different city driving features over time, then in one of those updates.)

    P.S. My account and/or TMC itself appears to have been hacked/compromised. Please be aware there may be a security vulnerability in the site and that messages from this account are not guaranteed to be from me.
     
    • Like x 1
  7. GrimRe

    GrimRe Member

    Joined:
    Aug 18, 2019
    Messages:
    161
    Location:
    Sydney, Australia
    I agree that Tesla are purposefully holding back AutoPilot features based on HW2 hardware limitations. They aren’t ready to upgrade everyone to HW3 that bought FSD option so they have to keep the feature set roughly equal on HW2/3 for now.

    What I assume the rollout may look similar to that of Smart Summon:

    Q4 2019
    - Handful of Early Access Program release candidate of the City Navigate on AutoPilot tuned for California

    Q1 2020
    - Wider release to Early Release Program members of City Navigate on AutoPilot (CNoAP) geo-fenced to certain road systems in North America (much like regular NoAP)

    Q4 2020
    - Initial release of CNoAP for all FSD owners in jurisdictions where approved and on road systems whitelisted by Tesla.

    Q4 2022
    - First Tesla RoboTaxi service launched in a partnered municipality in the U.S limited to cars using HW4 onboard computer due to its improved safety.

    Q2 2024
    - CNoAP now available in most markets on most road systems. RoboTaxi operating in most parts of North America.
     
    • Like x 3
  8. DanCar

    DanCar Active Member

    Joined:
    Oct 2, 2013
    Messages:
    1,670
    Location:
    SF Bay Area
    Stanford University says that the amount of compute power available for dojo work doubles every 3.4 months:
    https://hai.stanford.edu/sites/g/files/sbiybj10986/f/ai_index_2019_report.pdf
     
  9. strangecosmos2

    strangecosmos2 Koopa Troopa

    Joined:
    Nov 24, 2019
    Messages:
    177
    Location:
    New Donk City
    At PyTorch DevCon, Karpathy said the goal of Dojo is to “improve, basically, the efficiency by roughly an order of magnitude at a lower cost”.
     
    • Like x 1
  10. ratsbew

    ratsbew Active Member

    Joined:
    Mar 3, 2012
    Messages:
    1,116
    Location:
    O'Fallon, IL
    I'm hopeful that Tesla is taking all of the HW2.5 from cars that are being upgraded to HW3.0 and using that as "free" computing. Basically take all of the HW2.5 cards and build a supercomputer from them.
     
    • Like x 3
  11. DanCar

    DanCar Active Member

    Joined:
    Oct 2, 2013
    Messages:
    1,670
    Location:
    SF Bay Area
    @jimmy_d Gives us an overview of Dojo, dated Nov 10, 2020. Check out the table of contents in the overview.

    TIMESTAMPS:
    0:00 - Intro
    0:45 - Who is James Douma
    1:30 - What is Dojo?
    4:05 - How Tesla makes the system better
    4:30 - Labeling data
    5:15 - How to leverage labeling
    6:00 - What is 3D structure through motion
    8:55 - Dojo bringing down computational costs
    10:20 - GPUs for Neural Nets
    11:50 - Custom silicon for neural nets
    12:20 - Dojo’s current progress
    13:50 - What is labeling
    18:55 - Training neural nets
    21:24 - How neural nets work with human labelers
    23:30 - Supervised training
    24:00 - Misperception of what Tesla is doing
    25:20 - What data the fleet sends to Tesla
    29:50 - Embedding
    34:00 - Compounding improvement to system
    35:20 - Detecting edge cases
    36:25 - Driver interventions
    41:35 - What can be send to cars without a big firmware update?
    44:05 - Perception engine for FSD vs planning system
     
    • Informative x 1

Share This Page

  • About Us

    Formed in 2006, Tesla Motors Club (TMC) was the first independent online Tesla community. Today it remains the largest and most dynamic community of Tesla enthusiasts. Learn more.
  • Do you value your experience at TMC? Consider becoming a Supporting Member of Tesla Motors Club. As a thank you for your contribution, you'll get nearly no ads in the Community and Groups sections. Additional perks are available depending on the level of contribution. Please visit the Account Upgrades page for more details.


    SUPPORT TMC