TMC is an independent, primarily volunteer organization that relies on ad revenue to cover its operating costs. Please consider whitelisting TMC on your ad blocker and becoming a Supporting Member. For more info: Support TMC

Successful data recovery of broken eMMC chip MCU1

Discussion in 'Model S' started by widodh, Jan 20, 2020.

  1. widodh

    widodh Model S 85 and 100D

    Joined:
    Jan 23, 2011
    Messages:
    6,741
    Location:
    Venlo, NL
    #1 widodh, Jan 20, 2020
    Last edited: Jan 20, 2020
    For more information about what happens when the eMMC chip dies, see: Consolidated eMMC Thread (MCU repair)

    My story was that last summer (September 2019) the eMMC chip of my Model S from 2013 failed. It has 260.000km on the ODO and the chip failed before I was able to replace it.

    I did put on a new chip (64GB Swissbit), but I lost my (Open)VPN certificates.

    Through the Service Centers I tried to get Tesla to perform a re-install of my car, but they refused. I wrote to their European HQ a couple of times, they refused. The only thing they would do is replace my MCU and only charge for the parts.

    As I refused to pay the ~EUR 2000 for a new MCU I send my failed eMMC chip to Poland: Multi-Com.eu » Cell Phone Service, Repair, Unlock - Software, Tools, Devices

    In my own equipment this chip wouldn't respond anymore, but these guys did an amazing job!

    At first they said: The chip is toast.

    Then they came up with a solution: Rrder exactly the same chip and by reading the NAND memory directly using ECC corrections using the new donor chip they might have a shot at it.

    My chip needed to be cooled to -28C while they were reading data from it.

    Reading was slow; super, super slow. It took 8 weeks to get the bits of the chip and to restructure the data. But they did it!

    This morning I received the files I needed:

    - car.crt
    - car.key
    - ta.key

    They charged me a flat-fee of EUR 250 for this service. They say it took them way to much time to make a profit on it, so I paid them something in addition. It was still way cheaper then replacing the whole MCU.

    So should you be in the same boat as I was: Use these guys from Poland!
     
    • Informative x 11
    • Like x 8
    • Helpful x 3
    • Love x 2
  2. amiral_sub

    amiral_sub Member

    Joined:
    Sep 20, 2019
    Messages:
    65
    Location:
    Bordeaux, France
    unbelievable ! I'm very impressed.
     
  3. widodh

    widodh Model S 85 and 100D

    Joined:
    Jan 23, 2011
    Messages:
    6,741
    Location:
    Venlo, NL
    Yeah, me too! I honestly didn't expect this.

    But when I received the e-mail this morning with "openvpn.zip" as an attachment containing the files I needed I was very happy.
     
  4. T.R.T.e.s.l.a.

    Joined:
    Nov 8, 2018
    Messages:
    284
    Location:
    LX
    That is great Wido, and thanks for letting us know about those guys in Poland.
    But i always thought you had successfully saved all the data from the emmc you replaced on your S85.
     
  5. rooter

    rooter Member

    Joined:
    May 13, 2018
    Messages:
    440
    Location:
    Corona del Mar
    Nice, Wido. Good idea to cool it. One thing I've noticed about failed chips is you don't just get read errors on partition 3. (var) You often see them on all other parts too. This implies that it is the support circuitry to blame, which wouldn't surprise me. (yaay, Hynix)

    It's not clear why vpn certs were necessary, unless you want Tesla to continue to reach the car. (Assuming you have the same SIM card)

    Also you might ask Multi-Com for birthday, vin, and gateway.cfg files.
     
  6. SoCal Buzz

    SoCal Buzz Supporting Member

    Joined:
    Oct 9, 2018
    Messages:
    273
    Location:
    Orange County, CA, USA
    Nice job...! What was the status of your car while you waited to get the data files? Completely down?
     
  7. widodh

    widodh Model S 85 and 100D

    Joined:
    Jan 23, 2011
    Messages:
    6,741
    Location:
    Venlo, NL
    I kept it silent as I was trying to get Tesla to help me. I didn't want them to read on TMC that I replaced my eMMC and lost my certs.

    I want the VPN so that firmware updates work, Spotify works, the Tesla app, etc, etc. My wife drives this car.

    It was working. We put in a new eMMC with the proper firmware. So the car would drive, charge, everything. Remote access and firmware updates weren't working though.
     
    • Informative x 1
  8. T.R.T.e.s.l.a.

    Joined:
    Nov 8, 2018
    Messages:
    284
    Location:
    LX
    Yeah, that's understandable.

    Regarding firmware updates it's probably better that you just leave it as it is... haha.
    What version do you have installed currently?
     
  9. whitex

    whitex Well-Known Member

    Joined:
    Sep 30, 2015
    Messages:
    5,449
    Location:
    Seattle area, WA
    Out of curiosity, have you tried using your other car's certs/keys? I wonder if Tesla bothers to check whether the cert matches the car and/or how many simultaneous sessions there are (if only the latter, a donor dead MCU could be cheaper than new MCU too).
     
  10. widodh

    widodh Model S 85 and 100D

    Joined:
    Jan 23, 2011
    Messages:
    6,741
    Location:
    Venlo, NL
    No, I haven't. I also heard that it doesn't work. They strictly check this. The car also signs it's HTTP request using certificates.
     
  11. whitex

    whitex Well-Known Member

    Joined:
    Sep 30, 2015
    Messages:
    5,449
    Location:
    Seattle area, WA
    The 3 files you recovered look like standard openvpn configuration files (public cert, private key, pre-shared HMAC key used to add an additional layer of tls authentication to each packet). How does HTTP browser interact with those files?
     
  12. dannycamps

    dannycamps Member

    Joined:
    Apr 8, 2019
    Messages:
    327
    Location:
    Northeast USA
    Perhaps compare the files you received to a set from another car and see what the differences are.
     
  13. widodh

    widodh Model S 85 and 100D

    Joined:
    Jan 23, 2011
    Messages:
    6,741
    Location:
    Venlo, NL
    If you check the running processes you'll see that it's the hermes proxy doing this:

    Code:
    /opt/hermes/hermes_proxy --log-level=info --ca=/etc/ca.crt --cert=/etc/openvpn/car.crt --key=/etc/openvpn/car.key --api-server-host=api-prd.vn.tesla.services --unix-socket-buffer=200000 --http-signature-pub-keys=/etc/carserver-pub-keys/production.json
    
    When it sends HTTP requests to Tesla's servers it seems to sign requests with this so that the HTTP server can verify who is doing the call.

    So without these certificate files you can't connect to OpenVPN and if you are connected your VIN is encoded in all the requests you do.
     
    • Informative x 1
  14. LaurS

    LaurS Member

    Joined:
    Oct 2, 2019
    Messages:
    32
    Location:
    Camas, WA
    I plan to start taking things apart at the end of next week. I am worried that the eMMC will get damaged during removal from the board. I found this video where it show data being read while the eMMC is still on the board, only using 4 connection points: minute1:04. I assume two wires are power an two data. Does anybody know what communication device they used and what connection points? Is it worth trying to figure this out or just go with removal and reballing? I am in Portland OR area, any recommended shop to do the eMMC removal, reball and new eMMC install? Thank you.
     
  15. squall333

    squall333 Member

    Joined:
    Jan 31, 2020
    Messages:
    6
    Location:
    NJ
  16. squall333

    squall333 Member

    Joined:
    Jan 31, 2020
    Messages:
    6
    Location:
    NJ
    [​IMG]
    Based on this image it looks like possible

    CLK
    dD

    and two pins that arent on this maybe a 1.8v thats located from somewhere else?
     
  17. rooter

    rooter Member

    Joined:
    May 13, 2018
    Messages:
    440
    Location:
    Corona del Mar
    #17 rooter, Feb 1, 2020
    Last edited: Feb 1, 2020
    squall333 you need to credit that image. It is not yours.

    And the guys who made the above video know absolutely nothing about static, its effects on electronics, nor preventative measures.

    BTW, I now have permission to publish the all-powerful Fusee Gelee hack adapted to Tesla by Sid. In theory this should make possible downgrading firmware.

    But don't expect it to be easy. TonyT et al learned how to replace the eMMC, root, and pull images from my articles. Maybe at some point they'll be able to offer firmware downgrade.

    All I have to do now is figure it out...
     
  18. squall333

    squall333 Member

    Joined:
    Jan 31, 2020
    Messages:
    6
    Location:
    NJ
    @rooter its not common internet etiquette to credit reference images posted freely on the internet. Taking credit for images you didnt take is another story
     
  19. rooter

    rooter Member

    Joined:
    May 13, 2018
    Messages:
    440
    Location:
    Corona del Mar
    #19 rooter, Feb 1, 2020
    Last edited: Feb 1, 2020
    Typical priggish NJ attitude.

    I'll do it for you then. That picture was taken by ce2078 and it is his reverse-engineering depicted.

    In open-source we all stand on the shoulders of others and give credit for their science, as we hope others give credit for our work. A squall333 wouldn't know this but we share to help others, and as others share our work we deserve credit as the basis.

    To quote George Costanza, "We are supposed to be having a CIVILIZATION here people!"
     
  20. Slexs

    Slexs Member

    Joined:
    Jan 27, 2020
    Messages:
    17
    Location:
    Kristiansand Norway
    Im a bit confused when you say pin dD, what pin are you referring to here?
    Im in the process of planning emmc swap out for my own car, whst pins are actually connected/used to ie. AllSocket to get the data out?
     

Share This Page

  • About Us

    Formed in 2006, Tesla Motors Club (TMC) was the first independent online Tesla community. Today it remains the largest and most dynamic community of Tesla enthusiasts. Learn more.
  • Do you value your experience at TMC? Consider becoming a Supporting Member of Tesla Motors Club. As a thank you for your contribution, you'll get nearly no ads in the Community and Groups sections. Additional perks are available depending on the level of contribution. Please visit the Account Upgrades page for more details.


    SUPPORT TMC