Welcome to Tesla Motors Club
Discuss Tesla's Model S, Model 3, Model X, Model Y, Cybertruck, Roadster and More.
Register

Help: VIN 729 Roadster not starting, VMD-VDS communication fault

This site may earn commission on affiliate links.
... The files are not damaged, they are simply removed. And these files should not be written in normal circumstances so should not be actively being changed during the shutdown. ...

That is super weird, busybox removing these files? Only way I can think of when these files are touched is doing a forced update over CAN (that gets interrupted).
The correct way to power off the VMS imho is to a) inhibit APS, b) stop the VMS application, c) pull the service disconnect (optional?), ...
On b) no easy way to do that via the VDS, can you program OVMS in to telnet in (over CAN) to issue the VMS shutdown command?
 
That is super weird, busybox removing these files? Only way I can think of when these files are touched is doing a forced update over CAN (that gets interrupted).
I don't think it is busybox - those are just userspace tools. I suspect the bootloader; either that or it is a huge coincidence that those are the files used to boot Linux from. Everything else on disk looks fine.
On b) no easy way to do that via the VDS, can you program OVMS in to telnet in (over CAN) to issue the VMS shutdown command?
That is what I have been working on for a while. Hard with zero technical information from official channels. So far, we've decoded most of the CAN bus traffic, and have a good understanding of the proprietary engineering tools that tesla use (but are not generally available). We think we can use OVMS either as a USB gateway for laptop engineering tools to use, or directly to talk to the VMS (or other systems in the car). Up until now, OVMS has only used the driver feedback bus. That said, this will require a different cable (as CAN4 is where the VMS diagnostics are). I think the approach will be to connect diag, ess/hvac, and driver feedback (ignoring the powertrain bus as mostly useless for what we need anyway).

On the positive side, Tesla have implemented multiple ways of doing each task, and that is an excellent approach. But so far all the ways I've found to shutdown the VMS applications cleanly require IP-over-CAN connectivity (which is quite a bit of work to get done in an embedded system like OVMS).

If anyone wants to help with this engineering work on roadsters, with the goal of creating an open set of engineering tools for third party repair, please get in contact with me. We already have a few kind volunteers working on it.
 
  • Like
Reactions: drewski
Having worked on this, I feel the need to warn users about powering off the VMS without proper shutdown.

There appears to be a bug in Tesla VMS firmware (I suspect in the bootloader) where if the VMS is improperly shutdown (power pulled when running, or some other interruption during boot) the file system can become corrupted. The symptom for this is that either the linux.bin (Linux kernel) or vms.initrd (initial ram disk) files are missing on /flash disk. In such cases, the VMS will be reachable on the CAN4 diagnostics, but none of the other VMS traffic will respond, the VDS will report a communication prompt and be useless. VMSIO is unaffected. Reportedly this has already happened in several cases.

I don't think this is a normal filesystem corruption. The files are not damaged, they are simply removed. And these files should not be written in normal circumstances so should not be actively being changed during the shutdown.

The fix is not trivial, and requires proprietary engineering tools to accomplish. An equivalent replacement linux.bin or vms.initrd image needs to be uploaded to the VMS and put back on /flash with the correct name. Making a backup of linux.bin and vms.initrd on the same flash is useful, as it will simplify recovery should the problem happen again in future.

The correct way to power off the VMS imho is to a) inhibit APS, b) stop the VMS application, c) pull the service disconnect (optional?), and d) cleanly and smoothly powering down the VMS. Similarly, when powering on, the power should be applied smoothly. I would plug in J3 before J2. As the VMS is powered by both APS permanent power as well as BPS (little 12v) in 2.x cars, care should be taken to ensure that the BPS is good before messing with service disconnect.
I am really lucky to have Mark helping me debug the VMS and eventually managed to locate the problem and solve it with his amazing skills. It's amazing to still have those talented people like Mark and others (Spaceballs, ML Auto, Gruber etc.) around and help save so many roadsters. I'll keep updating the status of this vehicle after I put the VMS back. I also hope this thread can help other owners avoid some potential problems.
 
Just want to give an update about the vehicle.

So the VMS was indeed the cause of the problem. After I put the VMS back into the vehicle the VDS can finally talk to the VMS and enter the service menu. The "VMS-VDS comms fault" is gone, and VDS no longer asking me to set a new PIN.

Well, that's the good news part. The bad news is that one of the bricks is dead (1.02V) during this period of debugging the VMS. While I managed to keep brick #29 alive with the balancing, I did not expect #66 to drop its voltage so quickly as it was not the lowest brick and service plug was pulled, and I did not have an easy way to check its voltage because the VMS was not working, except pulling the BMB board.

I trickle charged the dead brick to around 2.8V so that at least I could turn on the APS and to put the vehicle into tow mode.

I called up the guy in Hangzhou who does cell-level battery repair and shipped the vehicle over.

Will update how this goes.
 

Attachments

  • 1d36333b159a83649fd9c125caf64ea.jpg
    1d36333b159a83649fd9c125caf64ea.jpg
    301.1 KB · Views: 123
  • 10c1c3dd78027df785f76925b828273.jpg
    10c1c3dd78027df785f76925b828273.jpg
    135.6 KB · Views: 86
  • Helpful
  • Informative
Reactions: markwj and tvuolo
I am really lucky to have Mark helping me debug the VMS and eventually managed to locate the problem and solve it with his amazing skills.
@steve1an …. What was the problem and how did you solve it? I’m having a similar issue and have pulled the two larger plugs to the VMS but things actually got worse after doing that. I have a few threads up top now so struggling with 2 different issues, no HVAC cooling and no comms between VMS and VDS. Any sharing of your fix would be greatly appreciated. Thanks!
 
@steve1an …. What was the problem and how did you solve it? I’m having a similar issue and have pulled the two larger plugs to the VMS but things actually got worse after doing that. I have a few threads up top now so struggling with 2 different issues, no HVAC cooling and no comms between VMS and VDS. Any sharing of your fix would be greatly appreciated. Thanks!
Issue was corrupted firmware in the VMS resulting in it being unable to boot. Fix is not trivial and involves using CAN bus tools to re-upload firmware files to the VMS.

In USA, either Medlock or Gruber should be able to help if you have the same issue. If you have OVMS with an OVT1 cable and wifi connection, I can connect in remotely to check. eMail me at mark (at) openvehicles (dot) com if you need that.