TMC is an independent, primarily volunteer organization that relies on ad revenue to cover its operating costs. Please consider whitelisting TMC on your ad blocker or making a Paypal contribution here: paypal.me/SupportTMC

Log Parsing tool available

Discussion in 'Roadster: Technical' started by scott451, May 1, 2010.

  1. scott451

    scott451 KWH-PWR#1349Sprt,S Sig#96

    Joined:
    Apr 2, 2009
    Messages:
    251
    Location:
    Palo Alto
    #1 scott451, May 1, 2010
    Last edited: Nov 15, 2010
    I’ve decided to release my tesla log parsing tool. It’s very basic. It will convert your tesla log file into an excel csv file. There are a lot of interesting values in the log like speed, current, torque, state of charge, etc. There are also a lot of values that are not decoded. I am releasing this tool to enlist your help to decode the remaining values. You can access the parser at the following website. [EDIT 11-15-2010. Website is off line. Please use attached zip file.]


    You will still need to do some data crunching in excel to produce useful plots. As an example I added efficiency as ESS_I * ESS_V / (Line_I * Line_V) to the data collected in the log. Attached is an example plot created in excel from the output of my parser.

    attachment.php?attachmentid=672&d=1272728135.gif

    -Scott

    Note: the temp data in the PERM_Daily is wrong. Also the chargeV2 record for 2010 is not supported. (I still need to collect the CAN bus data off a 2010 to figure this one out... )
     

    Attached Files:

  2. doug

    doug Administrator / Head Moderator

    Joined:
    Nov 28, 2006
    Messages:
    15,913
    Location:
    Stanford, California
    pretty cool
     
  3. Alpine Driver

    Alpine Driver Member

    Joined:
    Mar 24, 2008
    Messages:
    92
    Location:
    Austria
    well done, Scott, thank you.
    It worked for my 2010 EU signature.
    After download, Excel said "file could not be fully loaded", but that might be because it is more than 65k lines?
    Now I sit here with -a lot- of data. As I am not an Excel expert, I will have to find someone who will make "nice curves" out of that data. What I could read in the plain text is a lot of error messages like "Motor Sensor 1 (or 2) overTemp warning. Torque limited." which might reflect my driving style ;-).
    I also could clarify some charging errror messages I had in the VDS "Charging Error check power source" which in the logfiles much more clearly says "Line underVoltage warning".
    But what I do have extremely often, nearly every day in a dozen is an "isolation resistance warning".
    Everyone else has this "isolation resistance warning"? It is not in the VDS nor in the speedo, so I think it might not be of that influence ... hopefully ...

    best
    Martin
     
  4. scott451

    scott451 KWH-PWR#1349Sprt,S Sig#96

    Joined:
    Apr 2, 2009
    Messages:
    251
    Location:
    Palo Alto
    #4 scott451, May 2, 2010
    Last edited: May 2, 2010
    The "file could not be fully loaded" is an Excel 2004 and earlier problem. Excel 2007 doesn't have this limitation. To get around it, look in the spread sheet and find the "utime" that is nearest the event you want to track. Copy that utime into the website "start time" field and re upload the file. The new csv file will start at the new utime. Additonally you can shorten the csv file in wordpad first and then load it into excel.

    Once it is in excel, to get nice curves, data->sort on Column A and then select the data you are interested in and click the chart wizzard.

    The error messages in the log file, will be slightly different than the ones you see on the VDS. The reason is that I use the "verbose error text" not the standard errors. The verbose errors are more descriptive. Collecting the error messages for the parser is a tedious process of analizing the data on the CAN bus and hand typing into my code. I've just about finished some simulation hardware, so this should make it somewhat easier, rather than having to collect the data from my car...

    I should have the 2010 V2 charge data worked out in a few weeks. I need to bring my CAN bus stuff over to a 2010 owner and collect the data, but I've been too swamped at my real job...

    -Scott
     
  5. bolosky

    bolosky Member

    Joined:
    May 5, 2009
    Messages:
    422
    I saw this once on the VDS. It was when I was running on a race track. I had gotten a number of the motor overtemp warnings that you mention, and once the isolation resistence one, too. I never saw it before or since.

    Maybe what we should take away from this is that your everyday driving is like what I do on the track. :)
     
  6. tomsax

    tomsax Member

    Joined:
    Jun 19, 2008
    Messages:
    867
    Location:
    Sammamish, WA
    Scott,

    Thanks for making your log analysis available to other owners.

    The log files contain a bunch of information including your VIN, your driving second-by-second, your charging, and the GPS coordinates of everywhere you charge, which presumably includes your home location.

    Although there's apparently no GPS data included with the driving records, I suspect that there is enough data that driving routes could be reconstructed by comparing driving patterns (speed, distance and stopping times) with stop lights and traffic flow data in your local area, thus potentially revealing a lot of personal information from those logs.

    What's your data retention and privacy policy for the data that owners upload to your web server?
     
  7. scott451

    scott451 KWH-PWR#1349Sprt,S Sig#96

    Joined:
    Apr 2, 2009
    Messages:
    251
    Location:
    Palo Alto
    #7 scott451, May 12, 2010
    Last edited: May 12, 2010
    Not exactly. The log contains the places where you "limited" the charge. the log does not contain the location of every place you charge. the GPS data is ignored by my parser. The time of charge is not included in the GPS location so it is difficult to correlate to the charge record.

    good luck :) the log does not contain distance travailed per second but speed sampled every second. It also doesn't contain the heading. So you don't know when the GPS location was stored, you don't know how much distance you've travailed in a second, and you don't know which direction you are going. Pretty tough to get any useful data. (e,g where I work, the route I took, etc)

    Log files are used to improve the tool (e.g add more error messages). Parsed data is created and then deleted. uploaded log files are deleted regularly.


    -Scott
     
  8. tomsax

    tomsax Member

    Joined:
    Jun 19, 2008
    Messages:
    867
    Location:
    Sammamish, WA
    I limit charging at home, so my home coordinates are in the log file. I know others do the same.

    Using speed * time = distance on the once-per-second records yields a reasonable estimate of distance traveled, with maybe a 5% to 10% error. That can be improved using the once-per-ten-minute records that contain the odometer to the 1/10th mile and interpolating. I haven't done the work to see how much accuracy can be obtained, but I can't dismiss that good accuracy is possible.

    You can definitely reconstruct information like this: drove 10.2 miles at low speeds with several stops (city driving), drove 26.3 miles at 75 mph on average (got on the freeway at the exit which is 10.2 miles from home, then got off at the only exit 26.3 miles from the first), then more figuring things out by distances between stop lights. I'm not saying you could always figure things out exactly, but I'm sure someone who wanted to could reconstruct a lot of information.

    I'm sure you're not doing this with the data that owners submit, but it is a privacy issue that I think owners should be aware of when they turn their log files over to Tesla, or anyone else.

    So you're not keeping any identifiable data for more than a short period of time? That's great to know. I'd recommend stating that on your upload page.

    Despite asking the annoying questions, I think it's great you're sharing your program with the community. Thanks for taking the time to answer my post.
     
  9. scott451

    scott451 KWH-PWR#1349Sprt,S Sig#96

    Joined:
    Apr 2, 2009
    Messages:
    251
    Location:
    Palo Alto
    #9 scott451, May 13, 2010
    Last edited: Jun 12, 2010
    Also. The latest version of the [2008, 2010, and sport] firmware encrypts the GPS charging location, vms and user config files, so they can't be read by anyone but tesla.
     
  10. scott451

    scott451 KWH-PWR#1349Sprt,S Sig#96

    Joined:
    Apr 2, 2009
    Messages:
    251
    Location:
    Palo Alto
    #10 scott451, Jun 12, 2010
    Last edited: Jun 12, 2010
    log reader parser Version _v12 online

    I've updated my log parsing tool. The new version includes the following features:

    Correct parsing of ChargeV2 records from 2010 Roadsters (thx to Richard for alowing me to log his 2010)
    Added brick numbers to min temp, max temp, and min v, max V fields.
    Added brake and regen bits to drive1s records (from tomsax PM)
    Added support for parsing older hidden records cause by wraped logs.
    Merged temp and perm logs into one parsing for both 2008 and 2010.
    Added more error text messages based on id# from uploaded logs.

    Known issues:
    For 2010 roadster logs, the Sleep record does not contain the correct the min/max/brick voltages.
    For 2010 roadster logs, the Charge30m record does not contain the motor temperature, so it is set to zero.
    Brick numbers may be off by 1.

    -Scott

    hxxp://logparse.servebbs.com:10376 (change the xx to tt )
     
  11. DrTaras

    DrTaras R254->R725->S1364-->X769

    Joined:
    Dec 2, 2007
    Messages:
    1,247
    Location:
    Woodland Hills, CA
    Scott-
    Did you ever get any interesting or useful info out of my logs? :confused:
     
  12. NEWDL

    NEWDL R#350 R#1323 Sig23 8136

    Joined:
    Jan 5, 2009
    Messages:
    473
    Location:
    NE OHIO
    Paranoia will Destroy ya...


    Paranoia will Destroy ya...:eek:

    How many ways can someone get your address? Really?:rolleyes:
     
  13. scott451

    scott451 KWH-PWR#1349Sprt,S Sig#96

    Joined:
    Apr 2, 2009
    Messages:
    251
    Location:
    Palo Alto
    #13 scott451, Jun 25, 2010
    Last edited: Jun 25, 2010
    Standalone log parse tool available

    Attached is the standalone version of my log_parse tool. I have included both a Windows and a Mac version. The web version will always have the most current version of the parser and should be used for unknown PERM_error messages (e.g. id3001).

    Enjoy,

    -Scott

    Special thanks to Steve Casner who was very helpful in figuring out the scaling of many of the values used in the early developement of this tool. Tomsax figured out the brake/RGbrake bits used in the Drive1s record. And Thanks to all the other owners who provided log files and 2010 data logging sessions.
     

    Attached Files:

  14. TEG

    TEG TMC Moderator

    Joined:
    Aug 20, 2006
    Messages:
    17,249
    Location:
    Silicon Valley
    Will it ever go out as OpenSource?
     
  15. scott451

    scott451 KWH-PWR#1349Sprt,S Sig#96

    Joined:
    Apr 2, 2009
    Messages:
    251
    Location:
    Palo Alto
    #15 scott451, Jun 25, 2010
    Last edited: Jun 25, 2010
    Yes. At some point when I get tired of working on it... It's hack code though, because it was written to group bytes into words or long words (by identifying msB/lsB rollover). Later scaling was added to convert the data into meaningful values. Lastly a look up table was added to convert error message IDs into text messages. It may be more useful for me to publish the record types, values and scaling, so that someone could write a more user friendly (graphical) tool.

    -Scott
     
  16. scott451

    scott451 KWH-PWR#1349Sprt,S Sig#96

    Joined:
    Apr 2, 2009
    Messages:
    251
    Location:
    Palo Alto
    As far as wireless TPMS, I don't think it's an issue because the TPMS sits on the instrument bus and the PEM, Motor, and battery sit on totally different busses. The only attack (DoS) you could send externally would be a flood of TPMS low pressure errors that might slow down the instrument bus and cause other errors. But you would still be able to pull over safely and stop the car.

    -Scott
     
  17. pbostley

    pbostley #740

    Joined:
    Dec 6, 2009
    Messages:
    4
    version timeout.

    Hey Scott,

    a) Thanks for the tool.
    b) is a new version or source available?
    This version expired on Thu Sep 23 11:29:29 2010, 1285262969

    -p.j.
     
  18. scott451

    scott451 KWH-PWR#1349Sprt,S Sig#96

    Joined:
    Apr 2, 2009
    Messages:
    251
    Location:
    Palo Alto
    #19 scott451, Oct 5, 2010
    Last edited: Oct 5, 2010
    Sorry, I've been spending most of my spare time analyzing CAN bus data, I'll get a new version out in a week or two. I'm trying to release the source too, but I need to get the OK from work.

    -Scott
     
  19. Doug_G

    Doug_G Lead Moderator

    Joined:
    Apr 2, 2010
    Messages:
    15,850
    Location:
    Ottawa, Canada
    Source would be great. I have a nice little data plotting utility that I wrote for another purpose, which would be fun to adapt to reading And displaying Roadster logs.
     

Share This Page