TeslaMS tools for telemetry data visualization

[dirkhh]

I have two different setups I tried it on. One being a Win 7 "server" and the other on a raspberry pi "server". Stopped the visualize.js, downloaded the new package, and restarted both.
I have used chrome and ie on a Win 8 system to access the data. Also tried viewing the map on an ipad with IOS7 and got the same thing.

I'll do a clean install from git when I'm home and try to reproduce. I tested here with chrome and Firefox on Linux and it worked just fine...

 

[dirkhh]

I tried a clean reinstall and two more browsers. Works on all of them.
Can you open a web site debugger (something like "Inspect Element" on Firefox or Chrome) and tell me what the CSS property of div#container are? Not the computed ones, but the rules...

 

[hans]

I added a 10 second delay into the streaming.js code to avoid sending too many requests to Tesla and potentially getting blocked for too many requests. Now I am concerned that it also adds 10 second dead spots in between each poll. I can change it so that the delay only kicks if needed but I thought I should mention the current behavior in case you are wondering where these data gaps come from.

 

[dirkhh]

I added a 10 second delay into the streaming.js code to avoid sending too many requests to Tesla and potentially getting blocked for too many requests. Now I am concerned that it also adds 10 second dead spots in between each poll. I can change it so that the delay only kicks if needed but I thought I should mention the current behavior in case you are wondering where these data gaps come from.

10 second data gaps suck, Hans. You should only add the ten seconds if the re-connect FAILS.

 

[hans]

10 second data gaps suck, Hans. You should only add the ten seconds if the re-connect FAILS.

Yes I know. It's just there are so many ways to fail that I can't even trust HTTP OK anymore. I think I will add a request counter and only throttle back if the number of request per minute gets above a threshold.

---update---

Delay is removed when no HTTP errors. 10 second delay still there for calls to wake_up, which should be only every 30 minutes so not a big deal.

 

[dirkhh]

Yes I know. It's just there are so many ways to fail that I can't even trust HTTP OK anymore. I think I will add a request counter and only throttle back if the number of request per minute gets above a threshold.

That I like

 

[moviemaker2000]

I tried a clean reinstall and two more browsers. Works on all of them.
Can you open a web site debugger (something like "Inspect Element" on Firefox or Chrome) and tell me what the CSS property of div#container are? Not the computed ones, but the rules...

media="all"

div#container {

• margin-top: 0;
• padding-top: 1em;
• padding-bottom: 1em;
• border-radius: 0 0 10px 10px;
• background-color: rgba(244,244,244,0.9);
• border: 3px solid rgba(2,2,2,0.28);
• width: 94%;
• left: 3%;
• right: 3%;
• position: absolute;
• top: 74px;

}


user agent stylesheet
div {

• display: block;

}

 

[moviemaker2000]

I did a fresh install today on the newest posted build and the maps are working again on both platforms.:biggrin:

 

[brianman]

• background-color: rgba(244,244,244,0.9);
• border: 3px solid rgba(2,2,2,0.28);

Random observations:

Both of these struck me as odd. The first one is "so close to white that some monitors won't even display it differently" and also made me think "oh, maybe they're mistakenly attempting to provide a premultiplied rggba...but no 244 is not 0.9*255". The second is even more extreme "most people won't be able to tell the difference between #222 and #000 even if the monitor can display the difference accurately".

As the mix of %age and pixel offsets in left/right/width vs. top might produce some weird behavior. They're difference axes so you might be ok, but I have my suspicions anyway.

 

[dirkhh]

media="all"

div#container {

• margin-top: 0;
• padding-top: 1em;
• padding-bottom: 1em;
• border-radius: 0 0 10px 10px;
• background-color: rgba(244,244,244,0.9);
• border: 3px solid rgba(2,2,2,0.28);
• width: 94%;
• left: 3%;
• right: 3%;
• position: absolute;
• top: 74px;

}


user agent stylesheet
div {

• display: block;

}

This is NOT the latest version. It is missing the 'height' property - which is why it failed to show the map. I fixed this yesterday...

- - - Updated - - -

Random observations:

Both of these struck me as odd. The first one is "so close to white that some monitors won't even display it differently" and also made me think "oh, maybe they're mistakenly attempting to provide a premultiplied rggba...but no 244 is not 0.9*255". The second is even more extreme "most people won't be able to tell the difference between #222 and #000 even if the monitor can display the difference accurately".

As the mix of %age and pixel offsets in left/right/width vs. top might produce some weird behavior. They're difference axes so you might be ok, but I have my suspicions anyway.

I'm sure that they are not optimal. These simply came from color pickers and using pre-existing styles (that's where the alpha values come from).
I never claimed to actually know what I'm doing. I'm just doing it because I want visualization capability like this... and if I do things, I tend to do them open source.

 

[brianman]

@dirkhh - No worries. Was intended as "curious observations" and perhaps "constructive criticism". Your efforts are definitely appreciated.

 

[dirkhh]

@dirkhh - No worries. Was intended as "curious observations" and perhaps "constructive criticism". Your efforts are definitely appreciated.

Hey, I didn't take it any other way... and frankly... if you have any suggestions, patches, ideas... keep them coming!
I love open source software for a reason. It allows people who know more than me (and even if only on a small part of what we are doing) to help out and make things better!
So specifically on those colors, what would you suggest that I change these to?

 

[brianman]

Oh, well now, if you're asking...

• border-radius has a perf penalty in basically all cases, the degree of penalty will vary
• alpha has a perf penalty as well

... but if you're not looking for "why the f is this rendering so slow" then don't worry about it. It's not like you have 1000 elements each with alpha and border-radius (or do you?).

 

[dirkhh]

Oh, well now, if you're asking...

• border-radius has a perf penalty in basically all cases, the degree of penalty will vary
• alpha has a perf penalty as well

... but if you're not looking for "why the f is this rendering so slow" then don't worry about it. It's not like you have 1000 elements each with alpha and border-radius (or do you?).

No, only a handful of each.
The biggest performance issue is actually the database in the back. If you try to visualize the data for a couple of months even on a reasonably beefy server it takes about 10 seconds to get the response to the database query - the rest of the processing (also happening on the server) can take another 5-10 seconds (think about 2 months worth of samples, the more you were driving, the more of them are at 250ms resolution...). The rendering time is completely negligible compared to that.
So far no one (but me) has complained. That could be because few people are using it, or people don't mind waiting a little.
Once I have more of the features that I want done I will work on cashing some of the results of my calculations in the data base - that will make the queries MUCH faster.

 

[brianman]

No, only a handful of each.
The biggest performance issue is actually the database in the back. If you try to visualize the data for a couple of months even on a reasonably beefy server it takes about 10 seconds to get the response to the database query - the rest of the processing (also happening on the server) can take another 5-10 seconds (think about 2 months worth of samples, the more you were driving, the more of them are at 250ms resolution...). The rendering time is completely negligible compared to that.
So far no one (but me) has complained. That could be because few people are using it, or people don't mind waiting a little.
Once I have more of the features that I want done I will work on cashing some of the results of my calculations in the data base - that will make the queries MUCH faster.

The low time investment approach I've used in the past is throw an index (or multiple) on every table aggressively on fields that you search or join queries by. If I throw 20 indexes at my data, usually 1 of them nails the problem.

More expert folks will identify which indices you logically need and only make those, but for my hobby usage just throwing several indexes up is "wasteful" of database space but "simple and fast" with developer time.

 

[moviemaker2000]

No, only a handful of each.
The biggest performance issue is actually the database in the back. If you try to visualize the data for a couple of months even on a reasonably beefy server it takes about 10 seconds to get the response to the database query - the rest of the processing (also happening on the server) can take another 5-10 seconds (think about 2 months worth of samples, the more you were driving, the more of them are at 250ms resolution...). The rendering time is completely negligible compared to that.
So far no one (but me) has complained. That could be because few people are using it, or people don't mind waiting a little.
Once I have more of the features that I want done I will work on cashing some of the results of my calculations in the data base - that will make the queries MUCH faster.

I had noticed that too, but definitely not complaining with all the great work that has gone into this! Once I start looking at over a months worth of data it really starts to lag. My Windows server will return 3 months of results in reasonable time, but the raspberry pi chokes on that amount of data and the requesting browser will time out waiting for data. To get around that I have been opening multiple tabs and just loading one month per tab. I can quickly compare month to month that way. Not pretty but it works for now.:smile:

 

[dirkhh]

I had noticed that too, but definitely not complaining with all the great work that has gone into this! Once I start looking at over a months worth of data it really starts to lag. My Windows server will return 3 months of results in reasonable time, but the raspberry pi chokes on that amount of data and the requesting browser will time out waiting for data. To get around that I have been opening multiple tabs and just loading one month per tab. I can quickly compare month to month that way. Not pretty but it works for now.:smile:

A Pi is certainly not my target platform... but even on a beefy server I can see that doing this over, say, two years of driving data will kill performance very quickly.
So far no one has this amount of data, and by the time people do, I'll have the caching implemented.

 

[hans]

Request counters and throttling are now in streaming.js version 0.7.0 on GitHub. This should avoid any blocking from Tesla for too many requests.
Default is max of 6 requests per minute but it's tunable via the --maxrpm (-r) flag.
I added in support for a --zzz (-z) flag which will not wakeup a sleeping car (requires 5.0 or higher firmware and someone to test it for me).

If it runs without problem over the weekend I will make it the default install using npm.

----update---

Got a speeding ticket today. Figured I could check the logs in mongodb and fight it in court. Instead I confirmed to myself that he wrote me up for the exact highest speed I was going at the time. Bummer. I was hoping for a "man gets out of speeding ticket by hacking into Tesla" story.

----update(2)---

Published version 0.7.0 to npm repository. It's now the default install using "npm install teslams" or "sudo npm install -g teslams"


Also, I have noticed the odd GPS issue with visualize.js where I think a lat/long of null/null is being interpreted at lat/log of 0/0, See the map below.

 

[kwenix]

Here are a couple of examples of the latest you can do with visualize.js application that's part of the teslams project (it's in the examples folder of the teslams project on github). Not quite as fancy (needs some love from someone better with UIs than I am), but fairly functional and easy to run on your own system.

All you need is Node.js, MongoDB and a web browser. You can then stream your telemetry data into a database and connect to that database with two small web applications that do some analysis of the data. One view tracks your car on a map, visualizing speed through color, the other view tracks energy used / regeneratted over speed + SOC. They are linked to each other (so they cover the same time range) and the time range can be entered on the web pages.

Dirk,

maybe it's just me but I downloaded node.js, was able to find the config change (proxy settings) necessary to run the npm command to download the teslams package, but when running the tesla command, I see that the machine (apparently ?) ignores the npm proxy settings and tries a DIRECT HTTPS connection to the tesla servers. Did I miss something ?

Oh, and now that we are on the subject, please tell me what 'soc' stands for...
I tried this
SOC - What does SOC stand for? Acronyms and abbreviations by the Free Online Dictionary.
but that didn't help much

Thanks for helping out...

 

[dirkhh]

Dirk,

maybe it's just me but I downloaded node.js, was able to find the config change (proxy settings) necessary to run the npm command to download the teslams package, but when running the tesla command, I see that the machine (apparently ?) ignores the npm proxy settings and tries a DIRECT HTTPS connection to the tesla servers. Did I miss something ?

I am in charge of the front end (the visualize.js part) - Hans is writing the back end that is used to collect the data. So this is a question for Hans

Oh, and now that we are on the subject, please tell me what 'soc' stands for...
I tried this
SOC - What does SOC stand for? Acronyms and abbreviations by the Free Online Dictionary.
but that didn't help much

Thanks for helping out...

Hehe - I had the same reaction. In my world SOC stands for System On a Chip. But here in the EV world it means "State Of Charge" - or the percentage to which your battery is charged.