TeslaStream - Add Video + Display Mirroring Support To Your Tesla Browser!

[NotGalaxy]

Hello to fellow TMC members! I recently put together a quick app/script to enable video support directly on your Tesla browser! As many of you may have experience with, no video elements will work in the Tesla browser. Since we have a Model S, I thought I would just play around a little and see what I can come up with. There's still some significant restrictions and downsides of using this service at the moment, as you would need to have another device to stream the content from. The browser itself would not display the video content, so currently, this service converts the video and display the content as a stream of compressed photos. Of course, the video would not be playing anywhere near 30/60FPS, and you should be expecting about the same fluency similar to a VNC session(5FPS to 15FPS). This will work if you want to use the middle console as a bigger monitor for your laptop(Works offline as long as you can create a hotspot on your device), or even stream some videos if you don't mind low frames per second. I only tested this on the LTE version, so I could not comment on how laggy it would be if used on 3G. The program will be improved as time goes on, so what you see in the current Alpha build does not signify what the video quality could be in beta/release.


I'm currently announcing TeslaStream as WIP/Alpha, and I'm hoping a small amount of fellow TMC members would like to help out with testing it along with the development process. If you'd like to contribute to this project or help with testing, please leave a comment and PM me.


Warning And Legal Disclaimer: DO NOT IGNORE THE FOLLOWING - This service should only be used when the car is PARKED (0mph/0kph)! It is ILLEGAL to drive while watching video in ALL states!!! I am not responsible for your irresponsible use of this program! DO NOT consider using this while driving, as watching a couple videos is not worth it if you scratch your 100k Tesla! The use of this program may be restricted based on your local jurisdiction regarding built-in video player in your vehicle. Please consult with your local law before applying for access!

I'm not sure if anyone had done something similar before, so if there were similar projects in the past, please let me know!

Time For Some Photos Please excuse the fingerprints on the display.

 

[Ande]

Cool ting to tinker with, that I see.
But beside the security/privacy issues with another server, and software that interact with computer to provide the data, and setup time, there is the issue of a very slow browser, and added latency compared to a decent phone.

 

[bob_p]

Many people are already using their mobile devices for music and navigation, due to the limitations of the built-in media player and navigation apps. Since mobile devices can also stream and display video - with chips and software designed to support that, not sure if it's worth much effort to add low quality video streaming into the current browser.

When we're travelling, if the passenger wants to watch a video while the car is moving - we'll use an Android tablet - and stream the audio through the car's audio system over Bluetooth. The video player we will be using has an audio delay setting, so we'll be able to adjust the audio timing to get it in sync with the video on the tablet.

Another factor to consider - the current browser is pretty bad and old. Musk has promised for several years that we'll soon see an improved browser - and maybe that will finally happen this year. So spending a lot of time on working around the constraints of the current browser may not have much long term value.

Eventually Tesla will enable video on the console display. They could do it today when the car is in Park. When FSD is operating, the laws preventing video viewable by the 'driver' should be relaxed - and all manufacturers will likely expand the onboard infotainment systems to provide entertainment while the car is driving itself.

 

[NotGalaxy]

Cool ting to tinker with, that I see.
But beside the security/privacy issues with another server, and software that interact with computer to provide the data, and setup time, there is the issue of a very slow browser, and added latency compared to a decent phone.

You are spot on with the tinkering part. I love tinkering around with random projects For security/privacy, I'm releasing the source code on GitHub and I'm probably not going to do hosting as this mirrors the screen which will require me to set up hypervisor + a flavor of linux. Those are really good points you have brought up, and it does indeed add quite a bit of latency.

Many people are already using their mobile devices for music and navigation, due to the limitations of the built-in media player and navigation apps. Since mobile devices can also stream and display video - with chips and software designed to support that, not sure if it's worth much effort to add low quality video streaming into the current browser.

When we're travelling, if the passenger wants to watch a video while the car is moving - we'll use an Android tablet - and stream the audio through the car's audio system over Bluetooth. The video player we will be using has an audio delay setting, so we'll be able to adjust the audio timing to get it in sync with the video on the tablet.

Another factor to consider - the current browser is pretty bad and old. Musk has promised for several years that we'll soon see an improved browser - and maybe that will finally happen this year. So spending a lot of time on working around the constraints of the current browser may not have much long term value.

Eventually Tesla will enable video on the console display. They could do it today when the car is in Park. When FSD is operating, the laws preventing video viewable by the 'driver' should be relaxed - and all manufacturers will likely expand the onboard infotainment systems to provide entertainment while the car is driving itself.

We got ours on Dec 31,2014 and still no change in the browser situation. It led me to believe that they purposely disabled html5 video + audio element as the browser is actually HTML5 compatible. Not sure if it might be due to a contract between them and AT&T or they simply don't want to deal with the hassle. My original thought with this project is that you wouldn't have to use your phone's data since it's streamed in the Tesla browser itself, which could save some mobile fees for those that don't have unlimited data. This is just a hobby project for me and I'd like to see if I can help out in the community. Once we get to the point that they actually fix the horrible browser that randomly crash on webpages with unsupported elements, I'll finally be able to do rm -rf * and use the browser

 

[NotGalaxy]

But keep in mind the latency issue could be resolved (At least for video playback) as I'm also looking to see a javascript approach. I'm currently considering switching from PNG 1 by 1 to simply creating a gif every x seconds and then serve that. Or an even simpler approach would be to just serve the entire video as gif and play it.

 

[digistyl3]

What about sending diffs? That would reduce the size of the PNGs in case only a portion of the screen changes, enabling higher FPS.

Maybe it also makes sense to do lossy PNG compression.

BTW, this is a really cool project

 

[daikonv]

Any progress on this with the faster mcu?

 

[NotGalaxy]

Any progress on this with the faster mcu?

Not really an issue with the speed of the mcu. It's more dependent on the speed of the download bandwidth and compression methods used. The script already incorporate lossless png compression, but I'm still looking for a way to stream diffs in photos rather than downloading entire images. However I'm not sure how plausible it might be depending on how much Tesla has disabled basic html5 elements.

 

[amund7]

Not interested in video, but screen mirroring from my phone. That means any app could be viewable (probably not possible to relay touch inputs back to the phone?), such as Spotify (one that works), even Scan My Tesla

 

[Krash]

Mirroring your phone that way is a brilliant idea. Even better if the phone is emulated, which would make getting touchscreen feedback easier.

 

[NotGalaxy]

Mirroring your phone that way is a brilliant idea. Even better if the phone is emulated, which would make getting touchscreen feedback easier.

Not interested in video, but screen mirroring from my phone. That means any app could be viewable (probably not possible to relay touch inputs back to the phone?), such as Spotify (one that works), even Scan My Tesla

The current script is based on Python, and if you are talking about iOS, they have really stingy rules regarding screen mirroring. As for Android, the easiest approach would be to just download a VNC server app and use noVNC on the Tesla browser.

 

[amund7]

The current script is based on Python, and if you are talking about iOS, they have really stingy rules regarding screen mirroring. As for Android, the easiest approach would be to just download a VNC server app and use noVNC on the Tesla browser.

Whaaaat, that works already?? Never heard of it. Will try!

 

[NotGalaxy]

Mirroring your phone that way is a brilliant idea. Even better if the phone is emulated, which would make getting touchscreen feedback easier.

That's exactly I'm thinking!

If there's enough interest, I could create a service where you get a VM with any OS you'd like to be installed. So essentially you are getting a VPS that can be directly accessed on Tesla's browser with controls optimized for touch along with a keyboard. And since it's open source, anyone can host their own instance as well. You can directly mirror your display, or you can even install a virtual display driver and extend the length of your monitor.

 

[Toppatop55]

Anything new on how to get video in the model S LTE?

 

[Krash]

Anything new on how to get video in the model S LTE?

If you mean watching video over Tesla GSM LTE, that would require a software hack rather than this hardware hack. If you hijack the backup camera to show a movie, you still have to stream that movie on your own connection.

 

[Marcjero]

Hello I think I can build an android mirroring app. But I don't own a s or x, I will get a 3 when available in Europe. So I will.need beta testers.

First I need to.know more about the image rendering capabilities of the Tesla browser. I suppose animated gift are supported but I'm interested to know if webp or apng are working. Does.anyone can test the 3 first samples available here and report which ones are working : Animated PNG format: How does it compare to Animated WebP and GIF?

Thank you !

 

[Krash]

...Animated PNG format: How does it compare to Animated WebP and GIF?...

So that page may not be the best example. I can't follow the link to it from TMC in my car at all. It simply doesn't recognize the link. (I have the old MCU but recent firmware). If I load the page you reference manually none of the first examples load. The second and last aPNG load in my car but are not animated.

On my edge mobile browser, the second and last aPNG examples also load but also are not animated. The second and last WebP examples don't parse correctly and show the code for the image. The others don't load.

On my windows desktop the WebP and aPNG examples both do load but are not animated.

May I ask why you are trying to use animation? I think the whole idea is to use static images as screen shots while capturing screenpresses. The old MCU will never be fast enough to pass any length of animation along.

 

[Marcjero]

Thank you. aPNG is not well supported in general. Webp is interesting because of higher compression ratio than gif but unfortunately it doesn't work.
I think that streaming an animated gif would be the best option. The browser just need to wait for a.new frame and then render the picture. No JavaScript needed. A frame could be a full screen or just the changed bits.

But it looks like animated gif are not working either...

 

[Marcjero]

Thank you. aPNG is not well supported in general. Webp is interesting because of higher compression ratio than gif but unfortunately it doesn't work.
I think that streaming an animated gif would be the best option. The browser just need to wait for a.new frame and then render the picture. No JavaScript needed. A frame could be a full screen or just the changed bits.

But it looks like animated gif are not working either...

I think that the network speed can be an issue. If possible process the tests using WiFi.

 

[Marcjero]

Would be.nice to know if this gif is rendered on your browser
https://kornel.ski/lossy-cat.gif