Autopilot in 360 Degrees

[verygreen]

I wonder what's so special about the backup camera. Why would it freeze, and not the others? Is it because the signal is split where it goes to the MCU and the drive computer?

no idea, really. on ap2 signal is NOT split. it goes into ap2.0 and is then mirrored to the cid. hw2.5 adds hardware splitter.

on model3 backup camera works differently, over there they get the backup camera signal and apparently stream it over the network to the central screen (in h264 compressed form or some such). It is done by the "backup camera" process that was only included in model3 firmware at first. Fun fact, when "dashcam" feature was added, they just added ability to stream narrow cam in the exact same way to that same process (different command line switch), added it to hw2.5 cars and now cid listens on this feed when you want the dashcam footage.

 

[lunitiks]

BUC has a different sensor than the other cams as well.

On hw2.0 the BUC cmos was OV10635, while the rest were AT0132.

Not sure what the hw2.5 cameras use.

(Probably shouldn’t matter, but the BUC stands out in many ways, so to speak.)

 

[Soda Popinski]

Anyone create a 360 degree video that can be played on an Oculus Rift?

If you have Virtual Desktop, you can paste the YouTube URL into the video player, and it'll display it 360 around you. There may also be a YouTube VR player in Steam, reportedly only for Vive, but works on Rift too.

 

[pkodali]

Working on doing a better job on dealing with the edges for each video. Unfortunately, due to the different positions of the cameras, there are some overlaps that can't be fixed. I'll publish a longer, better version once I fix the issue and get my hands on some rearview cam footage

 

[S4WRXTTCS]

no idea, really. on ap2 signal is NOT split. it goes into ap2.0 and is then mirrored to the cid.

For clarification what's the difference between mirrored, and split for this signal?

To me mirroring is recreating the digital video signal versus just splitting it. But, it wouldn't be able to recreate the video signal if the incoming video stream was stopped/corrupted. So if it was mirrored this way you would never have a case of the CID working when the AP2 wasn't.
The kind of signal isn't like an analog video where you easily split it. Instead you receive the signal using a deserializer that has the capability of replicating the signal on a second channel. Something like the TI DS90UB954 deserializer, and then something else to serialize the second channel in order to send it to the CID.

On AP2.5 what does the hardware splitter do exactly? On your post I'm assuming the signal from the backup camera gets split so it goes to the Drive computer, and the CID. But, there is conflicting information that says there is redundant wiring on AP 2.5 so the backup camera actually goes to the drive computer twice and that's what the splitter is used for. Where one signal goes to the first Parker SoC, and the second signal goes to the second Parker SoC chip.

Post 13 of this link is where I'm getting that info.
v9.0 Dashcam - why only 2.5?

In that post it says AP2 has two autopilot engines. I wouldn't really call it that as the Parker SoC isn't the entire autopilot engine. But, the Parker has the ISP's for the cameras.

I'm confused about why the backup camera would go to both, and not having ALL the cameras go to both.

I should either find a diagram of how the cameras are connected in AP2.5 or to create my own. I have a Model 3, and so that's the only one applicable to me.

 

[verygreen]

split - on hw2.5 there's a hardware splitter (physically in ape housing, but a totally separate small board)), it receives the backup camera signal, then splits it somehow and then spits two versions, one goes into ape and one goes into cid.
on hw 2.0 camera signal goes into ape (the main board), gets processed there somehow and then spit out the other port to be given to the cid.

You are right that technically there should be no way to recreate the signal if the input is lost, but ap2.0 was more affected by the glitch on the cid display if I remember right, and also there might be several receivers for it internally? I did not look too much into it to have a super clear understanding of how it works, though.

I am not sure the second parker chip gets any separate camera inputs too though again I did not investigate it in any great details for now.

 

[DJP31]

I’m following this with interest although don’t understand the technical stuff.

My S AP2 on 42.3 is not showing anything from the rear camera, irrespective of speed or distance of cars directly behind me.

Only had V9 for a couple of days and wondered if it was calibrating but perhaps not. I’ve also noticed the car has been down/uploading significantly more than the usual daily amounts of data.

 

[verygreen]

I’m following this with interest although don’t understand the technical stuff.

My S AP2 on 42.3 is not showing anything from the rear camera, irrespective of speed or distance of cars directly behind me.

Only had V9 for a couple of days and wondered if it was calibrating but perhaps not. I’ve also noticed the car has been down/uploading significantly more than the usual daily amounts of data.

I don't know if it will work for you, but I noticed that if you approach the car and quickly turn on the backup camera view on the central screen (usually by turning into reverse gear) - there's a much higher chance the camera would work for autopilot detections. If you don't it usually flakes out within a minute of the car staring up.

Even when it does work you pretty much need to have a car stopped behind you at a traffic light for it to show anything.

 

[S4WRXTTCS]

I don't know if it will work for you, but I noticed that if you approach the car and quickly turn on the backup camera view on the central screen (usually by turning into reverse gear) - there's a much higher chance the camera would work for autopilot detections. If you don't it usually flakes out within a minute of the car staring up.

Even when it does work you pretty much need to have a car stopped behind you at a traffic light for it to show anything.

Is this with AP 2.5 car and MCU2?

If I recall correctly you have a fairly new Model X?

 

[verygreen]

Is this with AP 2.5 car and MCU2?

If I recall correctly you have a fairly new Model X?

no, I have MCU1, but it does not matter what MCU you have I imagine, this is all internal to the ape and we know both ap2.0 and ap2.5 are affected (and there are reports that even model3s).

 

[DJP31]

I don't know if it will work for you, but I noticed that if you approach the car and quickly turn on the backup camera view on the central screen (usually by turning into reverse gear) - there's a much higher chance the camera would work for autopilot detections. If you don't it usually flakes out within a minute of the car staring up.

Even when it does work you pretty much need to have a car stopped behind you at a traffic light for it to show anything.

Thanks @verygreen i will give that a try and report back.

 

[S4WRXTTCS]

no, I have MCU1, but it does not matter what MCU you have I imagine, this is all internal to the ape and we know both ap2.0 and ap2.5 are affected (and there are reports that even model3s).

It could potentially matter.

The Model 3 was the first one that had the Intel MCU. It's a noticeable departure from the Nvidia MCU in that they would have to create drivers for the ISP (on the Intel SoC) that takes care of bringing CSI video in.

My guess is they never made these drivers, and opted instead to have the compressed camera feed come in via Ethernet. Maybe at some point they planned on having drivers for the Intel MCU, but then the dashcam thing happened so now there is no point to it.

What this means is this compressed video signal to the MCU can't happen if the APE isn't getting valid video.

I'm pretty sure that regardless of appearances the rear view camera feed on AP2/AP2.5 was deserialized, and then split/replicated in order to send it to both the APE and the MCU. This meant there was two ISP's, and one of the ISP's could be working while the other one wasn't. This meant with MCU1 you didn't know if the backup camera was working on the APE.

With my Model 3 I wouldn't say the rear-view camera feed has been always working on startup, but if it's working on startup it seems reliable. At least over the first couple thousands miles since I got the car.

 

[DJP31]

I don't know if it will work for you, but I noticed that if you approach the car and quickly turn on the backup camera view on the central screen (usually by turning into reverse gear) - there's a much higher chance the camera would work for autopilot detections. If you don't it usually flakes out within a minute of the car staring up.

Even when it does work you pretty much need to have a car stopped behind you at a traffic light for it to show anything.

I selected reverse gear and in heavy nose to tail slow traffic, even with a car very close behind me there was no image. I stopped for a while and on returning to the car again selected reverse prior to engaging drive and this time, under the same conditions, an image directly behind me appeared. This lasted for about 5 minutes before the image disappeared, and didn't reappear despite very slow moving traffic and the car following me being extremely close at times. I also brought up the rear camera on the MCU to see if that triggered it - it didn't.

During the same drive I had the PIN pad appear on the MCU, and having opened the roof when I tapped the car icon to enable me to close the roof, it was unresponsive. An MCU reboot cured that.

S75D AP2 MCU1 42.3

 

[verygreen]

I selected reverse gear and in heavy nose to tail slow traffic, even with a car very close behind me there was no image.

I did not mean during driving. What I mean is: when you come to your car as it still sits sleeping in your garage/driveway/parking lot before you do the drive, hop into the car and turn on reverse gear/backup camera right away instead of "get into the car, do some stuff for quite a while, then drive away" like the second part of that message.

This trick only works when autopilot computer restarts so once the feed is lost during driving the only way is to have the autopilot computer restart by having a decently long stop (with you not in the car or nearby) or the car poweroff + wait for 2 minutes before restarting.

How long the backup cam works when it does work seems to be the luck of the draw, sometimes it might last for over an hour, sometimes it still dies withing a few minutes indeed. I am not sure if bringing the backup cam image from time to time would help it to last longer as I did not do too many experiments in this area. I am glad to hear at least the initial trick seems to be not just my imagination.

 

[DJP31]

I did not mean during driving. What I mean is: when you come to your car as it still sits sleeping in your garage/driveway/parking lot before you do the drive, hop into the car and turn on reverse gear/backup camera right away instead of "get into the car, do some stuff for quite a while, then drive away" like the second part of that message.

This trick only works when autopilot computer restarts so once the feed is lost during driving the only way is to have the autopilot computer restart by having a decently long stop (with you not in the car or nearby) or the car poweroff + wait for 2 minutes before restarting.

How long the backup cam works when it does work seems to be the luck of the draw, sometimes it might last for over an hour, sometimes it still dies withing a few minutes indeed. I am not sure if bringing the backup cam image from time to time would help it to last longer as I did not do too many experiments in this area. I am glad to hear at least the initial trick seems to be not just my imagination.

Thanks for coming back, and I could have been clearer in my reply about putting the car in reverse when driving lol.

If I'm understanding correctly, if I "jump in the car, select reverse gear, then select drive" I have a fighting chance the rear cam might operate. This certainly did work on my second test, unfortunately not for more than about 5 minutes though.

 

[verygreen]

If I'm understanding correctly, if I "jump in the car, select reverse gear, then select drive" I have a fighting chance the rear cam might operate. This certainly did work on my second test, unfortunately not for more than about 5 minutes though

well, it might be longer on other attempts.

 

[pkodali]

I have no idea how to warp the rearview camera to fit it in. Everything else is working nicely enough. Any suggestions?

 

[DJP31]

well, it might be longer on other attempts.

I've been driving normally (i.e. without engaging reverse first) and over the last few days the appearance of a vehicle directly behind me has been much more consistent.

I have also noticed my car has uploaded/downloaded significantly more data then usual over the last few days (since getting 42.3) - 100mb plus instead of the usual 10mb. Is there any substance in the belief that Tesla download minor bug fixes or patches and that I could be seeing the benefit of that viz a viz this issue?

 

[verygreen]

I've been driving normally (i.e. without engaging reverse first) and over the last few days the appearance of a vehicle directly behind me has been much more consistent.

I have also noticed my car has uploaded/downloaded significantly more data then usual over the last few days (since getting 42.3) - 100mb plus instead of the usual 10mb. Is there any substance in the belief that Tesla download minor bug fixes or patches and that I could be seeing the benefit of that viz a viz this issue?

nah, there are no updates other than what you see. The triggers are real but they don't modify the behavior.

I did not try 42.3 to say if it's any better with the backup cam, but looking at more footage from the backup cam that I amassed, I found that even when it does work, it's quite glitchy.

Look at this: bck-trim - Streamable (copy the file to local storage to look at it frame by frame)

pay attention to the jumpy cars in the top left corner. This is not artifact of my processing, that's the camera feed, and it seems to feed some stale data in places, up to 12 seconds stale! And the car then does detections based on that data. ewww.
(e.g. at 0:24 time mark you can see a ghost of the white car reappearing just for one frame, the white car that is long gone, but even before then plenty of jumpiness).

 

[lunitiks]

*sigh*