AP2.0 Cameras: Capabilities and Limitations?

[verygreen]

There's still this matter of the mystery "lb" device.
Are the SoCs on a dual setup represented as two different nodes? Could lb be the other part of it that's just off for now?

 

[chillaban]

No reasons to worry. I just mean that the actual ARM cpus are as underpowered as ever, running gzip -1 on the raw image dump takes ~1.5 seconds (including the startup time) so we cannot use it for any sort of heavy lifting like that, esp if e compete with other tasks for cpu time. I am sure Pascal stuff is powerful and all that, but we cannot really use it for our purposes because autopilot is using it already and it's not like a general purpose cpu.

Do you have any control over whether you run on the Cortex-A57 cores or the Denver? The A57 core is extremely extremely simplistic and weak. It's designed to be the little cores in ARM's AMP design. I'd expect the Denver cores to be as fast as a 4 year old desktop processor. But based off how the comments refer to them as the "vision" cores, I'd imagine they've set up cgroups or some other affinity mechanism so that the A57's are what you're using for random background tasks like poking around at the shell.


P.S. The commented out cpufreq hacks seem wrong/historical…. The boot spew seems to show 6 cores, with the A57's detected first.

 

[verygreen]

Do you have any control over whether you run on the Cortex-A57 cores or the Denver? The A57 core is extremely extremely simplistic and weak. It's designed to be the little cores in ARM's AMP design. I'd expect the Denver cores to be as fast as a 4 year old desktop processor. But based off how the comments refer to them as the "vision" cores, I'd imagine they've set up cgroups or some other affinity mechanism so that the A57's are what you're using for random background tasks like poking around at the shell.
P.S. The commented out cpufreq hacks seem wrong/historical…. The boot spew seems to show 6 cores, with the A57's detected first.

The cpufreq IS correct. in /proc/cpuinfo I now see that cpus 1 and 2 are different from 0, 3-5.
We can also see the taskset -p 6 used in the sata logging referenced (That I initially missed). I guess the vision threads do their own affinity to those cores since the startup scripts don't seem to be externally binding them anywhere.
The vision task is using a lot of cpu, 6+ threads (2 of them the camera threads), so those cpus might not have all that much extra capacity to do additional compression.

 

[bjornb]

The firmware image is a VERY stripped down linux rootfs, consisting mostly of busybox (+ all the autopilot thingies + some other related stuff + surprisingly full alsa setup (autopilot plans to output to audio? or access microphones? or both?))
It does not appear to be based on ubuntu or anything else I recognize readily.
Has custom looking init stuff with custom looking logging infrastructure (sv, svlog - rings any bells?)
Kernel is "3.18.21-rt19"
6 arm64 cpu cores ("Cortex A57") + the Pascal GPU.
/proc/cpuinfo is really unspecial.
They have sshd, but it's "locked down" to only accept key auth (root has public ssh rsa key embedded, no signs of private key anywhere, perhaps somewhere at the mothership?)
The initial lockdown is done on te first boot (triggered by the cid).

There's still this matter of the mystery "lb" device.
Are the SoCs on a dual setup represented as two different nodes? Could lb be the other part of it that's just off for now?

BTW, HW2 systems added 2 more computer systems (As in network-accessible) compared to HW1, one is eap itself, the other one is cryptically labeled lb, but I have no idea what is it and when is it on if ever.

Thanks!
Do you have any specs on the 'lb' device/host as well? Similar hardware?
Is the first device actually named 'eap'?

Regaring sound, according to the wiring diagrams the center speaker is connected through the Autopilot ECU, I assume for playing the autopilot related sounds/alerts (the connectors are labeled 'alert speaker' in/out).

Nvidia has presented the PX2 like this:

I have speculated earlier in another thread that the Tesla HW2 ECU is not a fully equipped PX2 (like the diagram above) based on the limited cooling (two fans + heatsink), while the development PX2 is shown with two tegras on top and two discrete pascals on the bottom side.
Based on the supplied log and what others have said above we have only seen one Parker SOC so far (for example TEGRA A above) on the 'eap' device. Does the 'lb' device also have a Parker SOC?
I did not see anything related to separate discrete GPUs, but I must admin I am not an expert interpreting these logs, I do have linux experience, but I usually dont have to deal with the low level hardware...

 

[verygreen]

hm, cloudflare is eating my reply, I woder how to figure out what is it it does not like?

 

[verygreen]

Anyway important parts - for lb I only know the name and adress, but it's not on ever as far as I can tell.
the diagram noting A-B link is gige seems to imply B host is a separate node, so a high candidate for this LB stuff.

EAP node is actually called ape.

 

[chillaban]

BTW, I forgot to say, thank you @verygreen and others, for digging into this. This has been literally one of the best TMC threads I've read since joining.

 

[AnxietyRanger]

Agreed, a great thread! And in that data gathering spirit I always gush about, except here it actually has happened. Thanks, guys.

Let us continue...

 

[bjornb]

Anyway important parts - for lb I only know the name and adress, but it's not on ever as far as I can tell.
the diagram noting A-B link is gige seems to imply B host is a separate node, so a high candidate for this LB stuff.

EAP node is actually called ape.

Yeah, that Gig-E link does indicate that A and B parts may be separate.
Do you have IPs and MACs for ape and lb? Are the MACs consecutive or similar which could indicate that they reside together perhaps?

Regarding the name ape, AutoPilot Ecu perhaps? They use that name in the wiring diagrams..

 

[verygreen]

the IPs and names are part of the file in the etc (hosts) that if I mention it by full pathname seems to be triggering the cloudflare warning

ape is .103 and lb is .104

APE is only on if the car is not locked or if you are driving.
It shutdowns when in park for too long as well.
cid never turns off
I never observed lb on at all - need to figure out how to turn it on, I guess. Must be a gw command or gpio somewhere

 

[MorrisonHiker]

EAP node is actually called ape.

Regarding the name ape, AutoPilot Ecu perhaps?

I had an idea of what was coming several days before the offical AP2 hardware announcment. I posted messages saying that "people would be going APE" but nobody got the hint.
Unveiling on Oct 17th
Unveiling on Oct 17th

APE = Autopilot - Enhanced

 

[bjornb]

I have merged the forward cameras in two other settings as well. I didnt spend a lot of time adjusting the images so its possible to make more details visible, but it shows again the FOV difference between the front cameras:

parkinglot scene:

trafficlightstop scene:

Previous examples here:
AP2.0 Cameras: Capabilities and Limitations?

 

[chillaban]

trafficlightstop scene:View attachment 226429

This example might explain why AP2 doesn't assign cars to adjacent lanes yet on the UI. It looks like only the fisheye lens's view is appropriate for that purpose.

 

[Saghost]

This example might explain why AP2 doesn't assign cars to adjacent lanes yet on the UI. It looks like only the fisheye lens's view is appropriate for that purpose.

The front fisheye isn't supposed to be part of eAP, only FSDC (according to the original description emails that named the cameras used specifically.)

The wider conventional front camera should be a close match to the AP1 camera - and AP1 doesn't put cars on the driver's display until they hit about the 45 degree line from the camera/radar (but it also won't change lanes into them - probably from ultrasound?)

 

[chillaban]

The front fisheye isn't supposed to be part of eAP, only FSDC (according to the original description emails that named the cameras used specifically.)

The wider conventional front camera should be a close match to the AP1 camera - and AP1 doesn't put cars on the driver's display until they hit about the 45 degree line from the camera/radar (but it also won't change lanes into them - probably from ultrasound?)

That's a good point, but at the same time, I'm not sure the wide front view is as wide as the EyeQ3 view. Maybe it's an optical illusion, but it seems to me that the front main viewing angle wouldn't put cars on your screen until they're maybe 2 car distances in front.

 

[Bladerskb]

This example might explain why AP2 doesn't assign cars to adjacent lanes yet on the UI. It looks like only the fisheye lens's view is appropriate for that purpose.

Not quite I can confirm that ap2 main camera fov is the same. It's just that tesla is still way behind eyeq3

 

[chillaban]

Not quite I can confirm that ap2 main camera fov is the same. It's just that tesla is still way behind eyeq3

Curious: How are you confirming this?

 

[bjornb]

This example might explain why AP2 doesn't assign cars to adjacent lanes yet on the UI. It looks like only the fisheye lens's view is appropriate for that purpose.

There is a 'driving about' folder as well, I did browse through it briefly, and I think I saw some images more relevant for highway driving. I'll try to find a good example tomorrow.

 

[verygreen]

I just got 17.17.4, looking into the code I see that it's still only using the same two cameras.
Also this time they were nice to include a note that says if I want the debug clips and visualizations, I need to compile the autopilot with --anvil flag.

 

[lunitiks]

APE is only on if the car is not locked or if you are driving.
It shutdowns when in park for too long as well.

Field observation: When we opened to have a look at the ECU (car in P), the graphic card fans were running initially. After a while, however, they turned off

AP2ECU_fans_running

• lunitiks
• May 10, 2017

AP2ECU_fans_off

• lunitiks
• May 10, 2017