HW2.5 capabilities

[jimmy_d]

So my backup method of getting a definitive answer about usage of the additional NNs failed as well, so I broke down and went for the indirect way (for now anyway).

If you jack up debug to the vision task - it only tells you that it loads and initializes the main_narrow NN, not the other two.

The file locations was a bit reshuffled compared to prior releases so I it's possible somebody forgot to filter the new names from th efinal release, or it might also mean that the code to use the new NNs is really around the corner.

Is it possible that the other cameras don't run all the time?

EDIT:

Here's a weird idea: what if the repeaters, just one of them, turn on when you activate a turn signal? So basically it looks in your blind spot when you signal. Conceivably that would only happen during autosteer, I guess.

 

[scottrobertson]

Is it possible that the other cameras don't run all the time?

EDIT:

Here's a weird idea: what if the repeaters, just one of them, turn on when you activate a turn signal? So basically it looks in your blind spot when you signal. Conceivably that would only happen during autosteer, I guess.

Would that explain why we only see cars in other lanes when doing a lane change?

 

[verygreen]

Is it possible that the other cameras don't run all the time?

EDIT:

Here's a weird idea: what if the repeaters, just one of them, turn on when you activate a turn signal? So basically it looks in your blind spot when you signal. Conceivably that would only happen during autosteer, I guess.

But... That does not mean they'd need to load and init the NN every time, right? I imagine they'd load the network once at startup and only run it as need in those cases, otherwise latency is going to be kind of high.

 

[bswn1]

Thread of the year

 

[buttershrimp]

Would that explain why we only see cars in other lanes when doing a lane change?

Easy! Because if a tree falls in the woods with no one around, it doesn't make a sound.

 

[calisnow]

Yeah, I might have overstressed the 'everybody uses this - even college classes' bit. Of course the reason it's popular is that it's a really good network architecture. As you say, really efficient. The inception blocks that were invented for it are something like a 5x improvement over regular CNNs in terms of compute needs. It's slower to train, though, but that's a good tradeoff for an embedded application. Google's dev team did a hero's work on this network design, so why fix it if it isn't broken?

I give it 30 days or less before major press has mangled this into headlines of "Tesla uses Google code for self driving"

 

[KyleDay]

I give it 30 days or less before major press has mangled this into headlines of "Tesla uses Google code for self driving"

How is that wrong?

 

[calisnow]

How is that wrong?

You missing dee point. It's over there. Go catch it.

 

[Kant.Ing]

Off topic, but

I give it 30 days or less before major press has mangled this into headlines of "Tesla uses Google code for self driving"

And the part where it's a real intern or a genius in disguise.

 

[jimmy_d]

But... That does not mean they'd need to load and init the NN every time, right? I imagine they'd load the network once at startup and only run it as need in those cases, otherwise latency is going to be kind of high.

Makes sense - though I don't know how long it would take to load up a network and set it running. Is it the kind of thing that happens in 100mS? or in 10S? I know zero about how the system manages it's components. I get the impression from what you've said that the ap stuff loads at startup. If you had asked me a month ago I would have guessed that they loaded it and ran it when it was needed. I guess in retrospect it has to be running so that it can tell you when conditions are ok to switch over to using it.

But then conceivably the repeater cameras are different - maybe the system doesn't need them except when autopilot is actually running?

I've got no idea here - just throwing stuff at the wall. Probably I want to believe that they have started using the repeaters and am just looking for reasons why you haven't seen them run yet.

 

[jimmy_d]

How is that wrong?

It's only wrong (from their point of view) if nobody clicks on the clickbait headline.

 

[jimmy_d]

Would that explain why we only see cars in other lanes when doing a lane change?

Actually, that raises an interesting point - why the repeaters? My reflex was to imagine it was for lane changes, but maybe that's wrong. There could be some other use case for repeaters.

Does sonar plus forward camera give you total coverage of the side for lane changes, or do you need the b-pillars too? It kind of seems like you need the b-pillars for stuff that's a bit forward but not yet into the forward camera view, but maybe they are using the fisheye for checking the forward side lane in addition to using it for wiper activation?

Or how about parking? Is there a situation where the sonar isn't good enough for telling you what's next to you when you park? Or maybe when you're in traffic? Is it important to know that the thing to the left of you is a wall rather than a truck (which presumably the sonar doesn't know)?

 

[buttershrimp]

It's only wrong (from their point of view) if nobody clicks on the clickbait headline.

This just in... "Wozniak calls out Musk, says his momma so stupid, she put airbags on her computer in case it crashed."

 

[verygreen]

Makes sense - though I don't know how long it would take to load up a network and set it running. Is it the kind of thing that happens in 100mS? or in 10S? I know zero about how the system manages it's components. I get the impression from what you've said that the ap stuff loads at startup. If you had asked me a month ago I would have guessed that they loaded it and ran it when it was needed. I guess in retrospect it has to be running so that it can tell you when conditions are ok to switch over to using it.

But then conceivably the repeater cameras are different - maybe the system doesn't need them except when autopilot is actually running?

I've got no idea here - just throwing stuff at the wall. Probably I want to believe that they have started using the repeaters and am just looking for reasons why you haven't seen them run yet.

16:00:53.398748 15017 net.cpp:122] Initializing net from parameters:
16:00:53.523308 15017 net.cpp:437] Network initialization done.

So ~120ms, yes (that's on system startup when everything is mostly idle, might be longer on a loaded system of course).

the main_narrow NN must be running all the time to get you pretty pictures of cars on your IC (and they run it even in park when nothing is displayed - so that they can start displaying stuff right away when switched into drive?)
I imagine they'd want to be running repeaters all the time for blind spot monitoring (oftentimes it works even without blinkers activated - lots of people don't use blinkers).

The rainsensing NN - again you'd want that to be running all the time so you can detect rain the moment it hits, without any user input.

 

[jimmy_d]

16:00:53.398748 15017 net.cpp:122] Initializing net from parameters:
16:00:53.523308 15017 net.cpp:437] Network initialization done.

So ~120ms, yes (that's on system startup when everything is mostly idle, might be longer on a loaded system of course).

the main_narrow NN must be running all the time to get you pretty pictures of cars on your IC (and they run it even in park when nothing is displayed - so that they can start displaying stuff right away when switched into drive?)
I imagine they'd want to be running repeaters all the time for blind spot monitoring (oftentimes it works even without blinkers activated - lots of people don't use blinkers).

The rainsensing NN - again you'd want that to be running all the time so you can detect rain the moment it hits, without any user input.

I think it's wonderful how you back up your statements with data.

Really - it's wonderful.

 

[_jal_]

I think it's wonderful how you back up your statements with data.

Really - it's wonderful.

Us: "what also floats in water?"
Us: "rain"
Us: "very small rocks"
Us: "cider"
Us: "great gravy"
Us: "churches"
@verygreen: "a duck"

 

[buttershrimp]

I'm been having so much fun reading this thread, feeling so optimistic about the future of Tesla! Then suddenly a thought occurred to me, "where the heck are @AnxietyRanger and @Bladerskb?

 

[bswn1]

I'm been having so much fun reading this thread, feeling so optimistic about the future of Tesla! Then suddenly a thought occurred to me, "where the heck are @AnxietyRanger and @Bladerskb?

 

[buttershrimp]

Beetlejuice Beetlejuice Beetlejuice!

 

[verygreen]

So... I was just decompiling the code that loads those NNs to get a definitive answer on what NNs are actually loaded when (still don't know), but another thing jumped at me that I did not notice when I was looking at this before.

So cameras 1 and 2 load main_narrow model - this much we know.
Now, cameras 3 and 8 load the fisheye model (so backup camera? really? I guess both are wide angle cams, but still....)
and now the kicker: cameras 4, 5, 6 and 7 are all loading the repeaters model (that used to be called "sides" in previous releases of the code) - that's really weird since the images you'd see from those cameras is kind of very different. I'd probably be less surprised if they had same model for main and repeaters I guess.

Anyway, so this code is called with a list of indexes of cameras to init NN models for and the way it's derived is pretty convoluted, so I've yet to understand it (it does not help that I don't really understand arm64 assembly all that much too).

  if ( ncameras )
  {
    i = 0LL;
    while ( 2 )
    {
      switch ( cameras_list[i] ) )
      {
        case 1:
        case 2:
          std::string::string(proto_path, "/opt/autopilot/share/vision/");
          std::string::append(proto_path, "main_narrow.prototxt", 0x14uLL);
...
        case 3:
        case 8:
          std::string::string(proto_path, "/opt/autopilot/share/vision/");
          std::string::append(proto_path, "fisheye.prototxt", 0x10uLL);
...
        case 4:
        case 5:
        case 6:
        case 7:
          std::string::string(proto_path, "/opt/autopilot/share/vision/");
          std::string::append(proto_path, "repeater.prototxt", 0x11uLL);
...
       }
       if ( ncameras != ++i )
         continue;
       break;
   }