AP2.0 Cameras: Capabilities and Limitations?

[BigD0g]

Just found this thread today (I'm usually on the investor threads). I've got a Model S with AP1. Lots of interesting info here about AP2.0 still falling short of AP1. Musk has recently reiterated the plan for FSD coast to coast in December of this year, presumably with AP2.0. Are the opinions here that the probability of this happening so soon is very low? Or, is AP2 now starting to improve pretty quickly to where this seems pretty feasible? Thanks for any informed opinions on this!

The only AP1 experience I had was with my test drive, but from what I understand AP2 is almost to a point of AP1, but not quite there from a driving perspective. It's definitely not jumping by leaps and bounds each release. It more smaller tweaks / adjustments speaking from a pure driving perspective not talking about parking and auto high beams etc. Tesla has a long ways to go to handle curves and such, and let's not talk about local roads, those are a PITA. I can't imagine Elon doing a coast to coast FSD drive from CA -> NYC without changing the rules and say something like limited FSD or something else. It's just has way to far to go, and far far more corner cases it can't even handle yet.

However, all that being said, we only know what we see, for all we know there is a super secret code base with FSD already working and learning and they are just giving folks the parity software so they can focus on the FSD branch. However, that being said Elon also said users would see a fork in 3 months maybe and 6 months definitely (Elon Musk on Twitter). We'll six months is up in three days and we aren't even at parity and we definitely have no notion of any EAP or FSD features.

I would place the odds of a coast to coast FSD drive by Tesla at 5 - 10 % maybe. I really hope I am wrong though and they have some other key branch that's working and already driving itself around, but I seriously doubt it. The silence from Tesla on this is deafening.

 

[bdy0627]

The only AP1 experience I had was with my test drive, but from what I understand AP2 is almost to a point of AP1, but not quite there from a driving perspective. It's definitely not jumping by leaps and bounds each release. It more smaller tweaks / adjustments speaking from a pure driving perspective not talking about parking and auto high beams etc. Tesla has a long ways to go to handle curves and such, and let's not talk about local roads, those are a PITA. I can't imagine Elon doing a coast to coast FSD drive from CA -> NYC without changing the rules and say something like limited FSD or something else. It's just has way to far to go, and far far more corner cases it can't even handle yet.

However, all that being said, we only know what we see, for all we know there is a super secret code base with FSD already working and learning and they are just giving folks the parity software so they can focus on the FSD branch. However, that being said Elon also said users would see a fork in 3 months maybe and 6 months definitely (Elon Musk on Twitter). We'll six months is up in three days and we aren't even at parity and we definitely have no notion of any EAP or FSD features.

I would place the odds of a coast to coast FSD drive by Tesla at 5 - 10 % maybe. I really hope I am wrong though and they have some other key branch that's working and already driving itself around, but I seriously doubt it. The silence from Tesla on this is deafening.

Those sure aren't odds I would want to bet my own money on. Perhaps we will hear more next Friday with the M3 reveal. Thanks!

 

[BigD0g]

Those sure aren't odds I would want to bet my own money on. Perhaps we will hear more next Friday with the M3 reveal. Thanks!

I suspect next Friday will be all about features and functionalities of the M3 range, charge time etc. But one can hope =).

 

[JeffK]

Who knows! It is possible that Tesla are building FSD as a completely different product to EAP, and is developing it seperately in parallel. It is equally possible that what we see with EAP now is pretty much all there is.

That's what I feel as well, it makes the most sense.

 

[verygreen]

That's what I feel as well, it makes the most sense.

This is somewhat supported by code drops of neural net stuff from what appears to be a totally different codebase.
But it's a wild speculation either way.

 

[mrkisskiss]

I think it must be a completely different codebase... the FSD challenge is pretty different, overall. We know that they take imagery from all of us AP2 guys, which they probably use to improve detection and train the models based from the different camera perspectives. But, the detection of objects is already very good - of course, the better it gets the, erm, better... but the detection problem is largely solved.

Localisation and path planning are huge missing bits right now. We see basic path planning in AP2 today - but it's not really taking into account anything other than its immediate surrounds via vision perception. FSD will have to do things like localise itself within a map (I think @verygreen discovered there's Mapbox rails in there - but only for Fremont?!) and then figure out the actual driving rules -- this is the hardest AI challenge, rather than detection or even localisation. Actual driving abilities - how it merges into traffic, how it deals with negotiations, cutting into queues, not getting cut up all the time in heavy traffic etc... they are all hard problems, and Tesla must be working on this in a separate codebase somewhere - since we saw genuine evidence of this way back in October.

Of course, I doubt it'll be level 5 - at best it'll be sort of level 4 features officially classed as level 2, because there's a steering wheel and we'll still probably have to use it occasionally.

NVidia's CEO is on record saying that the Xavier platform (which is one step above DrivePX2 in terms of efficiency) takes about 30W. They reckon that power draw can get them to level 4, eventually. Level 5 robotaxi is a much bigger problem, and they'll probably require compute that sucks around 1000W to do that. He was making this point with regards to parallel GPU vs sequential CPU processing, which he claims would require an even higher power draw - 3000-4000w - to execute the required models - too inefficient to be useful in an EV (or any car).

 

[Bladerskb]

This is somewhat supported by code drops of neural net stuff from what appears to be a totally different codebase.
But it's a wild speculation either way.

is this new and from the latest update?

 

[verygreen]

is this new and from the latest update?

no, for the limited time I have been watching this thing, the trained net and convolution kernels are procured from somewhere else as "drops", these drops might persist across multiple releases (i.e. 17.17.x to 17.24.x had used the same drop)

 

[Bladerskb]

no, for the limited time I have been watching this thing, the trained net and convolution kernels are procured from somewhere else as "drops", these drops might persist across multiple releases (i.e. 17.17.x to 17.24.x had used the same drop)

What you think about this? the largeness of the latest update. Is this unusual or are major updates usually 1GB?

I just got it on my AP2 Model X, and I noticed something a little odd. I always peek at the download sizes of these updates and I noticed that this particular update had a massive download size, which is odd for something that should be a minor release. I'm used to updates being around 100-300 MB but this update was a whopping 1.2 GB.

Here is a screenshot:

And here's the current update compared to the last one I received (17.26.76). The large spike in the graph on the right is 17.28 c528869 and the 2nd highest bar between July 2 - 9 is 17.26.76. The difference is huge, and the bar from today should actually be double that size (the app just didn't refresh it yet):

 

[verygreen]

What you think about this? the largeness of the latest update. Is this unusual or are major updates usually 1GB?

The update is two parts:
part 1: download the patch between what you currently run and the new release. (BUT sometimes the instructiosn tell to download the whole new image for cid, the full size for 17.26.79 was 843255872 bytes)
part 2: on ape2+ cars only, separately download ape firmware in full. For 17.26.79 ape firmware is 386281536 bytes).

It's also possible the new ape firmware is now getting maps (I don't have it so cannot know for sure), this is not yet cid maps update (those are ~4G).
It's also possible he just got a few snapshots scheduled that intersected with update download.

Scratch that,my car was just called to get the new update, be back soon with more details.

Update: From 17.26.76 to this new release, the patch size on the cid is only ~36M
APE firmware is 394092608 bytes long
Neural net model is the same as 17.26.76, in line with the changelog that concentrates on AP1 improvements.

Ooh, more framework preparations to download autopilot specific maps.

 

[bswn1]

Ooh, more framework preparations to download autopilot specific maps.

What does this mean exactly?

 

[verygreen]

What does this mean exactly?

More code is now present in the autopilot image, in particular map downloader was added, though it is not yet called.

 

[J1mbo]

Update: From 17.26.76 to this new release, the patch size on the cid is only ~36M
APE firmware is 394092608 bytes long
Neural net model is the same as 17.26.76, in line with the changelog that concentrates on AP1 improvements.

That'ts interesting. There has been a big variation in "finesse" in 17.26.x releases, suggesting either that it's not the NN that is responsible for actual control of the car, or that the NN is a fixed size for the feature set.

 

[mrkisskiss]

The NN is definitely not responsible for the control of the car - the model it's downloading is basically for image recognition/object classification. You wouldn't do control with a neural net at this stage - way too unpredictable.

 

[BigD0g]

More code is now present in the autopilot image, in particular map downloader was added, though it is not yet called.

Two questions if you don't mind...

1. What the heck is ape2+????
2. I'm assuming the code is in C? Are you just recompiling the headers to see what calls are made?

 

[croman]

@verygreen -- I wanted to say thanks for letting us know about everything. You don't have to and it is greatly appreciated. Tesla should be doing the same but there's no point in wasting bits on that.

 

[BigD0g]

@verygreen -- I wanted to say thanks for letting us know about everything. You don't have to and it is greatly appreciated. Tesla should be doing the same but there's no point in wasting bits on that.

Ditto! Thank you @verygreen Just awesome !!

 

[verygreen]

Two questions if you don't mind...

1. What the heck is ape2+????
2. I'm assuming the code is in C? Are you just recompiling the headers to see what calls are made?

ape2+ is ape2 and ape2.5 (HW2 and HW2.5 (currently present in Model 3, but possibly making its ways to Model X and S at the time of final model 3 delivery?))

the code is all c++, but while I sometimes dive at that level to investigate some particular bits, you can see a bunch of stuff even without going that low, there's process separation with different processes doing different tasks with somewhat descriptive names, as one example.

 

[mitchellh3]

More code is now present in the autopilot image, in particular map downloader was added, though it is not yet called.

This likely means they're laying groundwork for when the AP firmware downloads HD maps for AP usage, right? These maps would presumably be very different from CID maps since well... a human doesn't need to look at it and it needs a lot more weird metadata targeted towards AP usage.

 

[verygreen]

This likely means they're laying groundwork for when the AP firmware downloads HD maps for AP usage, right? These maps would presumably be very different from CID maps since well... a human doesn't need to look at it and it needs a lot more weird metadata targeted towards AP usage.

It's possible the maps on the cid and ap are going to be at least somewhat shared, but without seeing a sample of ape maps I cannot know for sure.
A lot of info is useful on the cid too. Every time you see the information about the number of lanes on the navigation screen (for purposes of which one to use for your exit) - the number of lanes is useful for ape. And I assume there are some coordinates for every one of them since navigation supposedly knows which lane are you in too.