"We observed one vehicle improving in ability when driven over the same roads over time"

[EVNow]

I don't know how many of you follow SoylentBrown on twitter - but he had an interesting tweet the other day. Essentially it says, the car seems to learn the route it is driven on and drives better on AP after some time. From what is known generally about NN and what Tesla has told us, this seems not possible.

What is your theory of how this might be happening ? I've some ideas but I want to hear your ideas first.

SoylentBrown on Twitter

@BrownSoylent Aug 18

One final thought for everyone: We observed one vehicle improving in ability when driven over the same roads over time. But a 2nd identical Tesla did not improve... Until we repeatedly drove the 2nd car over the same course. Would indicate learning isn't universal/centralized...​

 

[paaltio]

For me on repeated routes there are difficult portions where there’s a lot of variance in how AP drives through them. The car acts like it sees a twist in the lane lines earlier than some other times, and then manages to navigate it better. It makes me briefly think it has improved. But then next time it’s right back to how it was before.

 

[diplomat33]

I've noticed AP both improve and regress over time on the same route that I drive daily. The fluctuations in AP performance happened from software updates to updates. Some updates will improve. Some will cause a slight regression. But I also see fluctuations in AP performance based on changes in the driving environment. For example, in one instance, it is a bright sunny day with no cars in front of me and AP handles lane keeping perfectly but in another case, there is a big truck in front of me partially blocking view of the left lane line and AP lane keeping wobbles a bit.

So no, I don't think that AP learns from driving the same route. People think it is based on misinterpreting what is happening on a small sample of data. In other words, you do a few drives, maybe the driving conditions happened to be easier for AP, so it handled the drive better, so you mistakenly associate the improvement with learning when it is not.

AP improves based on changes to the NN that happens at Tesla when they develop a better NN and then upload it to our cars in the next update. Driving the same route may help send more data to Tesla that in turn helps improve the NN. But the improvement is not happening locally on your car.

 

[boonedocks]

I drive the same ~90+ mile commute daily. I can honestly say the performance of AP has gradually gotten better buy by software updates and not by route traveled. Weather, sun light, clouds, time of day and position of the sun, traffic, AP can perform as “miraculous” or “just stop using it for to avoid pissing off other drivers” on any given day from months and years ago to today. All in all it has MUCH improved over time but not at all from “learning” anything about a particular route.

 

[EVNow]

So no, I don't think that AP learns from driving the same route. People think it is based on misinterpreting what is happening on a small sample of data. In other words, you do a few drives, maybe the driving conditions happened to be easier for AP, so it handled the drive better, so you mistakenly associate the improvement with learning when it is not.

May be you are unfamiliar with Soylent Brown. He leads a team that hacks/tests cars for various big investors / other OEMs. Think of him as verygreen++.

 

[diplomat33]

May be you are unfamiliar with Soylent Brown. He leads a team that hacks/tests cars for various big investors / other OEMs. Think of him as verygreen++.

So are you saying that our cars do learn on their own locally?

 

[EVNow]

So are you saying that our cars do learn on their own locally?

That's what he says - I'd not just dismiss his finding.

BTW, recently he got invited to visit GF3 in China. So, definitely not just another twitter user.

 

[diplomat33]

That's what he says - I'd not just dismiss his finding.

BTW, recently he got invited to visit GF3 in China. So, definitely not just another twitter user.

Thanks. I'm just a bit skeptical. I have not really seen any evidence that AP does learn locally. I drive the same route to work every day for over a year now in my Model 3 and yes, AP handles it extremely well but it has gotten better over time with updates, with a few regressions from updates and then bigger improvements with the next updates. So I credit the software updates, not local learning. Although it is possible that there is some local learning but that software updates mostly override any local learning over time or incorporates the local learning over time.

All, I am saying is that it might be possible that Soylent is seeing improvements and thinks it is local learning but the improvements are actually caused by something else entirely.

 

[EVNow]

All, I am saying is that it might be possible that Soylent is seeing improvements and thinks it is local learning but the improvements are actually caused by something else entirely.

Here is what I think they did.

His team used 2 "identical" Model 3s. One was driven over a particular route many times. When the other one was driven on it the first time - it behaved much worse than the first 3. This should not be the case unless a car is locally learning some things that only that car "knows". That learning is not shared with other cars. The second car also got better over time when driven on that route repeatedly.

ps : In other words, if you take another Model 3 (with same version of s/w) and drive on your every day route, you would see worse behavior. You could try this if you have a friend who can lend you their Model 3.

 

[diplomat33]

Here is what I think they did.

His team used 2 "identical" Model 3s. One was driven over a particular route many times. When the other one was driven on it the first time - it behaved much worse than the first 3. This should not be the case unless a car is locally learning some things that only that car "knows". That learning is not shared with other cars. The second car also got better over time when driven on that route repeatedly.

ps : In other words, if you take another Model 3 (with same version of s/w) and drive on your every day route, you would see worse behavior. You could try this if you have a friend who can lend you their Model 3.

If local learning is happening, it might explain the different AP behaviors that people observe. I've always wondered how folks can say that AP is terrible, yet I observe that AP is amazing on my route that I take every day to go to work.

I have observed that after a brand new update, AP sometimes regress a little bit but then seems to improve on its own a few days later (same update version). It could just be AP silently calibrating again after a new update or it could be local learning.

 

[Uncle Paul]

I have a private road that my X had trouble with when I got it. Drive it several times a week.

Don't know if it is software improvements or adaptation, now it gets down that road much better.

 

[EVNow]

I have a private road that my X had trouble with when I got it. Drive it several times a week.

Don't know if it is software improvements or adaptation, now it gets down that road much better.

I have observed that after a brand new update, AP sometimes regress a little bit but then seems to improve on its own a few days later (same update version). It could just be AP silently calibrating again after a new update or it could be local learning.

Here is my thinking.

Locally the car keeps track of where disengagements happen and where they don't. We know NN produces results with % probability. Using that probability the procedural code figures out what to do. For eg. at a particular place, NN might produce lanes that are straight with 70% probability and curved with 30% probability. If AP took the 70% route with some hesitation - and there was no disengagement, and this happens a few times, the procedural code would "learn" to take the straight path with little hesitation. Ofcourse the learning is happening outside the NN. Also, if there was a disengagement, that 70% probability will be discarded for 30% curved path. Basically each car can increase or decrease various probabilities that NN gives it over time. This can be easily done (from programming complexity perspective) and won't even take up too much space.

While I've not noticed the AP get better over time on my regular roads, I definitely see the AP being worse in unfamiliar roads with similar road conditions. This would point to the same kind of "learning".

Ofcourse, all this observation (all of us, including Soylent's team) could be just interpretation of something else that is happening which is not learning.
​