2017.44 AP worse than .42

[Padelford]

I have an AP “torture” course south of Seattle that I’ve used to evaluate AP 2.0 performance since 2017.36. The course has two right-lane exits after freeway turns, and the second exit occurs just after a rise and drop of the freeway, plus a sweeping right turn. There is also a freeway rise and right turn without an exit. I set the vehicle to the speed limit (60 MPH) before starting each test run in the right lane.

2017.36 did poorly on all three, losing lane control on the rise and turn and veering into the left traffic lane. It performed particularly poorly on the second exit where the vehicle nearly hit an abutment head-on separating the exit from the main freeway. It oscillated left and right passing the first exit but got back to the main freeway lane on its own (but the experience was a bit scary).

2017.42 was a big improvement over .36. Both exits were largely ignored (with some weaving on passing the second exit), and the vehicle maintained better lane control thru the rise & right turn.

Somehow, 2017.44 performs poorer than .42 in this test on the second exit. The vehicle gets confused like with .36 and heads for the abutment. I’ve tried the test with two different MX’s and observed the same result. The vehicles both ignored the first exit and the rise & right turn.

I’m puzzled why Tesla would decrease AP 2.0 performance in .44. The combination of rising and falling freeway pavement, a right turn after the peak plus an immediate right exit is a potentially dangerous combination while using AP 2.0 and 2017.44.

 

[HookBill]

My experience has not been negative. .44 has had some minor regressions from .42 but overall my car has improved more than it has gotten worse. Just my own observations using my car. YMMV.

 

[buttershrimp]

My .44 is much superior than .42... I'll post a video ....

 

[vandacca]

I have an AP “torture” course south of Seattle that I’ve used to evaluate AP 2.0 performance since 2017.36. The course has two right-lane exits after freeway turns, and the second exit occurs just after a rise and drop of the freeway, plus a sweeping right turn. There is also a freeway rise and right turn without an exit. I set the vehicle to the speed limit (60 MPH) before starting each test run in the right lane.

2017.36 did poorly on all three, losing lane control on the rise and turn and veering into the left traffic lane. It performed particularly poorly on the second exit where the vehicle nearly hit an abutment head-on separating the exit from the main freeway. It oscillated left and right passing the first exit but got back to the main freeway lane on its own (but the experience was a bit scary).

2017.42 was a big improvement over .36. Both exits were largely ignored (with some weaving on passing the second exit), and the vehicle maintained better lane control thru the rise & right turn.

Somehow, 2017.44 performs poorer than .42 in this test on the second exit. The vehicle gets confused like with .36 and heads for the abutment. I’ve tried the test with two different MX’s and observed the same result. The vehicles both ignored the first exit and the rise & right turn.

I’m puzzled why Tesla would decrease AP 2.0 performance in .44. The combination of rising and falling freeway pavement, a right turn after the peak plus an immediate right exit is a potentially dangerous combination while using AP 2.0 and 2017.44.

AI software is not like traditional software. There is no human trying to fix an algorithm on a particular scenario. No human knows how the AI software works or how well it will perform after any changes to the neuro-net or training. Sometimes it improves, and sometimes it gets worse in order to make it handle more generic scenarios.

It's like truing a bicycle wheel. As you tighten one spoke to straighten the rim, it'll pull the rim in the other direction at the opposite end of the wheel. You have to continually keep tweaking the spokes until the wheel slowly becomes completely true.

 

[verygreen]

Given that there were no NN changes between .42 and .44 (other than the unused wide angle cam NN) whatever changes there might be were due to adas maps change (if you still have 17.42 car - try that same course again and iss if it degraded too) or they just changed something on software side about how they interpret the NN and radar data, I guess.

 

[ChrML]

AI software is not like traditional software. There is no human trying to fix an algorithm on a particular scenario. No human knows how the AI software works or how well it will perform after any changes to the neuro-net or training. Sometimes it improves, and sometimes it gets worse in order to make it handle more generic scenarios.

It's like truing a bicycle wheel. As you tighten one spoke to straighten the rim, it'll pull the rim in the other direction at the opposite end of the wheel. You have to continually keep tweaking the spokes until the wheel slowly becomes completely true.

That's one approach, but I hope Tesla hasn't went for the 100% NN undebuggable blackbox approach. It might be that the publicly available autopilot uses some temporary software like that.

I think Tesla would split the logic into fairly small manageable blocks. Where each block is a NN, algorithms, or a combinations of both. In this way the developer might not know exactly how the Tesla recognizes signs for example, but he can at some later point improve it, retrain it or even replace the entire block and there is no way it can affect other parts of the autopilot (e.g. lane holding).

To be honest I cannot see how a end-to-end neural network FSD would work at this point. Yes, you can train it quite fast to work 99% of the time, but then there's an edge case. Fix it with some additional training, and you cause another edge case or unknown sideeffects. End-to-end neural networks will not work until generalised artificial intelligence is good enough, ie like teaching a kid how to drive. With enough practice he masters it. But that is probably 10 years down the road at minimum, anyones guess really. But with a properly managed code I can see a really good FSD being developed in 1-1.5 year.

 

[lunitiks]

or they just changed something on software side about how they interpret the NN and radar data, I guess.

I'm really interested in this particular 'software side'... Where is it, what's it called, what language does it speak? Is it some 'control logic' file(s), separate from e.g. the power steering firmware? How does this tie together do you think? I mean what happens after NN decides it's a car or a curb ahead of you?

 

[Padelford]

Given that there were no NN changes between .42 and .44 (other than the unused wide angle cam NN) whatever changes there might be were due to adas maps change (if you still have 17.42 car - try that same course again and iss if it degraded too) or they just changed something on software side about how they interpret the NN and radar data, I guess.

My vehicle with 2017.42 had no problem with the hill/exit that 2017.44 essentially failed.

 

[Padelford]

AI software is not like traditional software. There is no human trying to fix an algorithm on a particular scenario. No human knows how the AI software works or how well it will perform after any changes to the neuro-net or training. Sometimes it improves, and sometimes it gets worse in order to make it handle more generic scenarios.

It's like truing a bicycle wheel. As you tighten one spoke to straighten the rim, it'll pull the rim in the other direction at the opposite end of the wheel. You have to continually keep tweaking the spokes until the wheel slowly becomes completely true.

I almost added a phrase to the original post, "Perhaps in updating the AP software, they sought to improve one aspect but degraded something else." They may need more free parameters in their AI solution space to avoid ending up with a mediocre overall solution that can't be optimized without sacrificing something else.

 

[scottf200]

My vehicle with 2017.42 had no problem with the hill/exit that 2017.44 essentially failed.

I think there is some confusion. I think he is saying that if you *still* had the .42 access that it would have changed "due to adas maps change" and not NN since there were no change to NN from 42 to 44.

 

[Padelford]

I think there is some confusion. I think he is saying that if you *still* had the .42 access that it would have changed "due to adas maps change" and not NN since there were no change to NN from 42 to 44.

I'm curious - how does someone know that the NN/AP software parameters in 2017.44 did not change? Has someone done a full line-by-line compare between .42 and .44 firmware code?

Also, does the phrase "adas maps change" refer to activating vectorized Google map data discussed in another thread?

 

[buttershrimp]

I hate not having the actual cliff notes of the Tesla Autopilot Team... can't imagine what sort of placebos I'd be hallucinating in the most recent firmware update if it weren't for @verygreen, @JimmyD, or as I call them, Tesla's Woodward and Bernstein... or Tesla's Assange and Snowden... or well, wait.... or is it Chelsea Manning? Oh, I can't remember, but I think Trump tweeted about them, "@verygreen and @JimmyD, if you are listening, please find the thousand of missing Clinton emails in the latest firmware update"

 

[scottf200]

I'm curious - how does someone know that the NN/AP software parameters in 2017.44 did not change? Has someone done a full line-by-line compare between .42 and .44 firmware code?
Also, does the phrase "adas maps change" refer to activating vectorized Google map data discussed in another thread?

Search for various post by verygreen and you'll quickly realize his technical abilities and sources of information. He is well known here for a long time. He provides quit a bit of technical information and is clear when he making guesses vs empirical evidence.

 

[verygreen]

I'm curious - how does someone know that the NN/AP software parameters in 2017.44 did not change? Has someone done a full line-by-line compare between .42 and .44 firmware code?

It's relatively easy to gain access to in-car firmware (as they call it at Tesla) to examine what it does and what changed.
In particular it's really easy to see if anything in the NN changed or not (the actual software changes are harder to tell what they are of course).

 

[cybergates]

2017.42 actually works quite well and (best one yet) and for some reason I haven't 2017.44 but I shouldn't complain. Actually 2017.42 saved me today from trouble - during an auto-lane change the car started to move over but suddenly the car pulled back (which it used to do a lot in prior versions but was working smoothly - I thought it was a bug when I saw a car had moved in really close into my blind spot. Now that's AI at its best!

 

[Padelford]

I note that on ev-fv.com, the distribution of 2017.44 has greatly slowed down compared to the distribution history of 2017.42. I could speculate that Tesla found a bug and is stopping distribution to fix it.

 

[verygreen]

I note that on ev-fv.com, the distribution of 2017.44 has greatly slowed down compared to the distribution history of 2017.42. I could speculate that Tesla found a bug and is stopping distribution to fix it.

You would think if the bug was severe, there would be at least some complains about some breakage?

 

[Padelford]

Search for various post by verygreen and you'll quickly realize his technical abilities and sources of information. He is well known here for a long time. He provides quit a bit of technical information and is clear when he making guesses vs empirical evidence.

Thanks for that info. I haven’t been a MX owner very long & caught that.

 

[Padelford]

You would think if the bug was severe, there would be at least some complains about some breakage?

The failure I experienced may be a “corner case” that hasn’t received a lot of attention. It involves three conditions happening simultaneously.

I could agree with others that 2017.44 is in general a really good update.

 

[chillaban]

I have to say, in general I have been finding .44 as good as if not slightly better than .42, but I’m sure there’s counterexamples there too. The changes seem very minor between those releases.