AP disengaged while driving - Radar Failure (release 2019.8.3)

[GolanB]

I had the following experience today while driving. Two days ago, I received 2019.8.3 - and today I decided to test AP on my way home. I should point out that the car is very clean, and there is no obstruction to the radar or any of the sensors.

About 30 seconds after engaging auto-pilot, AP disengaged suddenly and got a warning to take over driving immediately (which I promptly did).

I tried re-engaging AP and got the warning - "Cruise not available - Reduced front radar visibility" and "Autosteer temporarily unavailable".

The vehicle also lost tracking of the cars surrounding it, and the display only showed my car.

I attempted to submit a bug report, but the MIC was not receiving input (it could no longer hear me).

Approximately 10 minutes later, I started noticing cars could be seen again by the sensors, and I was able to re-engage autopilot.

I called Tesla Customer Service (1-hour wait). The technician had me recalibrate the sensors by going into park, pulling the steering wheel all the way to the left, then all the way to the right, then center before resetting (two steering wheel buttons) and then shutting down.

I was told this is either a software bug or a radar sensor failure. If it happens again I will schedule service. I asked the support representative whether or not anyone else reported this issue with this release, but he said I'm one of the first to be on the release, and he wasn't aware of other reports yet.

Has anyone else reported a similar issue?

(Sorry for the poor image quality, its a screenshot of a video)

 

[opiuman]

There are some reports that this is location based due to interference as well.

 

[GolanB]

There are some reports that this is location based due to interference as well.

Interesting theory, although I can tell you that the interference would have had to stretch between NJ and NY. It occurred right as I got into the Holland Tunnel on the NJ side, and continued until I was well in Manhattan more than 10 minutes later. It doesn't feel like interference in this case.

 

[eli_]

Yeah this is looking like a regression in 2019.8.3, lots of people are hitting it on reddit, etc.

 

[mtndrew1]

Exact same happened to me yesterday and then it resolved itself while the car was parked at the office, but when Autopilot returned there was no NoA.

This morning NoA came back and everything was back to normal. Super weird.

 

[GolanB]

Exact same happened to me yesterday and then it resolved itself while the car was parked at the office, but when Autopilot returned there was no NoA.

This morning NoA came back and everything was back to normal. Super weird.

Did your issue have all or some of the same attributes mine did?
Did your car warn you that the front-facing radar was experiencing an issue?
Did the visualization of the cars around yours disappear?

FYI, Cool shot of the EV1. Did you own one?

 

[mtndrew1]

Did your issue have all or some of the same attributes mine did?
Did your car warn you that the front-facing radar was experiencing an issue?
Did the visualization of the cars around yours disappear?

Yes to all, exactly as you described. “Reduced front radar visibility” was the warning combined with “cruise unavailable” and “autopilot unavailable.” No cars displayed on the screen after the red hands warning.

It took me three times to file a bug report because the car wasn’t understanding what I was saying.

All back to normal this morning.

 

[GolanB]

Odd indeed - this feels like a software related issue, but we'd need more confirmation from other users. Not yet validated, but someone on our local Tesla WhatsApp group reported that users who had not yet installed this release but that had notifications for it are no longer able to schedule the update (release pulled?). I don't see evidence of this on TeslaFi yet, they had over 300 installs today.

 

[fasteddie7]

How was the weather? Here in Pennsylvania, if it's mid to high 30s (it was 37 at the time) the wet, heavy snowflakes cause this to happen all the time, after a few minutes it melts off and goes away on its own. There was a discussion on this in the FSD thread as with level 5 there would have to be a redundant radar to take over when this happens.

 

[GolanB]

How was the weather? Here in Pennsylvania, if it's mid to high 30s (it was 37 at the time) the wet, heavy snowflakes cause this to happen all the time, after a few minutes it melts off and goes away on its own. There was a discussion on this in the FSD thread as with level 5 there would have to be a redundant radar to take over when this happens.View attachment 391331

Weather was amazing today; clear skies and sun. Didn't need to run the heat, it was in the 50's.

 

[GolanB]

Seems like Tesla took inventory of the event; my vehicle uploaded approx 900mb of data to the Tesla Amazon Web Services S3 Storage Cloud just a short time after I arrived at my WiFi location. Hope they learn something important!

 

[M3Nicola]

I had almost the same scenario as you today, but received different error messages.

Running same software as you and similar situation that caused the errors. My car is not dirty or anything. Afterwards, I parked my car for a couple hours and I’m still seeing the error message.

 

[hckyplyr]

Reboot

 

[GolanB]

In my mind, this raises serious questions about the underlying architecture of the onboard autopilot computer and it’s integration with sensors (both radar and cameras.). One would hope that there would be enough built in redundancy to avoid this sort of immediate failure of autopilot- both on the computer logic side as well as with regards to sensor data from radar and cameras. If the issue is with the onboard computer, where is the secondary logic processing unit? If the issue is with radar input why didn’t the camera sensors provide backup. Why is there only one onboard computer, and why is there only one radar? If I recall on the space shuttle, there were more than 5 onboard computers and multiple sensors to handle multiple failures. We all know what went wrong with errant sensor data on both Boeing and Airbus aircraft. I’m also troubled by the fact that the Tesla support engineer could see the errors I encountered but was not able to tell me whether or not the autopilot computer experienced an issue or if the issue was with the sensors themselves. This failure is serious enough that there should be more analysis done and better communication. I would hope that each of you who’ve experienced a similar issue in this release reported the problem and indicated the severity of your concern. Had my hands not been in the wheel it’s possible there would have been a collision as the car was handling a curve at the time.

 

[TampaRich]

Only a thousand or so people have this version of the software. That’s a very small sample. We are all early adopters of this tech and need to treat it as such. Use caution when validating new software releases. Thanks for sharing your experience with the community to help raise awareness.

 

[Mark_T]

Only a thousand or so people have this version of the software. That’s a very small sample.

That really isn't true, TeslaFi isn't in all that many cars so don't look at the absolute numbers, look at the percentages and right now 19.8.3 is in over 36% of the TeslaFi 'fleet'.

Looks like this is the new general release right now, replacing 18.50.6 which is down below 30% now...

 

[dondpw]

I had the following experience today while driving. Two days ago, I received 2019.8.3 - and today I decided to test AP on my way home. I should point out that the car is very clean, and there is no obstruction to the radar or any of the sensors.

About 30 seconds after engaging auto-pilot, AP disengaged suddenly and got a warning to take over driving immediately (which I promptly did).

I tried re-engaging AP and got the warning - "Cruise not available - Reduced front radar visibility" and "Autosteer temporarily unavailable".

The vehicle also lost tracking of the cars surrounding it, and the display only showed my car.

I attempted to submit a bug report, but the MIC was not receiving input (it could no longer hear me).

Approximately 10 minutes later, I started noticing cars could be seen again by the sensors, and I was able to re-engage autopilot.

I called Tesla Customer Service (1-hour wait). The technician had me recalibrate the sensors by going into park, pulling the steering wheel all the way to the left, then all the way to the right, then center before resetting (two steering wheel buttons) and then shutting down.

I was told this is either a software bug or a radar sensor failure. If it happens again I will schedule service. I asked the support representative whether or not anyone else reported this issue with this release, but he said I'm one of the first to be on the release, and he wasn't aware of other reports yet.

Has anyone else reported a similar issue?

(Sorry for the poor image quality, its a screenshot of a video)


View attachment 391308 View attachment 391309

Yes I saw something very similar a few days ago. Called tech support and they grabbed the logs. All returned to normal after the car sat in the garage for 30-40 minutes.

 

[GolanB]

Since I was unable to get very much sleep last night, I spent some time thinking about the situation in more detail. The more I thought about the reliability of autopilot, the more concerned I became.

The first question that comes to mind is what level of reliability targets (or SLA's) does Tesla use when approaching the safety and reliability of autopilot. While it is impossible to hit 100%, in many industries (in particular in navigation computers on aircraft, but also communications and some hospital and vital organ facilitation equipment) those systems are designed with five "9's" of reliability, meaning they are expected to be functional 99.999% of the time or greater. To put this another way, this means that if the car was on the road for 24 hours a day, in one year this level of reliability would require less than 26 seconds of "downtime." In order to accomplish this level of reliability, it requires complete redundancy in all components and systems, from power to processing and software. It is not easy to get it right, and even with all systems in place, there can still be challenges in achieving "five 9's" of reliability.

One of the big challenges with the design of the recent Boeing 747 MAX aircraft is that they deployed software to compensate for compromises they made to the hardware architecture of the plane itself. Two of the primary compromises included the forward positioning of larger engines due to ground clearance problems where they'd normally be right under the wing (which causes the aircraft to unnaturally pitch upwards) and perhaps most egregious compromise was its reliance on a single "Angle of Attack" sensor which confused the onboard maneuvering augmentation computer (MCAS) and worked against the pilot. Although Boeing did apparently make models of the 737 MAX available that made use of more than one AOA sensor, it was not made standard! On top of this, Boeing did not ensure that pilots were properly trained on these systems and pushed the aircraft as part of the 737 fleet in order to avoid introducing it as an entirely new aircraft.

Boeing is now desperately rushing to reduce the MCAS response through a software fix when it is clear that it needs to do more (whether adding an additional AOA sensor, or building motion models from other sensors including from its pitot tubes which measures wind speed or perhaps accelerometer and positioning sensors).

Although these two systems bare very little resemblance to one another in practice, Tesla can learn much from Boeings mistakes. In the long run, the overall cost of implementing reliable systems are much lower than getting it wrong (both from human and economic factors).

In the communications industry, when building reliable systems, redundancy becomes a key factor. That redundancy is even extended to software, and it is not uncommon for two releases of code to be running simultaneously on a given network. In some sense, Tesla is examining the reliability of its software through a staggered release approach, but as far as we know, its onboard navigation computer is currently only capable of running a single release at a time.

Getting it right on the other hand can be extremely challenging, and not every software developer coming out of school today is necessarily versed in building highly reliable systems. Tesla's rapid rate of software change means that each release will have its own set of exposure and bugs. Adding complexity to the system also brings with it its own set of challenges, and it's usually best to keep these systems as simple as possible.

Has anyone on the forum studied Tesla's target architecture as it relates to its navigation systems? It would be very interesting to examine.

 

[bd7349]

This happens (“red hands on wheel” alert, beeping, autopilot/cruise not working for a few minutes) when the Autopilot computer crashes. Once it reboots (takes about 3-5 minutes from what I’ve seen) all Autopilot and cruise functionality returns. Happened to me back in December shortly after I took delivery, but has never happened since then in over 13,500 miles of driving (I use Autopilot A LOT). I agree there should be some level of redundancy for Autopilot, but maybe they’ve already addressed this with their Autopilot HW3 hardware.

There’s already multiple different computers for redundancy of basic drive systems (steering, acceleration/braking, turn signals, etc.) which is why you can reboot the main computer while driving with no problems, so I’m curious why Autopilot doesn’t also have a secondary redundant computer system to fall back on in case one crashes like in this case. I’m going to go with the assumption that Tesla’s Autopilot team has probably already thought of this/how to prevent it from happening. They’ll have to if they ever want FSD to be anywhere near fully autonomous.

 

[GolanB]

This happens (“red hands on wheel” alert, beeping, autopilot/cruise not working for a few minutes) when the Autopilot computer crashes. Once it reboots (takes about 3-5 minutes from what I’ve seen) all Autopilot and cruise functionality returns. Happened to me back in December shortly after I took delivery, but has never happened since then in over 13,500 miles of driving (I use Autopilot A LOT). I agree there should be some level of redundancy for Autopilot, but maybe they’ve already addressed this with their Autopilot HW3 hardware.

There’s already multiple different computers for redundancy of basic drive systems (steering, acceleration/braking, turn signals, etc.) which is why you can reboot the main computer while driving with no problems, so I’m curious why Autopilot doesn’t also have a secondary redundant computer system to fall back on in case one crashes like in this case. I’m going to go with the assumption that Tesla’s Autopilot team has probably already thought of this/how to prevent it from happening. They’ll have to if they ever want FSD to be anywhere near fully autonomous.

There is no question that there will need to be architectural changes to hardware and software for Tesla to get to autonomous driving in FSD- there is no way the current systems level of reliability would be enough, even if it learned to drive as good, or better than a human.

There has been one published diagram on HW v3, and it shows redundant inputs for multiple radars, as well as power, ethernet and other interfaces:

It still begs the question, even if the hardware is capable of running multiple processors, will the software be there to support it (?), and does it make sense to do everything in a single AP computer? What if the inputs were driven into two AP computers (in an active/active state, or in an active/passive state) where one either shares the load, or can immediately take over (within 50ms of a failure say).

Does this architecture mean that for those of us who've purchased FSD, we will need to come back not only for new AP hardware, but for a secondary sensors?

It seems to me that this would be a requirement above level 3 (for level 4 and 5 autonomous driving) which dictates that the car be able to handle all safety-critical driver functions and monitor roadway conditions for an entire trip.