Software Update - don't you find it intriguing how Tesla test/validate it?

[NeilOnCa]

I have been a happy S75D owner for more than half a year now. While I'm still getting anxious every two weeks hoping I can be the early adopter of a new software update, I suspect I'm definitely not the only who finds it intriguing to know a bit about how Tesla test/validate its software. Think about these points:

1. Tesla release a new (minor) version of software usually every two weeks - so the internal QA/Testing team has at most two weeks to test the current version before they have to start testing the next release. (While for traditional auto makers who usually release a firmware once a year, they have a full year to do tests!)

2. Although the changes are usually small between adjacent releases, the risks of regression are always there for complex systems like it in Tesla - think about the regression bugs recently: charging cable connection issue from 2018.14, air suspension issue from a release in 2017, etc. You think Tesla has figured out a way to do full regression test every two weeks while other auto makes need a full year to do the same?

3. The software are mission critical. They control functionality such as AP, auto-braking, response to acceleration pedal (e.g. chill vs regular mode, released through a software update), etc. Even the smallest bug can cause huge damage. One of the advantages Tesla has is the gradual releases to users in batch. But even for releasing to a small batch of early adopter, the potential risk needs either bravery or some amazing testing mechanism to overcome.

Anyways, being a software engineer myself, I have been more than impressed by how Tesla has been holding a high bar on the quality and speed of their software releases. Those small bugs aside, I haven't heard of any major incidents that were caused by bugs introduced by software updates (knock on wood).

 

[cwerdna]

FWIW, some companies update their apps every week or two. Example: Look at version history of Facebook on the App Store.

Facebook is also nuts (IMHO) in that the push to their web site (and I guess associated back-end services they (web site and client apps) depend on) TWICE a day: Ship early and ship twice as often (from 2012). It used to be once a day.

I talked to someone who works there a few years back when the were doing 2x a day and they said yeah and that their big releases are once a week.

On many other cars, esp. w/o any OTA update capabilities (most of the ones on the road), you'll be lucky to get any updates at all other than (often customer paid) navigation system map updates or updates w/an associated TSB or service campaign. Otherwise, there are no updates.

 

[bob_p]

Tesla cars have multiple processors. We are only seeing the software that is driving the dashboard and console display - which appear to have separate processors, since they can be rebooted separately.

Mission critical operations are running on other processors. That's evidenced by rebooting both the dashboard and console processors - while the car is moving - and everything works, except you don't have a visible display and you don't hear turn signals.

We don't know what development and testing methodology Tesla is using for the other processors running mission critical software. I can't recall anyone reporting major problems like the car won't accelerate/brake or won't turn. So whatever Tesla is doing to develop and test that functionality, it seems to be working.

AutoPilot is different. With AP2, it is supposed to run on the separate AP processor. Since Tesla rarely issues release notes explaining what changes they are making with the AP software, it's almost impossible for customers to determine what is or isn't working. Tesla labels the entire AP system as "beta". Because of the lack of information, customers are left to figuring out what Tesla might or might not have changed in each release - which results in forum posting or online articles declaring that AP is better or worse in new releases - and we're all just guessing, because only Tesla knows what they changed.

The console and dashboard software is the most visible - and Tesla is trying to balance quick time-to-market vs. more longer & better tested releases. Clearly Tesla could do better, since obvious bugs periodically get missed by their internal testing and their small hand-picked group of beta testers.

There are a number of steps Tesla could take to improve the quality of the releases. Most of them risk slowing down the release process and doing more thorough testing, though it's also possible to implement testing improvements without slowing down the releases (such as increased use of automated testing, opening the "formal" beta program to more customers, ...).

While Tesla's testing and release strategy could use some improvement, at least so far, when bugs are released, they appear to be in the non-mission critical portions of the software, and while they are annoying they haven't impacted the ability to drive the car safely.

 

[arubinst]

I have been a happy S75D owner for more than half a year now. While I'm still getting anxious every two weeks hoping I can be the early adopter of a new software update, I suspect I'm definitely not the only who finds it intriguing to know a bit about how Tesla test/validate its software. Think about these points:

1. Tesla release a new (minor) version of software usually every two weeks - so the internal QA/Testing team has at most two weeks to test the current version before they have to start testing the next release. (While for traditional auto makers who usually release a firmware once a year, they have a full year to do tests!)

2. Although the changes are usually small between adjacent releases, the risks of regression are always there for complex systems like it in Tesla - think about the regression bugs recently: charging cable connection issue from 2018.14, air suspension issue from a release in 2017, etc. You think Tesla has figured out a way to do full regression test every two weeks while other auto makes need a full year to do the same?

3. The software are mission critical. They control functionality such as AP, auto-braking, response to acceleration pedal (e.g. chill vs regular mode, released through a software update), etc. Even the smallest bug can cause huge damage. One of the advantages Tesla has is the gradual releases to users in batch. But even for releasing to a small batch of early adopter, the potential risk needs either bravery or some amazing testing mechanism to overcome.

Anyways, being a software engineer myself, I have been more than impressed by how Tesla has been holding a high bar on the quality and speed of their software releases. Those small bugs aside, I haven't heard of any major incidents that were caused by bugs introduced by software updates (knock on wood).

What makes you think that whenever a new feature is introduced, it was developed and tested in the last two weeks prior to release?

The fact that new firmware versions are released two weeks apart has nothing to do with the time it takes to develop and test a particular feature or the whole firmware itself.

Being a software engineer, you must be familiar with new-feature-branches. It may take several weeks or months of development and testing (using a development beta branch of the firmware on a reduced subset of cars) before a feature finally makes it into the master branch and it is ready for global distribution.

 

[NeilOnCa]

Tesla cars have multiple processors. We are only seeing the software that is driving the dashboard and console display - which appear to have separate processors, since they can be rebooted separately.

Mission critical operations are running on other processors. That's evidenced by rebooting both the dashboard and console processors - while the car is moving - and everything works, except you don't have a visible display and you don't hear turn signals.

Great point! I would imagine there are multiple subsystems in the car:

- Core driving control subsystem: directly controlling the very basic movement of the vehicle, such as increasing/decreasing the current into the motor, applying forces to the brake pads, etc. This subsystem provides APIs for other components (e.g. brake pedal, acceleration pedal, AP subsystem) of the car to use. This software/firmware for this subsystem would be very stable and maybe even never change after the model was released in the first time. (And I would expect a redundant fall-back mechanism for brake pedal, maybe a mechanical linkage)

- AP/Driving assist subsystem: running on its own processor/hardware stack (e.g. Nvidia hardware). This subsystem apparently changes often between software releases and has very limited testing time available to it. The subsystem sends instructions to the core driving control subsystem to move the vehicle autonomously. In terms of the risks of potential bugs due to short testing cycle, at least the instructions from AP can be overridden by instructions from brake/acceleration pedals which apparently should have higher priority.

- Other subsystems: MCU subsystem (controlling functionalities such as climate control, navigation, media, etc.); IC subsystem; ... These components apparently can be updated as frequent as they want without severe bugs as a risk.