Tesla's Autopilot Software Process, and the term "Beta"

[trils0n]

Just wanted to post a quick primer about the process of how Tesla makes Autopilot software updates, and the term beta as it applies to Autopilot, as Tesla hasn't done a good job with education on this subject, and has let the media run wild with the concept.

Step 1:
Internal Testing (simulation and engineering prototypes), internal fleet testing (employee testing, mostly engineers and executives I believe).

Step 2:
Early Access Program Beta Testing: This is traditional user beta testing, like all software companies do. These people volunteer / are selected by Tesla to participate in the Early Access Beta Program and test software before it makes it to wide release.

This is where lots of the confusion about the Beta label comes from -- Tesla isn't releasing autopilot software that is "in beta" in the traditional sense. It has gone through beta testing (the EAP) but for some reason Tesla decided to keep the Beta label after this step.

After passing these initial stages, the software can be approved for wide release.

Step 3:
Wide release: Two versions of autopilot software are installed on every vehicle, an active version that controls the car, and a dormant/background version that runs concurrently, but doesn't have control over the vehicle.

The background version can then be compared to the actions of the active version, and the actions of the driver. If the action the background version would have taken differs from the active version, or from driver input, the event gets flagged, and sent for review so engineers can review the data and see how to make it better/more accurate. This allows Tesla to test and gather millions of miles of real world usage data for their new autopilot software before pushing it as the active software. New Autopilot software must best the old software in background mode before it graduates to active software.

At any point the software can be sent back to the start if problems are found, or if it doesn't work effectively.

Elon has tweeted that the "Beta" label remains on wide releases until at least 1 billion miles of real world driving data has been collected. 1 billion miles the the minimum amount of verification Tesla considers appropriate to remove the "Beta" label from their Autopilot features, but it doesn't automatically remove the label once the 1 billion mile marker is hit.

"Use of word "beta" is explicitly so that drivers don't get comfortable. It is not beta software in the standard sense."

Perhaps they should come up with another way to state this, so they don't take flak for using a term that has a well understood meaning in a different way.

 

[bob_p]

For most software projects, software leaves beta testing when the software is "feature complete" and it has been tested enough to reduce the risk of major bugs being introduced in the "commercial" release.

It really feels like Tesla's "beta" testing is closer to what many software groups would consider as early stage "alpha" testing - and that the officially released software is still in "beta" mode - since we often see major bugs and lacking functionality.

Even though we continue to get periodic maintenance releases for Version 7 - the software needs periodic reboots, the media player gets confused after the car goes to sleep, and we're missing features typically present in the apps.

Tesla is correct in labelling some of their software as "beta" - like the Trip Planner or Android smartphone app. They probably should extend the "beta" label to other apps - such as the media player or the entire navigation system. And for the browser - "prototype" is probably a better label.

Tesla can do much better with their software - and while it was understandable that when the Model S was first introduced they had resource constraints limiting the functionality and quality of the early software releases - that's probably not going to be acceptable as Tesla ramps up production with the Model 3.

 

[trils0n]

I'd say you are correct for the infotainment software, but this thread wasn't talking about that software. The infotainment stuff needs serious work and has so many bugs it makes me wonder if they test at all.

However, the vehicle control systems and software (like Autopilot), are rock solid and have been since the car's release. The difference in quality between the mission critical vehicle control software and the infotainment is night and day.

Autopilot's components are feature complete: auto emergency braking brakes for rear end collisions as designed. Forward collision warning goes off when it detects a potential rear end collision. Auto lane change changes lanes, etc. Autopilot doesn't crash or freeze or have egregious (or for that matter, any known bugs that I'm aware of) like the infotainment does. It is rock solid like the rest of the vehicle control systems.

 

[SomeJoe7777]

Also, the media and general public are unaware that "AutoPilot" refers to an entire suite of hardware and software features:

AutoPilot Safety Features (included on all Model S and X vehicles manufactured after 10/2014):

• Blind spot warnings
• Side collision avoidance
• Lane departure warning
• Forward collision warning
• Automatic emergency braking

AutoPilot Convenience Features (optional package):

• Traffic-aware cruise control
• Autosteer with Auto lane change
• Auto parallel park
• Auto perpendicular park
• Auto high/low beam headlights
• Summon

Of all of these features, the only two that are labeled "beta" are Autosteer and Summon. The entire AutoPilot suite is not "beta". And as Elon said, the "beta" label is to explicitly inform drivers to remain vigilant when using those features. It is not the same meaning as the software development cycle definition of "beta".