Tracking issues in Tesla vehicle software

[Paul Leopardi]

I have created a prototype of an issue tracker for Tesla vehicle software, and am starting to populate it with issues and instances. Ideally I would like owners clubs to take over the running of an issue tracker like this. See the README for my initial thinking on why such a tracker is needed and who I think should administer it. I welcome feedback, suggestions, and alternatives. I am especially interested to hear from anyone with experience in either running Tesla owners club web sites or in building and hosting similar issue tracking systems.

GitHub - penguian/tesla-issues-discussion: Third party reporting and tracking of issues with Tesla software

Third party reporting and tracking of issues with Tesla software - penguian/tesla-issues-discussion

github.com

 

[jjrandorin]

I dont think this is any different than when you asked about this in Jan of 2021:

This topic has been mentioned before, but I still find it terribly opaque. How does Tesla track bugs? How do they perform software merges? How do they target releases for specific bug fixes? It all serms very mysterious as opposed to my experience with open source software such as gcc.

For example, there is currently a bug in 2020.48.30 where Cabin Overheat Protection does not work as per its documentation. How do I find out how Tesla is tracking this bug, and which release is targeted for the fix?

Is this level of transparency too much to ask for?

Thats a lot of work gathering information that wont go anywhere.

 

[Paul Leopardi]

The difference is that this time I actually created a prototype of an issue tracker to illustrate the types of issues that I want to track.

Speed sign and traffic light misrecognition is a big deal in a country like Australia that actually enforces speed limits and has red light cameras. As a driver, I need to pay very close attention to what the car is doing at any time that the actual speed limit changes or the car thinks that it changes. On a typical drive, my car get speed limits wrong half a dozen times and get traffic lights wrong at least twice, and has been doing so since this feature was made available.

If the information is gathered by a Tesla owners club then why will it not go anywhere? Surely Tesla owners clubs are in contact with Tesla?

 

[Paul Leopardi]

Also, the situation in North America is now quite different from the situation elsewhere.

In North America, a large proportion of interested owners are enrolled in the FSD Beta program, which closely tracks issues and gives very detailed release notes.

Elsewhere, we have the standard software, where bug reporting is write-only, and where the release notes just say "Minor bug fixes and improvements" if you are lucky. We also have no idea if, let alone how and when the FSD Beta will become available in our countries.

 

[jjrandorin]

If the information is gathered by a Tesla owners club then why will it not go anywhere?

They likely are, but tesla has proven time and again they are not interested in feedback in the manner of which you are talking about. They are a data company and they feel they have all the data. They also spot check social media to see how their changes are received, but owners group or no, they are not going to be interested in "here is a bunch of error feedback" because peoples recollections of events is many times not accurate.

In any case, I wish you the best in your endeavour.

 

[OxBrew]

I've thought about the value of exactly such a tracker, just for our own use outside Tesla.

It would have to be supported by correct owner reports of model, configuration, build date, software/hardware, region. There will always be holes in it, but it could be better than not having it.

Main reason for it: right now, I scan threads about each software release, looking for multiple reports of the same issues, to see if the release seems safe or better. I judge each release to see if the prevailing sense is better or worse. It's REALLY difficult to judge at all accurately, and in the end it's always a crap shoot.

It would be great to catalog bug reports and follow ups, see when issues finally got resolved, and for which models. If enough reports of a certain issue built up, Tesla might actually look at it and decide to prioritize based on visibility, but who knows.

I've not started exactly such a tracking project only because I'm not sure the end result would be any better than scanning these forums. They act like a bit of a filter, surfacing bad issues organically.

The effort to maintain the system would be enormous. Not sure the end result would be better than what happens here anyway.

I'm sort of talking myself into and out of this idea at the same time. Managing it would be a full time hobby.

 

[Paul Leopardi]

Here is the current contents of GitHub - penguian/tesla-issues-discussion: Third party reporting and tracking of issues with Tesla software

Third party reporting and tracking of issues with Tesla software.

This repository is based on teslacommunity / tesla-bugs by @rickbassham. It is intended to provide a proof of concept and a focus for discussion on ways to improve the reporting and tracking of issues with Tesla software, in particular the software that runs Tesla vehicles.

Note​

The current repository cannot be used to provide a public reporting service for such issues, for reasons discussed in detail below and elsewhere. The main reason is that a GitHub personal repository does not provide the access controls that would be needed for public reporting of issues.

Why report and track issues in this way?​

What Tesla currently does​

Tesla provides an in-car "bug report" voice command. This sends a driver's comments to Tesla, along with metadata. The advantage of Tesla's method of bug reporing is that it automatically includes relevant metadata. The disadvantage is that it is essentailly write-only. The driver does not receive a copy of the bug report and has no way to track any actions taken to address it, unless Tesla contacts the owner of of the car directly.

Tesla is well known for providing scant details about bug fixes in release notes, with the exception of Full Self Driving Beta software releases. Often the release notes will say nothing other than, "This release contains minor bug fixes and improvements." or will list feature changes and then say "Bug fixes," or will not mention bug fixes at all.

Even when a software fix is part of a NHTSA recall, it is not always clear which Tesla software release fixed which defect.

Previous attempts and why they failed​

There have been many instances of people suggesting a public bug tracking system for Tesla vehicle software, either operated by Tesla or by a third party, such as a public forum, or even GitHub. As far as I know, none of these has succeeded.

Tesla has not produced such a system itself possibly because:

1. It may not be in Tesla's best interests to have a publicly available list of known issues. Witness how news articles refer to Tesla recalls that are addressed by over the air updates.
2. Tesla already has such a system for its own internal use, and is using an enhanced bug reporting system for its Full Self Driving Beta program, complete with comprehensive release notes.

Third party schemes such as forums or GitHub may have failed for various reasons:

1. Tesla owners may be unwilling to contribute to a publicly visible bug tracking system because it may put Tesla in a bad light.
2. Forums are too unstructured, with issues mixed in with other discussion, and it is difficult to track the resolution of issues via such a discussion forum.
3. The use of public systems like GitHub is too difficult to administer with respect to access controls.
4. There may not have been enough interest in such schemes.

Types of issues to report and track​

A Tesla issue reporting and tracking system should concentrate on issues with Tesla's in-vehicle software, especially the non-FSD Beta software. Some specific software features lend themselves to issue reporting and tracking, such as:

1. Speed Assist via either speed sign detection, or maps, or both.
2. Traffic light detection.
3. Traffic egulation sign detection (e.g. stop and give way signs).
4. Auto high beam.
5. Auto wipers.

Issue reporting​

Reported issues should be specific enough to be addressible, but generic enough to apply to many instances. For example:

• "Auto high beam does not work properly" is too vague to be addressible.
• "Auto high beam turns on high beams on Northbourne Avenue in Canberra" is too specific.
• "Auto high beam turns on high beams on roads where there is street lighting" is about right.

Instance reporting​

Since a reported issue can apply to many instances, it is also useful to report these instances. Instance reporting should at least include: Software version, date and time of occurence, location of occurence, expected behaviour, observed behaviour, and can also include enough vehicle identification to distinguish vehicles, and any extra notes. Such instance reporting can highlight that:

1. Instances of an issue can persist across many software versions.
2. Instances of an issue can be confined to a specific geographical area or jurisdiction, or can be global in nature.

Capturing metadata​

If you are reporting instances, one way to remind yourself of the time an location of occurrence is to save a TeslaCam clip. As stated above, Tesla's own bug reporting also automatically captures metadata, but unfortunately such metadata is not (yet?) captured with TeslaCam clips. A possible exception to this is the FSD Beta program. Automatic capture of metadata with TeslaCam clips would greatly help instance reporting, so it should be suggested to Tesla as a feaure request.

Issue tracking​

Since Tesla, outside of the FSD Beta program, is vague about which software releases affect which issues, the onus would be on the people who reported instances of the issue to test to see if the issue persists or is solved by each software release. Confirming that an issue has been addressed by a software release would lead to the issue being closed in the tracking system. The remaining open issues would be those which have not yet been confirmed to have been addressed.

A public or a private issue tracking system?​

A completely public issue tracker that would allow any person to self-register and create issues would probably be too open to abuse. It might not be wise to have a completely publicly visible issue tracker, but this is debatable.

Who should run this type of system?​

Ideally, Tesla Owners Clubs should run this type of system on behalf of members. This would be a good compromise between a completely public system and a private system. All members would have access to report issues and instances, and make comments, and possibly a smaller number of members would have the ability to close issues and perform other admin tasks.

Candidate hosts for the issue tracker​

There are a number of candidates for an issue tracker that provides suitable access controls, including Bugzilla, GitHub Enterprise, Gitlab, Jira and Trac. The choice of which one to use would consider factor such as:

1. Does the issue tracker support the required features?
2. Does the issue tracker make it easy to control access, including integration with existing user athentication systems?
3. Does the issue tracker have an open source licence? (e.g. Trac is
4. Who would provide user support and which issue trackers are they most familiar with?
5. Cost of hosting, etc.

How would access control work?​

Ideally, if a Tesla Owners Club ran an issue tracker, it would use the same user authentication system for Club members as the Club's main web page. All members would have access to report issues and instances, and make comments, and possibly a smaller number of members would have the ability to close issues and perform other admin tasks. If Tesla Owners Clubs had some method of identifying members of sister clubs, they might be able to make their issue tracking system visible to members of all such sister clubs.