FW Beta testing and bug reporting

[mnx]

At least the beta testers.

How does one become a beta tester?

 

[gg_got_a_tesla]

How does one become a beta tester?

Some early Signatures were apparently handpicked for this.

I'd love to become a beta tester too if anyone at Tesla's reading this. Ben/Dave/Nigel, please PM me if there's any other way to sign up

 

[NigelM]

Some early Signatures were apparently handpicked for this.

I'd love to become a beta tester too if anyone at Tesla's reading this. Ben/Dave/Nigel, please PM me if there's any other way to sign up

No use asking me, I'm not on the magic list!

 

[brianman]

No use asking me, I'm not on the magic list!

Ditto.

 

[dflye]

Had my car in for minor service today. I asked for the 5.0 update when I checked in. They said they needed to find out if it was GA.

When I picked up, they said it was still "beta" and that they weren't allowed to do the update. But new cars have it.

To me, that is utter BS when it comes to QA where the real testers seem to end up being owners vs Tesla's QA FW/SW group.

The folks that have had their vehicles for many months now are very aware of every existing quirk, and would pick up on any new quirks in a heartbeat.

Some new owner who is completely green to the entire legacy of prior software updates would have no clue what was a longtime existing feature, a new feature, a new bug, a regression of an old bug, etc.

Even more annoying to me being a Sig owner in the SW industry; both their UI design and testing groups could definitely use a bit more "road testing" in a lot of cases... they obviously aren't trying a "hit the target" test in a vehicle going down a bumpy road and accidentally disconnecting a call when aiming for the answer button, as the default of detecting a press that misses the target button is to drop the call! Bad UI design led to a poor implementation followed by QA apparently not done in the real world.

One of many interface failures that if a regular user complains about, the whining is lost in the noise, but if there was a beta tester or otherwise sanctioned source of bug reporting, crap like that would actually get fixed!

 

[AlMc]

To me, that is utter BS when it comes to QA where the real testers seem to end up being owners vs Tesla's QA FW/SW group.

The folks that have had their vehicles for many months now are very aware of every existing quirk, and would pick up on any new quirks in a heartbeat.

Some new owner who is completely green to the entire legacy of prior software updates would have no clue what was a longtime existing feature, a new feature, a new bug, a regression of an old bug, etc.

Even more annoying to me being a Sig owner in the SW industry; both their UI design and testing groups could definitely use a bit more "road testing" in a lot of cases... they obviously aren't trying a "hit the target" test in a vehicle going down a bumpy road and accidentally disconnecting a call when aiming for the answer button, as the default of detecting a press that misses the target button is to drop the call! Bad UI design led to a poor implementation followed by QA apparently not done in the real world.

One of many interface failures that if a regular user complains about, the whining is lost in the noise, but if there was a beta tester or otherwise sanctioned source of bug reporting, crap like that would actually get fixed!

Agreed, But I thought there were/are beta testers? I would like to believe (may I am naïve) that they would have some existing owners testing the upgrades before general release. If so, they are giving feedback and getting rid of the bugs.

I figured beta testers would be a combo of Sig owners, 60/85 KW battery, perf vs non perf people.

 

[dflye]

Agreed, But I thought there were/are beta testers? I would like to believe (may I am naïve) that they would have some existing owners testing the upgrades before general release. If so, they are giving feedback and getting rid of the bugs.

That is definitely true that multiple long-time existing owners are beta testers, but apparently they have lower standards than I when it comes to acceptable UI!

After identifying and documenting how to reproduce any new faults with a beta release, I'd be bitching non-stop about the existing UI issues that have been there since day 0!

 

[brianman]

Even more annoying to me being a Sig owner in the SW industry; both their UI design and testing groups could definitely use a bit more "road testing" in a lot of cases... they obviously aren't trying a "hit the target" test in a vehicle going down a bumpy road and accidentally disconnecting a call when aiming for the answer button, as the default of detecting a press that misses the target button is to drop the call! Bad UI design led to a poor implementation followed by QA apparently not done in the real world.

One of many interface failures that if a regular user complains about, the whining is lost in the noise, but if there was a beta tester or otherwise sanctioned source of bug reporting, crap like that would actually get fixed!

That is definitely true that multiple long-time existing owners are beta testers, but apparently they have lower standards than I when it comes to acceptable UI!

After identifying and documenting how to reproduce any new faults with a beta release, I'd be bitching non-stop about the existing UI issues that have been there since day 0!

You're making a pretty large assumption that they (beta testers) aren't doing that as well. You're making an even larger assumption that it somehow directly affects whether Tesla decides to prioritize fixing that change high enough that they hold the release for it. And finally you're then insulting the beta testers as having "lower standards".

That's a lot of leaps to make because you happen to be disappointed with one particular behavior that is non-critical to the car's behavior as a transport vehicle.

 

[dflye]

You're making a pretty large assumption that they (beta testers) aren't doing that as well. You're making an even larger assumption that it somehow directly affects whether Tesla decides to prioritize fixing that change high enough that they hold the release for it. And finally you're then insulting the beta testers as having "lower standards".

That's a lot of leaps to make because you happen to be disappointed with one particular behavior that is non-critical to the car's behavior as a transport vehicle.

If Tesla had a public site where folks could view existing bugs/feature requests, and let hoi polloi up-vote/down-vote those requests, I think you'd find I'm FAR more nit-picky than your typical beta tester. Just ask anyone I've worked with in SW in the last few decades, I don't accept cutting corners for any reason as an excuse for delivering an inferior product.

That isn't to say that the existing beta testers aren't already doing a great job, I'm just saying that there is no publicly available way for the general public to know what is known to Telsa as a bug or feature request, to provide ranking of those requests, etc. Basically, there is no visibility into the process, when they have a VAST audience of experts in the field who would gladly be providing pro bono detailed jira (or similar system) tickets documenting any quirk they've found.

If you've ever tried reporting a bug or requesting a feature through either email or phone channels, after the first few tries, you give up, as the folks on the other end apparently have absolutely no clue how to deal with technical details.

 

[brianman]

If you've ever tried reporting a bug or requesting a feature through either email or phone channels, after the first few tries, you give up, as the folks on the other end apparently have absolutely no clue how to deal with technical details.

I have (with Tesla) and have gotten acceptable responses thus far. Sparse/non-in-depth responses, but acceptable. Then again, I don't expect to get a direct line to the engineering teams.

- - - Updated - - -

If Tesla had a public site where folks could view existing bugs/feature requests, and let hoi polloi up-vote/down-vote those requests...

... there is no publicly available way for the general public to know what is known to Telsa as a bug or feature request, to provide ranking of those requests, etc. Basically, there is no visibility into the process, when they have a VAST audience of experts in the field who would gladly be providing pro bono detailed jira (or similar system) tickets documenting any quirk they've found.

Now I can agree with you.

What I took issue with was in the prior post was assumptions about the beta tester behavior based on not seeing your desired fixes implemented.

And, no I'm not one, though I'd like to be. If for nothing else than to explore REST while they adjust/expand it rather than after it's already baked in a given release.

 

[dflye]

Now I can agree with you.

What I took issue with was in the prior post was assumptions about the beta tester behavior based on not seeing your desired fixes implemented.

And, no I'm not one, though I'd like to be. If for nothing else than to explore REST while they adjust/expand it rather than after it's already baked in a given release.

Sorry, didn't mean to imply that beta testers aren't racking up huge volumes of tickets in the internal Tesla feature tracker (whether through direct access or some watered down process hopefully involving an engineer not of the junior grade), just saying that on the anal-retentive scale for UI design and implementation, I'm usually waaaaay off the far end of the chart.

Now if I could just get the ear of someone involved with the Navigon integration with regards to when they will support custom POI loading, I'd be in heaven! Attempting that through the regular Tesla channels on numerous occasions has gotten me absolutely nowhere, so for the time being I use an old TomTom when I need to do any real navigation, as the current navigation system is a toy at best.

 

[dirkhh]

If Tesla had a public site where folks could view existing bugs/feature requests, and let hoi polloi up-vote/down-vote those requests, I think you'd find I'm FAR more nit-picky than your typical beta tester. Just ask anyone I've worked with in SW in the last few decades, I don't accept cutting corners for any reason as an excuse for delivering an inferior product.

Having done software development and product delivery for a couple of decades... no, having random people vote their issues up/down almost never gives you good data to base your productization decisions on.

That isn't to say that the existing beta testers aren't already doing a great job, I'm just saying that there is no publicly available way for the general public to know what is known to Telsa as a bug or feature request, to provide ranking of those requests, etc. Basically, there is no visibility into the process, when they have a VAST audience of experts in the field who would gladly be providing pro bono detailed jira (or similar system) tickets documenting any quirk they've found.

Since you are mentioning Jira... do you think that Google is actually fixing the bugs that joe user reports in the public Jira for Android? Yeah right, let's rethink that argument as well.

If you've ever tried reporting a bug or requesting a feature through either email or phone channels, after the first few tries, you give up, as the folks on the other end apparently have absolutely no clue how to deal with technical details.

I have yet to report a bug to Tesla - but not because there isn't a public bug tracker, but because overall it was clear that they were laser focused on getting the European cars to market. So reporting bugs in 4.5 seemed like an utter waste of everyone's time.

Now once we have 5.0 rolled out, yet, I will be reporting all the little issues that I find and yes, a public forum to see what other bugs have been reported would be nice. So let's do this here! This forum has been exceptionally good at managing some of the processes that Tesla doesn't want to provide in public (for whatever reason). It should be easy enough to track the bugs for them as well (and avoid duplicate reporting of the same bugs over and over again).

I'll start a new thread for that (and will need some moderator help to turn the post into a wiki so people can edit the summary, not just add comments at the end.

 

[ElSupreme]

That is definitely true that multiple long-time existing owners are beta testers, but apparently they have lower standards than I when it comes to acceptable UI!

After identifying and documenting how to reproduce any new faults with a beta release, I'd be bitching non-stop about the existing UI issues that have been there since day 0!

Well maybe they don't wan UI feedback. Maybe they want bug feedback.

Not saying that is the right way to do it, but I could definitely see that desire.

 

[viperboy]

Terminology-related drama happens all the time for stuff like that.

• "That behavior is by design."
• "But it's a bug."
• "No, that's incorrect. It's by design."
• "But it sucks."
• "Yes, that's correct. We hope to make it better next release, but it's by design for this release."
• "But it sucks. It's a bug."
• "No, it's not."
• "Explain."
• "Imagine you're the programmer and you get this spec for a retarded feature. You implement it perfectly."
• "Right, so that's a retarded feature. It's a bug."
• "Agree, disagree. Yes it would be a retarded feature. But it's not a bug. The specification was implemented perfectly."
• "Oh. So it's a spec bug?"
• "Some would use that term, yes."

I get this daily. I prefer the term "broken as designed". This typically occurs when the requirements / design are incomplete, written in a vaccuum (without proper vetting) and/or people just dont think. I find an equal mix of the two.

Edit: a majority of these cases are when the designed function doesnt execute as planned. The un-happy path if you will...

 

[dirkhh]

I get this daily. I prefer the term "broken as designed". This typically occurs when the requirements / design are incomplete, written in a vaccuum (without proper vetting) and/or people just dont think. I find an equal mix of the two.

I love this term... and stole it for the bug tracking wiki