How Do You Think Telsa's Software Development Team is Organized?

[AmpedRealtor]

So why not do anything and everything reasonable to inform your customers of the situation, so that they don't wind up in danger or inconvenienced? A simple e-mail, sent on Monday morning, would have gone a long way towards accomplishing this. If Tesla wanted to go even further, they could have started some sort of phone campaign.

I realize that Tesla had good reasons not to do these things, but I don't believe any of them had their customers' best interests at their core. Sending out a mass e-mail alerting their customers to this problem would be bad PR, and could be bad for business. Customers that would never experience the issue would be made aware of it unnecessarily. Even so, I feel Tesla should have taken this approach. The PR could have easily been spun into a positive.

Several P85D owners in the thread about this problem indicate that they were contacted by Tesla regarding this issue.

- - - Updated - - -

Really disappointed in those of you who know and understand software development.

Answer these question ...

- How many times have you had a perfect release?
- How many times have you developed something as complex as a car?
- How many times have you continually added new features to the product after it was already in the field?
- and ... How many times did you have to read about your customers bashing you publicly after you've been working crazy hours?

You ALL know better. Every one of you know there are a myriad of reasons that could lead to the problems you've experienced. But instead, you point fingers at one group. Awesome back seat driving. Just. Awesome. You don't know all the facts, but are ready to pass damning judgment. Wow.

I don't know that we should be excusing Tesla just because what they are trying to do is difficult. If they do not have the correct processes in place to ensure that mission-critical drivetrain software is secure and bug-free, they should not embark upon a journey that requires those processes. That's pretty simple in my mind.

It's the severity of this bug that is causing the reaction, not the expectation of bug-free software. It's one thing when the navigation system freezes or the route line disappears. It's another when the car totally loses power while driving. At its worst, one is an annoyance while the other could be life threatening. There is no excuse for an owner to be put in the latter situation due to a software issue. None whatsoever.

What kind of reputation is Tesla building for itself, and is that reputation earning trust with customers or alienating them? When Autopilot finally comes around, will we be able to trust Tesla to get it right the first time and without throwing someone into a lane of oncoming traffic? Or will that bug only be exposed when someone is killed? Tesla is playing in the big leagues here and needs to step up.

We're not talking about a phone that you can chuck out the window if it doesn't work properly. We are talking about over 4,000 lbs of metal that is charged with carrying our most valuable cargo - our families. Personally, that's where I draw the line. And we're not just talking about one incident... in the last few weeks, Tesla has released software with obvious bugs in the following areas:

• Braking bug that allowed car to be put into gear and driven before boosted/power braking was available.
• Steering bug that caused steering wheel to lock up at full turn while in Sport mode.
• Drivetrain bug that caused reduced power or loss of power when in Range Mode.

Two of the above three bugs affected more than just a single model, I believe. This is a concerning pattern, which, if not changed, may destroy Tesla's credibility in the areas of software and technology. For a technology company, that's a death sentence.

 

[tomas]

Early on, I was proactively contacted by Tesla (by phone) to schedule non-urgent work due to anomaly they saw in logs. I believe this happens some.

Mass notification of owners regarding a fault they may have (or may not have) could be construed as a recall. Recalls have certain PR implications, as well as certain legal and regulatory implications. Tesla is pioneering in the area of frequent, feature-laden, OTA software updates. If every change they make were to be construed as a recall, this whole thing doesn't work. Part of the balancing act between customer service, legal, regulatory, PR... all this with an ICE auto industry gunning for you.

 

[Citizen-T]

I wish I could participate more in this conversation, but given my relationship with the company, I can't. I'll just say that what they are doing in QA is extremely impressive, innovative, and constantly improving---just like everything else they do.

Also (and this isn't based on any inside information) let's just keep in mind that a Tesla likely have tons more code in it than any other car on the road and their QA department probably has a tenth the budget and 1/100th the time to carry out their testing. I mean, when you compare their quality to something like Ford's SYNC system (which I have) Tesla comes out way ahead. Ford's infotainment was so buggy (not to mention slow) it was practically unusable for the first year or so that I had it. Oh, and no over the air updates to save Ford. About a year after I got my car I finally got an update which I had to download to a USB stick, then sit in the driveway with the car running for 45 mins while it installed. After that, SYNC was just barely acceptable.

I think Tesla has done a great job of balancing time, cost, and quality. Not that they can't do better, but I think they know that, and they are driving hard towards improving.

 

[richkae]

Software quality is a choice.

There are axes of complexity, time and expense to manage. These are in addition to process, methodology and acceptable quality level.
I would argue that if Tesla had tried to achieve NASA quality levels, the car would both cost a lot more and still be in prototype form, and none of us would be having this discussion.

Tesla can choose a higher quality level by changing some or all of the other axes, and it is quite often possible by improving methodology and process without penalties on complexity, time, or expense. It is frequently a false economy to shortcut by shipping bugs, only to have to fix them once they have reached customers - the impact of that can be hard to measure.

Saying it's "just a software bug, they shouldn't have those at all" is misguided.
One important thing to remember is that there are mechanical failures, design flaws, manufacturing defects that require correction, and there will be software failures that also require correction. It wouldn't make sense to overspend to have one system flawless and the other unreliable.

One of the features of the Model S that people value is the frequent updates that provide new functionality.
Ultimately Tesla should be allocating the resources to produce the product that the marketplace wants while balancing quality, features and cost. These are all choices, the marketplace will judge.

I am satisfied with my purchase and will continue to recommend it. If you are dissatisfied with a flaw, software or otherwise, you should give them feedback to attempt to change their choices.

 

[bonnie]

I don't know that we should be excusing Tesla just because what they are trying to do is difficult.

I don't believe I ever said anyone should be excused.

What I did say was it was a harsh judgment by several people. Even if you are not a software developer (or ever involved in getting software to the market), I doubt there is a single person here who hasn't experienced being judged by others who don't know all the facts. It's pretty easy to sit on the sidelines and say 'they should do THIS', when you're not in the middle of it.

And it turns out, some of those judgments are already being proven incorrect - as you've shown by pointing out that some people were contacted.

I appreciated Tomas' even-toned responses.

 

[AmpedRealtor]

What I did say was it was a harsh judgment by several people. Even if you are not a software developer (or ever involved in getting software to the market), I doubt there is a single person here who hasn't experienced being judged by others who don't know all the facts. It's pretty easy to sit on the sidelines and say 'they should do THIS', when you're not in the middle of it.

I can't disagree with this. It is very easy for us to pass judgment. I can't imagine the type of pressure that software team must be under, not just to delight owners but also to satisfy management and deliver on schedule. I'm not expecting perfection, just don't want the car's basic functions to be affected. But when everything, including major aspects of the drivetrain, are all reliant upon software... stuff is bound to happen. But it's still kind of scary to me.

And it turns out, some of those judgments are already being proven incorrect - as you've shown by pointing out that some people were contacted.

It seems that some were but many were not.

 

[Lloyd]

To be fair, I think the software team has a very hard job to release a bug free design. Throughout the 3 years of development of the Model S there have been equipment upgrades, different configurations etc. It must be a really tough job to test every configuration and combination that has been released.

 

[dirkhh]

Tesla's software and QA teams are doing a great job so far as I can tell from the outside. I am sure they are using all the best tools and process they can, constrained mostly by the time it takes engineers to learn new things. The job they are doing is much harder than the space shuttle software, mostly due to the heterogeneous hardware they run on (remember Elon's claim that there are a few changes to production cars every week), but also to the unpredictable environment they are dealing with (the space shuttle doesn't have to deal with children running into its path, nor the sudden application of liquids to its sensors (rain, sleet, dog pee, etc.), nor a sudden eruption of liability lawyers).

And here we go again. Amazing. You have no data to back up what you're saying, yet the relentless cheer-leading continues. You don't know what tools and processes they use, yet you are "sure they are using the best tools and processes they can"...
No, this is not "much harder than the space shuttle software". Neither is it harder than many other of the existing machine control applications that are in the market. Nor is it harder than the auto-land feature that basically every single modern airplane has. Etc. Etc.

What boggles my mind is that THEY don't think that they are doing a "great job", yet you keep posting that they do.

One point of major agreement here is that Tesla can do better. I'm sure they're working on it.

Yay, we did find something we agree on.

- - - Updated - - -

Several P85D owners in the thread about this problem indicate that they were contacted by Tesla regarding this issue.

I looked and I didn't find anyone who was proactively contacted by Tesla... can you point me to a post?

I don't know that we should be excusing Tesla just because what they are trying to do is difficult. If they do not have the correct processes in place to ensure that mission-critical drivetrain software is secure and bug-free, they should not embark upon a journey that requires those processes. That's pretty simple in my mind.

Amen to that. And to the rest of your post that I deleted for brevity.

- - - Updated - - -

What I did say was it was a harsh judgment by several people. Even if you are not a software developer (or ever involved in getting software to the market), I doubt there is a single person here who hasn't experienced being judged by others who don't know all the facts. It's pretty easy to sit on the sidelines and say 'they should do THIS', when you're not in the middle of it.

I appreciate the point of view, but I beg to differ. The nav stopping to navigate, the center screen suddenly rebooting, Slacker stopping to play... I'll give them a pass for that.
It's when it comes to safety that I am drawing the line. And oh btw, that's a regulated area and that's where the regulators are drawing the line. IVI systems (in vehicle infotainment - and that includes navigation) is considered ASIL class A or maybe B. The regulations allow you to get something wrong. Break systems and drive train are considered ASIL class D - check here for some light reading on this topic: Automotive Safety Integrity Level - Wikipedia, the free encyclopedia

And no, random "doesn't work because, hey, software has bugs" is not allowable on ASIL class D issues.

And it turns out, some of those judgments are already being proven incorrect - as you've shown by pointing out that some people were contacted.

I didn't read Tomas' post quite that way, but it's possible I misunderstood. In that case, good for Tesla. I'm glad I'm wrong.

What gets lost here is that I am mostly upset because I believe in Tesla and in their model. I want to be able to say "look, there's a company that gets it right". But in this instance I have to say, sorry, no, I don't think they are getting it right.

- - - Updated - - -

To be fair, I think the software team has a very hard job to release a bug free design. Throughout the 3 years of development of the Model S there have been equipment upgrades, different configurations etc. It must be a really tough job to test every configuration and combination that has been released.

Absolutely. This is hard. Very very hard. But when it comes to safety, it's not optional.

 

[Bet TSLA]

And here we go again. Amazing. You have no data to back up what you're saying, yet the relentless cheer-leading continues.

Amazing! You too have no data to back up anything you say and yet the relentless bashing continues. You aren't inside Tesla. You apparently have little or no understanding of the pressures of consumer software development. And yet you feel you are in a position to dismiss other people's opinions and even ascribe scurrilous motives to them (because somehow you are also confident you can see inside their heads). I think you are oblivious to your own overreach here.

No, this is not "much harder than the space shuttle software".

Why do you think that? I gave specific reasons: a fleet of heterogeneous vehicles, an environment with weather and uncontrolled actors. And you provided? Nothing, just arrogant dismissal. Has it crossed your mind that you might just be wrong? Even a little?

What boggles my mind is that THEY don't think that they are doing a "great job", yet you keep posting that they do.

I believe they are doing a great job considering the constraints (you know, the real world they live in), and I don't think that your post of a quote from a claimed Tesla engineer is necessarily at odds with that. We agree that they are not doing a good job given a perfect world. But I don't know anybody who gets to work in a perfect world. Perhaps you do.

 

[dirkhh]

Amazing! You too have no data to back up anything you say and yet the relentless bashing continues. You aren't inside Tesla. You apparently have little or no understanding of the pressures of consumer software development. And yet you feel you are in a position to dismiss other people's opinions and even ascribe scurrilous motives to them (because somehow you are also confident you can see inside their heads). I think you are oblivious to your own overreach here.

Actually, most of the software I have been involved in has been in consumer and enterprise - I just happen to work with people who have spent their lives in safety critical embedded systems.
And that really is the disconnect, I think. You appear to consider the firmware of a car consumer software (because consumers buy cars). And apply the rules for consumer software to this. And I agree with you. If this was consumer software that yes, Tesla is doing an amazing job.
The regulators disagree with you on the question whether this is consumer software of safety critical firmware.

So yes, from your point of view I am overreaching. And I apologize for assuming motives that I have no knowledge of .You are right to call me out for that.

Why do you think that? I gave specific reasons: a fleet of heterogeneous vehicles, an environment with weather and uncontrolled actors. And you provided? Nothing, just arrogant dismissal. Has it crossed your mind that you might just be wrong? Even a little?

That occurs to me all the time. And I often am. I have actually actually given examples of software that I think is much harder. But we are digressing. And it's not an important point. So I'll concede that this is really hard to get right. Torque sleep is entirely non-trivial and getting this wrong shouldn't happen (one might argue 'must not happen'), but yes, that's really hard. Making sure that the main break system works before you allow disengaging the parking break? Err, I'd argue that that is slightly easier and an oversight that really really must not hapen.

I believe they are doing a great job considering the constraints (you know, the real world they live in), and I don't think that your post of a quote from a claimed Tesla engineer is necessarily at odds with that. We agree that they are not doing a good job given a perfect world. But I don't know anybody who gets to work in a perfect world. Perhaps you do.

You keep going back to the constraints that the engineers are under. I keep going back to the constraints that regulators have put on the auto industry for safety relevant systems. I've quoted the standards, I've quoted the rules. But hey, this isn't getting us anywhere. I agree with you, if this was consumer software for an iPhone, and given the constraints they are under, and given how hard this is, then they would be doing a "great job".

 

[Andyw2100]

Several P85D owners in the thread about this problem indicate that they were contacted by Tesla regarding this issue.

I don't believe we have read on this forum of Tesla contacting any P85D owners that had not contacted Tesla first. Tesla is responding to people reporting problems, or possibly responding to people inquiring about the potential problem. No one has reported being contacted proactively by Tesla. If they have, I missed it. Can you provide a link to some of those posts, please, because I've been pretty on top of this, and I honestly don't think I've missed that kind of post.

(Edit: I see dirkhh said something along these lines before I did. I was so anxious to respond to this, because I was so sure it was wrong, that I missed that.)

However, I didn't miss this post, from earlier today in another thread, and there have been others like it:

This post is spot on. I've been off these boards for a couple weeks (havent seen these messages), and while driving to work this morning I got the same "reduced power needs service" message as everybody. It's somewhat ridiculous that my immediate response is "I bet this is an issue with .167 that is being talked about on TMC but not communicated from Tesla". I pull over, log onto TMC and read the threads, and lo and behold I find this thread. What a joke. Thank you all for posting your experiences though, very helpful.

I'm baffled that Tesla is just letting people figure this out on their own on the side of the road. Seriously, just shoot me a 3 second email "turn range mode off". Done. I can wait for the update.

And it turns out, some of those judgments are already being proven incorrect - as you've shown by pointing out that some people were contacted.

Please see above, Bonnie.

Have you also seen the posts that AmpedRealtor is referring to, or were you just taking his word for it? (Because I'm pretty sure there aren't any.)

 

[ArtInCT]

We've diverged a bit from the original intent of the topic (sorry Art), but I just want to say that after the drivetrain issue introduced in the last update I'm inclined to wait it out a few days from now on before applying future updates to let other people be the guinea pigs.

I also wonder if I should change my screen name...

Oh no, do not change your TMC name for sure....

Haaa, I was reading the responses to my original post and, well, you guys and gals can SURE slide off into tangents pretty well, grant you that....

There is a big bru-ha-ha going on in the 6.1 sticky here in User Interface.... do not miss it.... how do you insert the just kidding nod & wink face thing? .... ok here it is... ka-Pow :wink:

Well it rather dawns on me that I am the anti-thesis of the EarlyAdopter.... I have about 14 months for my P85D or whatever it will be called to arrive in an early April 16 delivery.

By then, all this will be behind you all.... sigh... nice deep breaths now.... sigh.... that feels better right?

Put into perspective... with any change comes some risk.

Your suggestion to postpone firmware updates, and see what happens with others, may be prudent depending upon your own risk tolerance factors.
Then you would be ANALMOSTEARLYADOPTER.

This postpone tactic, was common-place for us in the broadcast industry with Operating System Upgrades. We would freeze the OS on a "good" build so as not to introduce any side-effects to purpose built machine constructs.

OOPS Should be read as... AN ALMOST EARLY ADOPTER... not ANAL MOST EARLY ADOPTER....

Just to keep the real SENSITIVES out there also calmed down...

Ok, if you are not smiling at least...right now,... then you perhaps have had too much coffee and are slightly.... on edge...

 

[ArtInCT]

On question for you all.... How Many Unapplied Firmware Updates can be stored in your Tesla S at the same time?
Will having an update downloaded but not applied stop further updates from arriving or being downloaded?

 

[dirkhh]

On question for you all.... How Many Unapplied Firmware Updates can be stored in your Tesla S at the same time?
Will having an update downloaded but not applied stop further updates from arriving or being downloaded?

We had people stay on old updates for a really really long time. First when the sleep feature was disabled, then when the auto-lowering was disabled. IIRC then the car won't download the next update until you install the one that you are rejecting. And one member had to actually have his service center do the firmware update for him because he was so far behind that the OTA update didn't work anymore

 

[bonnie]

Have you also seen the posts that AmpedRealtor is referring to, or were you just taking his word for it? (Because I'm pretty sure there aren't any.)

I was just taking the word of a forum member over something easily verified - apologies to you.

 

[tomas]

Space shuttle software:

HAL: I've just picked up a fault in the AE35 unit. It's going to go 100% failure in 72 hours.
HAL: I am putting myself to the fullest possible use, which is all I think that any conscious entity can ever hope to do.
HAL: It can only be attributable to human error.
Dave: Hello, HAL. Do you read me, HAL?
HAL: Affirmative, Dave. I read you.
Dave Bowman: Open the pod bay doors, HAL.
HAL: I'm sorry, Dave. I'm afraid I can't do that.
Dave Bowman: What's the problem?
HAL: I think you know what the problem is just as well as I do.
Dave Bowman: What are you talking about, HAL?
HAL: This mission is too important for me to allow you to jeopardize it.
Dave Bowman: I don't know what you're talking about, HAL.
HAL: I know that you and Frank were planning to disconnect me, and I'm afraid that's something I cannot allow to happen.
Dave: [feigning ignorance] Where the hell did you get that idea, HAL?
HAL: Dave, although you took very thorough precautions in the pod against my hearing you, I could see your lips move.
Dave: Alright, HAL. I'll go in through the emergency airlock.
HAL: Without your space helmet, Dave? You're going to find that rather difficult.
Dave: HAL, I won't argue with you anymore! Open the doors!
HAL: Dave, this conversation can serve no purpose anymore. Goodbye.

 

[Andyw2100]

I was just taking the word of a forum member over something easily verified - apologies to you.

No need to apologize, Bonnie.

I asked mainly because if you had seen posts of that nature I know you'd point to a couple, and I'd want to be able to acknowledge that I was at least partially wrong.

I do understand that there are probably many good reasons Tesla isn't being as proactive on this as I, and others, would like them to be. It's just that when I think of what guys like Rice390 experienced this morning, and think of all the others who have no idea the problem exists, who could, as he wrote, have been warned with a simple email message saying "don't use range mode" it frustrates me, for them.

I'm sure the people in charge at Tesla are much more frustrated than I am. And that, at least, is comforting. I know their hearts are in the right place, even if their hands are, for whatever reasons, tied.

 

[billarnett]

I wish I could participate more in this conversation, but given my relationship with the company, I can't. I'll just say that what they are doing in QA is extremely impressive, innovative, and constantly improving---just like everything else they do...I think Tesla has done a great job of balancing time, cost, and quality. Not that they can't do better, but I think they know that, and they are driving hard towards improving.

Are you saying that the three critical systems bugs that have been noted here (steering, braking, power) are all just anomalies, that they're processes really are up to snuff and these three are outliers and that we shouldn't expect anything similar to ever happen again? I hope that's what you're saying and I hope it's right. But I wish I had some facts to back it up.

 

[bob_p]

While more complex than a smartphone, tablet or PC, the Model S hardware and software are less complicated than systems produced by other companies - that may also have "mission critical" features controlling critical operations that can affect safety.

Based only on my experience as a Tesla customer and seeing the pace and quality of software releases over the last 2 years, at least from an external perspective, Tesla's software environment challenges may be a result of a combination of the following:

• Underestimating resources needed for software - this is a typical mistake most organizations make. However, while more software developers and testers can help, their effectiveness may be limited unless other improvements are not also made.
• Assuming what they are doing is new and unique, justifying ignoring what other organizations have done - again, another typical mistake, especially in new companies or organizations. While the combination of what Tesla is doing may be new and unique, overall they could benefit greatly by taking advantage of what other organizations have learned is needed to produce quality software products.
• Building the software through evolution and not design - while it's easy to do a series of increasingly more functional prototypes, there's a risk that by doing so, major architectural features may not have been implemented which can make even simple changes increasingly more difficult to do. For example, implementing a "rollback" feature for the firmware might be impossible at this point, if the software architecture wasn't designed for it - or adding a 3rd party API for applications may be risky if the software wasn't designed to isolate those apps from critical system features.
• Not designing testability into the system from the beginning - testing and quality assurance are not as effective if they are added at the end of the development process. Especially with a distributed, multi-processor, architecture - automated testing could help considerably to quickly verify software still functions properly after each 'build' of the software.
• Ineffective testing strategy - the start-stop firmware distribution process and the periodic presence of major bugs not encountered during Tesla's testing shows the testing process needs improvement. Tesla appears to have a small number of testers they've selected to help them privately test new releases - this might have worked reasonably well when Tesla only had a few cars, with a few combinations of configurations - but now with more cars on the road and an increasing number of configurations (and country/region-specific features), Tesla should be considering how they can greatly expand their testing community - to ensure better coverage of the configurations. It's probably (past) time for Tesla to consider a broader more "public" beta tester program.
• More focus on implementing major new features than in providing a complete set of capabilities for each feature - it's difficult to explain why the navigation and media playback software still lack basic features like waypoints or playlists - after almost 3 years since the first Model S was manufactured. While it may be challenging for Tesla to balance new features vs. improving existing features within their limited resources, at some point, Tesla's reputation will be impacted by continuing to deliver new functionality with only basic features, which are less capable than the other auto manufacturers (how many people are really using their small screen smart phones for navigation instead of the beautiful large touchscreen and dashboard???)

Admittedly, the above are assumptions only based on an owner's perception of what Tesla is doing with their software, after receiving software updates over the last 2 years.

And hopefully Tesla is listening to their owners and implementing improvements, which may take some time before we see the benefits of those improvements.

It is possible to implement frequent (quarterly?) software updates, with more functionality, and better software quality - let's hope they take that path rather than going the other direction with less frequent and longer development cycles (which is actually counterproductive, because testing becomes increasingly more difficult - when the number of changes in each release increases).

 

[AmpedRealtor]

I looked and I didn't find anyone who was proactively contacted by Tesla... can you point me to a post?

My apologies, I swear I thought I had read that one or more were proactively contacted by Tesla. I must have misread posts by those who contacted Tesla, not the other way around.

I'm sorry if I was incorrect, clearly the P85D owners here have not been notified by Tesla. That is really unthinkable in my mind why Tesla would stay quiet about this. I'm very sorry to have added confusion.

- - - Updated - - -

I was just taking the word of a forum member over something easily verified - apologies to you.

My bad. I was wrong.