How Do You Think Telsa's Software Development Team is Organized?

[ArtInCT]

I was just thinking back to my days as the director of software development for a few major companies.... "just how is Tesla's Software Development Team Organized?"

Thinking more about it here is one logical approach....

Tools Development Team
Monitoring & Telemetrics - Logs, Wellness, System Feedback, Service Menus
Drive Dynamics Interface
Controls - Voluntary
Controls - Involuntary
Audio - WIFI - NAV
Testing & Change Control & Release Packaging & Feedback Loop
New Feature & Product Development

The TM Model S software is essentially a real-time system(s) at the core.

Additionally, as in all SW teams such as this, there are a few guys and gals who
are subject matter experts "gurus" in just about all phases and disciplines.

What are your thoughts on how TM has its Software teams organized?

For instance, at the upper level are the teams working on the S and the X or are they separated?

 

[yobigd20]

Testing & Change Control & Release Packaging & Feedback Loop

given the number of major problems with 6.1.XXX, I don't think Tesla has a software testing or software QA team.

 

[scottm]

I would hope there'd be at least two separate teams for vehicle software, that are not allowed to interact with each other.

1. Drivability and safety. All the essential stuff to drive and operate the car safely. Steering, brakes, airbags, etc..
2. Applications. Everything else that appears on center console, and how hardware features interact with driver.

Applications would have access to a read only interface published by drive group.

The real software engineers are in the drive group, and application developers in the app group.

 

[AmpedRealtor]

given the number of major problems with 6.1.XXX, I don't think Tesla has a software testing or software QA team.

Funny, but agreed.

 

[cpa]

Probably like some of my clients who bring me their data in the proverbial shoebox and expect me to make a silk purse from a sow's ear.

 

[steve841]

Truly the biggest disappointment with tesla... The development to date is a major fail. Lots fluff, but the basics are amiss.

 

[bonnie]

Really disappointed in those of you who know and understand software development.

Answer these question ...

- How many times have you had a perfect release?
- How many times have you developed something as complex as a car?
- How many times have you continually added new features to the product after it was already in the field?
- and ... How many times did you have to read about your customers bashing you publicly after you've been working crazy hours?

You ALL know better. Every one of you know there are a myriad of reasons that could lead to the problems you've experienced. But instead, you point fingers at one group. Awesome back seat driving. Just. Awesome. You don't know all the facts, but are ready to pass damning judgment. Wow.

 

[kennybobby]

i would bet that the motor drive software is done using Matlab's motor controls package based upon the definition of terms and variables in the AWD patent. This is where the motor torque, traction control and fore-aft torque split is described.

They would not be the first silicon valley company to do this sort of thing, i know of an aerospace company out there that did the same thing for the guidance, navigation and control software for a nasa spacecraft.

 

[yobigd20]

Answer these question ...

- How many times have you had a perfect release?

For critical systems, 100%. There is no room for error. People could die.

- How many times have you developed something as complex as a car?

Own ships radar. Link 11 communications. Offboard countermeasures. Targeting systems. Etc. Several dozen million lines of code between just these subsystems alone, at least. It's all much more complex than a car.

- How many times have you continually added new features to the product after it was already in the field?

Right now, daily or weekly (different product).

- and ... How many times did you have to read about your customers bashing you publicly after you've been working crazy hours?

None. And I don't. 8 hours in and I'm out. I know what I'm doing so I don't need to put crazy hours in to get the job done. And I usually do a lot more than what's required. all the extra bells and whistles really makes a difference.

In Tesla's case, since it is a car they are screwing around with and making releases that screwed up brakes, steering, and now completely power failures, again if I was Elon I would fire everyone responsible. You can't act passively here because one wrong mistake and people will die and Tesla will fold. That's reality. They simply aren't structured right and don't have the right people on the job to recognize this. If they did we wouldn't be having this conversation right now.

 

[Andyw2100]

You don't know all the facts, but are ready to pass damning judgment. Wow.

I know nothing about software development. And as I've said elsewhere, while the fact that the most recent version of the firmware (.167) was pushed with a potentially dangerous bug does concern me, I understand that the software is incredibly complex and that this kind of thing is probably at least somewhat unavoidable. I certainly don't think anyone should be fired, (unless of course it's determined that someone acted maliciously, but I'm sure that's not the case.) I imagine this was just a mistake or a series of mistakes or possibly even just an unfortunate set of circumstances. What I'm concerned about is what happened after the issue was discovered. I do know something about customer service.

By Sunday night Tesla had multiple reports of the issue. Certainly by Monday morning Tesla was well aware of what was going on, and I expect working feverishly to correct the issue. That's great, and commendable. But this particular firmware issue has a very simple workaround: keep the car out of range mode, and the car works just fine. With the car in range mode the potential problems are complete loss of power, which could be quite dangerous, or partial loss of power, which could be scary and inconvenient. Both are serious problems. So why not do anything and everything reasonable to inform your customers of the situation, so that they don't wind up in danger or inconvenienced? A simple e-mail, sent on Monday morning, would have gone a long way towards accomplishing this. If Tesla wanted to go even further, they could have started some sort of phone campaign.

I realize that Tesla had good reasons not to do these things, but I don't believe any of them had their customers' best interests at their core. Sending out a mass e-mail alerting their customers to this problem would be bad PR, and could be bad for business. Customers that would never experience the issue would be made aware of it unnecessarily. Even so, I feel Tesla should have taken this approach. The PR could have easily been spun into a positive.

Tesla wants to be different from every other car manufacturer. Not being proactive in telling customers about the problem as soon as they found out about it was not different.

This troubles me as much or more than the firmware bug itself does.

 

[EarlyAdopter]

We've diverged a bit from the original intent of the topic (sorry Art), but I just want to say that after the drivetrain issue introduced in the last update I'm inclined to wait it out a few days from now on before applying future updates to let other people be the guinea pigs.

I also wonder if I should change my screen name...

 

[dirkhh]

I also wonder if I should change my screen name...

I kept thinking about you when commenting in the "Seat Gate" thread about the fact that early adopters always get screwed. :tongue:

 

[Bet TSLA]

given the number of major problems with 6.1.XXX, I don't think Tesla has a software testing or software QA team.

Given the quality of software releases so far, I'm very impressed with Tesla's QA team. With the degree of changes they've had to absorb in a very compressed timeframe, avoiding serious bugs is next to impossible.

 

[dirkhh]

Given the quality of software releases so far, I'm very impressed with Tesla's QA team. With the degree of changes they've had to absorb in a very compressed timeframe, avoiding serious bugs is next to impossible.

I love the relationship between your user name and your relentless cheer-leading. I'd suspect that you are a plant from Tesla PR - but they would have picked a less conspicuous user name, I guess.
Seriously - talk to the people programming the software used in commercial airplanes. It is ENTIRELY possible to avoid serious bugs. It's a legal requirement.
I understand that if your main goal is to hype the stock (again, just going off your user name) that you don't want to see any critical opinions - but try to stay connected to reality, please.

 

[Bet TSLA]

I love the relationship between your user name and your relentless cheer-leading. I'd suspect that you are a plant from Tesla PR - but they would have picked a less conspicuous user name, I guess.

My user name refers to my betting on Tesla stock, leading to my purchase of a P85 from my winnings. I don't think I'm cheerleading, merely trying to alleviate some general cluelessness about software development and release.

Seriously - talk to the people programming the software used in commercial airplanes. It is ENTIRELY possible to avoid serious bugs. It's a legal requirement.

The result is that software updates occur slowly. Very slowly. IBM does the same thing with their mission critical software, like their operating system. Hell, they offer their customers single fix capability which is truly mind blowing in its complexity and cost. But they charge customers dearly, and they do a single software cycle in the time that less fastidious companies do five.

I understand that if your main goal is to hype the stock (again, just going off your user name) that you don't want to see any critical opinions - but try to stay connected to reality, please.

Oh, come off it. You won't find me telling people that the stock will go up or go down. But I do think Tesla is a most excellent business and I will probably make a bet on it now and again, one way or the other depending on circumstances. I enjoy playing the market. Either way it won't affect what I say on the boards because nobody of any note pays the slightest attention to what's said on these boards. It's just a bunch of owners (and potential owners) talking to each other about things they care about. That is not the sort of thing that moves the stock market.

 

[neroden]

I was just thinking back to my days as the director of software development for a few major companies.... "just how is Tesla's Software Development Team Organized?"

Extremely poorly. Service has complained that they're a black hole for information: it goes in but it does not come out.

They probably don't have a proper bug or issue tracking system. IF they do, they have not given access to it to the right people.

Note that I'm not saying their programmers are bad -- they clearly have some very good programmers. But their *organization* and *process* seems to suck.

It became apparent during the legal requests for source code (which Tesla is still obligated to provide and still has not provided) that their source / version tracking system is primitive and their compilation system is also primitive. There's really not much excuse for this; you can get those things off the shelf *for free*. (The same is true of bug tracking.)

 

[yobigd20]

It became apparent during the legal requests for source code (which Tesla is still obligated to provide and still has not provided)

who made the legal request and what license was violated? (assuming GPL but how would someone find out about that?)

that their source / version tracking system is primitive and their compilation system is also primitive. There's really not much excuse for this; you can get those things off the shelf *for free*. (The same is true of bug tracking.)

yea no kidding. there are many mature products out there. anyone whose worked anywhere would know this.

At Northrop Grumman & Legg Mason we had staff on our teams whose sole purpose was Change Management and source code management and QA. all official merges with master branches were not touched by the programmers. various levels of testing not done by the programmers, merged software, integration tests, regression tests. some of these tests could take days or weeks depending on the systems. soak tests (load tests > 30 days with 0 failures and 0 memory leaks). integration tests created before the first line of code is even written. we couldn't afford to make releases that were buggy. plus once the code is deployed to something like a Navy ship, there is no changing it. it has to be flawless. EVERY single line of code reviewed down to each character. teams of 30 people or so only 6-10 were actually software engineers. the rest were divided up amount those various other roles. fully waterfall cycles, with much effort spent on design and testing - a very proven effective process. for software engineers maybe 20-30% of time was actually coding. that just shows the level of effort into making it right in each release. SEI CMMI Level 5. Capability Maturity Model - Wikipedia, the free encyclopedia

here's a good read about space shuttle software - They Write the Right Stuff | Fast Company | Business + Innovation "This software never crashes. It never needs to be re-booted. This software is bug-free. It is perfect, as perfect as human beings have achieved."

if you don't have the right team structure and you're just spending 100% of your time hacking software and making releases yourself (including doing merges with other people's code which you don't fully understand or even look at which could change the dynamics of your code) then unexpected bugs is expected behavior. fyi I am *NOT* an advocate of agile methodology, especially for this kind of software domain (automobiles) that involves critical safety systems and human life.

Yes, it is possible to write perfect code. As long as you have a good process. The downside to good software process and flawless code is that it's going to take much longer to make releases. But I'd rather have a car with perfect software than one that is buggy where I can drive away without the brake systems being active yet or park without the steering locking up or be able to make it to my destination without the randomly car powering down because I was on "range mode" (ironic).

 

[bob_p]

I have been responsible for more complex software systems, with mission critical elements that could pose major safety/health risks, and with software developed from multiple organizations distributed in multiple countries.

While it's impossible to release "bug free" software - Tesla can do better.

 

[tomas]

I'm going to suppose (without numbers) that one of the most common 3 professions among frequent TMC posters is software engineering. I'll confess to being among those. Among the other frequent posters, there is a high level of software literacy. So, there are a lot of informed "opinions" about what Tesla should do with their software development and deployment process.

I'm going to suppose also that Tesla has many very competent software people, and I'm sure they have a lot of lively debates as to their software process. Lawyers, PR people, and accountants all participate in their decision-making, which is why not everything will make sense purely from a software engineering perspective. Also why I predict there will never be an open beta or full detail of all bug fixes in every release.

We all have to keep in mind that what Tesla is doing has never been done before: frequent OTA updates offering significant new feature/function in an automobile. Software updates we've all had in every other car have been much less ambitious, much less frequent, and done only at service centers. This new thing is similar to the space shuttle, but different (more vehicles! more configs). Similar to the PC or iPhone analogy many make, but different.

Despite some people's fears expressed on this forum - Tesla's software updates have not been disabling cars. There is only one instance I can recall over 2+ years of MS deliveries where cars were at risk of going out of service due to a software release. That was very early on in 2013, when a small group of Illinois owners were unable to home charge at all after an update. I don't recall the specifics, but Tesla jumped right on it, and it was sorted out within days. Yes, a few people have recently reported power loss issues they've correlated with most recent dot release. Alarming, but mostly cleared by themselves or solved by reboot. Interesting thing is that a redundant system in the car identified the power loss, and generated the message. Similar to what owners in accidents have experienced: the diagnostic systems detect faults and warn drivers early before shutting down. So, there is a level of redundancy (as their should be) to detect and report potentially disabling problems before they disable or put driver at risk.

There's no doubt that Tesla will have to scale and improve their software process to enable order of magnitude increase in production. Just as they'll have to scale and improve vehicle and battery production. I'm pretty sure they know that, I'm pretty sure they are working on it, and I'm pretty sure they debate many of the same measures that are suggested on this forum.

I just hope - in the process - the baby does not go out with the bathwater. It would be disappointing if Tesla were to ever back down on OTA software upgraded car. Personally, I'm very impressed with what they've done so far and how they've gone about it. And, I'm more than willing to suffer a few bumps along the road for the great potential that a software upgradable car brings.

 

[Bet TSLA]

here's a good read about space shuttle software - They Write the Right Stuff | Fast Company | Business + Innovation "This software never crashes. It never needs to be re-booted. This software is bug-free. It is perfect, as perfect as human beings have achieved."

Exactly! And yet the truth is that there are bugs in that software too. Read about one case here. Tesla can't be this careful. Nowhere near. Software creation would take about ten times as long, and consumers wouldn't stand for it. If a poll were taken now, I'm quite sure that Tesla customers would prefer to take TACC as it is now rather than to wait for flawless next summer. And yes, I know a few people will say different. But I guarantee they would be demanding (along with everybody else) that Tesla deliver faster on the their promises.

Tesla's software and QA teams are doing a great job so far as I can tell from the outside. I am sure they are using all the best tools and process they can, constrained mostly by the time it takes engineers to learn new things. The job they are doing is much harder than the space shuttle software, mostly due to the heterogeneous hardware they run on (remember Elon's claim that there are a few changes to production cars every week), but also to the unpredictable environment they are dealing with (the space shuttle doesn't have to deal with children running into its path, nor the sudden application of liquids to its sensors (rain, sleet, dog pee, etc.), nor a sudden eruption of liability lawyers).

One point of major agreement here is that Tesla can do better. I'm sure they're working on it.