MrBadger
Badger out
You could save some time you know and just post a link to the scrum guide if you are just going to quote it pretty much verbatim.
Just a couple of values and principles from the Agile Manifesto
-
Want to remove ads? Register an account and login to see fewer ads, and become a Supporting Member to remove almost all ads.
Software testing
- Thread starter LongRanger
- Start date
Just a couple of values and principles from the Agile Manifesto
so here's the thing, there's at least 5+ folks in one page of a thread who get the points being raised and can apply common sense to what users are seeing in their "working software" - to then identify that maybe something is amiss.
why is it so hard for a company that can mine cheese from the moon on resuable rockets to follow the same logic ?
Not meaning to sound like a rant although it probably reads like it a bit - I just don't get how it's possible to royally feck things up like Nav and Charging.
why is it so hard for a company that can mine cheese from the moon on resuable rockets to follow the same logic ?
Not meaning to sound like a rant although it probably reads like it a bit - I just don't get how it's possible to royally feck things up like Nav and Charging.
Jason71
Well-Known Member
You would be surprised. Software is actually really really really hard. Just ask VW.
It's just not possible to test everything especially when your code relates to a physical device like a car. There are just too many variables.
First you have to figure out all of the possible impacts of your code change.
Then you have to decide which of those are the high risk items
Then you have to figure out a way to test them
Then you have to design the tests so they actually have a chance of efficiently identifying bugs in the area you are looking at
Then you have to actually set up the test scenario with the appropriate hardware and software and data
Then you have to run the tests repeatably
Then you have to retrieve and interpret the results.
Pretty sure most of the testing would not occur via the testers driving around in actual cars. Most of it would involve simulators and test rigs which themselves would have bugs. Do they have a charger at the factory that is designed to replicate UK grid power delivery or the data to simulate it? if they do is it accurate? I am pretty sure 2020.12 testing never involved a tesla employed tester loading the code into an actual car in the UK and charging it half a dozen times to see if it dropped to 4 KW. imagine doing that in every tesla variant ( current and past) in every country it just would not be practical.
So while I do not mean to make excuses for them the 4KW bug I actually understand. it could have been a really really hard one to find in the factory. Which is why they have a beta program. That is the sort of bug that maybe was only ever likely to have been found in the wild.
The real question here may be why did the beta (advanced) program miss it? It's pretty ubiquitous in the UK so how was it not identified before being rolled out to the standard users like me ( I test code at work I don't want to do it in my off hour for free). Ether they did not roll it out to enough beta users in the UK. Did not leave it long enough for the bug to become apparent or did not monitor the advanced user feedback properly and realise that there was a problem.
Is it really though? Most in this profession (including myself) seem to manage to spend many hours on this forum during the working day.
What puzzles me about the 4kW bug is that charging was working perfectly well after they fixed it the last time they cocked it up. If it ain't broke don't fix it is my motto. It's not like there was a problem with charging they needed to fix so what were they doing ?
Surely the charging code is not linked with the games or navigation or audio stuff in any way.
Surely the charging code is not linked with the games or navigation or audio stuff in any way.
Speculative of course but they're in the process of adding support for 250kW charging for model S and X.
If the regression tests for the previous bug weren't up to scratch it could have been reintroduced, assuming it's an identical issue from a code perspective rather than a symptom perspective.
Jason71
Well-Known Member
I don't think this is the return of an old bug. I know there have been charging bugs in the past but I don't think this is exactly the same as any of them is it? it's certainly the first one that has affected me.
It's a good point by the OP, but difficult to speculate on the root causes. As someone who also has decades of software dev work, including mission-critical systems (medical, aerospace), I can throw in a few comments:
First, it's hard to accurately quantify the software quality when outside of Tesla looking in. There is a big software load in the car, and the vast majority of it does indeed work as advertised pretty much all the time (remember, we only notice the bits that fail, which can cause unconscious bias). However, a significant percentage of the problems reported do seem to be regressions. This can be a sign of several things:
-- Poor testing methodology. They just dont test rigorously enough, and/or much of the testing is ad-hoc, rather than planned. As others have noted, it may also be related to the SDLC model they use, which (if agile) tends to focus on throwing lots of releases out the door.
-- Poor software design. If the software stack lacks good architectural rigor, then it may be difficult/impossible to test subsystems in isolation, and/or changes in one subsystem may have unexpected changes elsewhere (blast radius).
-- Resource Constraints. No software team is ever fully staffed, and always has multiple conflicting goals. When under pressure to deliver on features (and you can be certain from Elon down that is intense), something sometimes gives .. and that something may be quality.
-- Customers as Testers. It's quite possible that they have a dual quality system, where only critical (basic driving) system are rigorously tested, and other systems are tested to some level, but are let out so that we can act as testers.
-- Corporate Culture. While not as bad as some think, Tesla are not really a "quality" focused company like, say, Toyota. This culture may also extend to the software teams. (This is often the root cause of issues like this, all the other causes originating from this one).
But what if they fixed this? Rigorous testing takes time, and for a system as complex as the car, that can be a lot of time. Would we, as a community, be content to see only 1 or 2 updates a year? And slower evolution of the car features?
But let's also not underestimate the task involved here. Remember that Tesla now make several different car models, with different generations of computer hardware and AI, and who knows how many variations in component parts (sensors, cameras, motors, etc etc). And different screen layouts (S/X and 3/Y). And multiple languages (with languages like Chinese needing very different UI). And left/right hand drive. And different build options (LR,SR,AWD,FWD) that have accumulated over the years. And restrictions to features imposed by different countries/jurisdictions. And training the AI on different road layouts, signage, symbols etc.
That all adds up to a lot of variations in the software stack to account for these differences. That's a very large surface area to test. To be honest, I'm surprised they manage to ship as many new releases as they do.
I probably sound like I'm defending Tesla .. I'm not. But I am saying its important to understand the scope of the problem, and keep a balanced perspective. I've been in industries where software must work, and you are lucky to ship more than one release every 2-3 years.
First, it's hard to accurately quantify the software quality when outside of Tesla looking in. There is a big software load in the car, and the vast majority of it does indeed work as advertised pretty much all the time (remember, we only notice the bits that fail, which can cause unconscious bias). However, a significant percentage of the problems reported do seem to be regressions. This can be a sign of several things:
-- Poor testing methodology. They just dont test rigorously enough, and/or much of the testing is ad-hoc, rather than planned. As others have noted, it may also be related to the SDLC model they use, which (if agile) tends to focus on throwing lots of releases out the door.
-- Poor software design. If the software stack lacks good architectural rigor, then it may be difficult/impossible to test subsystems in isolation, and/or changes in one subsystem may have unexpected changes elsewhere (blast radius).
-- Resource Constraints. No software team is ever fully staffed, and always has multiple conflicting goals. When under pressure to deliver on features (and you can be certain from Elon down that is intense), something sometimes gives .. and that something may be quality.
-- Customers as Testers. It's quite possible that they have a dual quality system, where only critical (basic driving) system are rigorously tested, and other systems are tested to some level, but are let out so that we can act as testers.
-- Corporate Culture. While not as bad as some think, Tesla are not really a "quality" focused company like, say, Toyota. This culture may also extend to the software teams. (This is often the root cause of issues like this, all the other causes originating from this one).
But what if they fixed this? Rigorous testing takes time, and for a system as complex as the car, that can be a lot of time. Would we, as a community, be content to see only 1 or 2 updates a year? And slower evolution of the car features?
But let's also not underestimate the task involved here. Remember that Tesla now make several different car models, with different generations of computer hardware and AI, and who knows how many variations in component parts (sensors, cameras, motors, etc etc). And different screen layouts (S/X and 3/Y). And multiple languages (with languages like Chinese needing very different UI). And left/right hand drive. And different build options (LR,SR,AWD,FWD) that have accumulated over the years. And restrictions to features imposed by different countries/jurisdictions. And training the AI on different road layouts, signage, symbols etc.
That all adds up to a lot of variations in the software stack to account for these differences. That's a very large surface area to test. To be honest, I'm surprised they manage to ship as many new releases as they do.
I probably sound like I'm defending Tesla .. I'm not. But I am saying its important to understand the scope of the problem, and keep a balanced perspective. I've been in industries where software must work, and you are lucky to ship more than one release every 2-3 years.
Drew57
Active ember
That's the most balanced and thoughtful explanation I have seen. Thank you for reminding us of the challenges Tesla face.
^^ this. We would get on in the workplace drtimhill
What I like about this forum is there’s genuinely a balance of the bad and good and in a lot of situations people see both sides - the post originally was from a viewpoint of if there has been a tendency for quality to be more variable or even reduce over time. Neither for or against Tesla - just asking the question ....
That would be a sign of those quality systems failing or those in charge of them getting complacent - as you say right the way through product vision > feature > architecture > code > test > packaging.
Would be fascinating to understand their feature boards and how much of test is brought right forward in the product development process.
What I like about this forum is there’s genuinely a balance of the bad and good and in a lot of situations people see both sides - the post originally was from a viewpoint of if there has been a tendency for quality to be more variable or even reduce over time. Neither for or against Tesla - just asking the question ....
That would be a sign of those quality systems failing or those in charge of them getting complacent - as you say right the way through product vision > feature > architecture > code > test > packaging.
Would be fascinating to understand their feature boards and how much of test is brought right forward in the product development process.
Similar threads
