Welcome to Tesla Motors Club
Discuss Tesla's Model S, Model 3, Model X, Model Y, Cybertruck, Roadster and More.
Register
  • The final cut of TMC Podcast #21 is available now. We covered the recent wide release of FSD Beta and shared our experiences. We also discussed the best times to charge and Tesla's official CCS adapter. You can watch it on YouTube.

[UK] 2022.16.x

WannabeOwner

Well-Known Member
Nov 2, 2015
6,933
3,690
Suffolk, UK
I'm not sure Agile development and safety critical systems are good bedfellows

We use Agile. We have automated regression testing, very strict defensive-programming approach / conventions, peer review of all code changes, and I can't remember the last time we had a bug in production - and there is blood-on-the-carpet if we do ...

We are a small company of less than 50 people and 2 developers ...

Doesn't seem like Rocket Science to me ... but "throw it together and chuck it out the door" seems to be most company's axioms for software development these days, particularly in companies where Marketing has managed to wrestle the responsibility away from IT Department and has changed the acceptance requirement from proven-usability to pixel-perfect - most especially for all customer-facing web-based projects.
 

GeorgeSymonds

Active Member
Moderator
Mar 16, 2018
2,304
2,452
UK
We use Agile. We have automated regression testing, very strict defensive-programming approach / conventions, peer review of all code changes, and I can't remember the last time we had a bug in production - and there is blood-on-the-carpet if we do ...

We are a small company of less than 50 people and 2 developers ...

Doesn't seem like Rocket Science to me ... but "throw it together and chuck it out the door" seems to be most company's axioms for software development these days, particularly in companies where Marketing has managed to wrestle the responsibility away from IT Department and has changed the acceptance requirement from proven-usability to pixel-perfect - most especially for all customer-facing web-based projects.
Tesla have talked about simulation of driving with Autopilot as a way of trying to regression test.

I think there are three different elements (as 2022.16.x appears on hold maybe those interested can debate)

Agile development
Minimum viable product with a hardware component
AI and non deterministic models

I've split out MVP although some would say thats encompased by agile, but it has some specific issues with respect to hardware. You can't change hardware easily and that places quite a large constraint around your future development and I think there's increasing evidence thats starting to hurt. A few examples:

Tesla seem to be going to pixel peeking to try and extract the absolute maximum out of the images, thats actually a worry because it's admitting the cameras are right on the limit of what they can resolve and I think I heard other news about securing deals for better cameras. What about all those who bought cars with the promise they had all the FSD hardware needed?

Blind spot warning. Pretty much the standard way to implement this is a flashing indications in the side mirrors. If you have no light to flash, you can't flash it so we get a camera image on the screen in the opposite direction to where you need to be looking.

Sentry mode. High power drain when in use because the hardware doesn't seem to have been designed with a low power mode for sentry purposes like a regular dashcam.

Then we get to AI and decision making. It's generally accepted that once you get a rough model working, the challenge is the edge case management and balancing false positives and false negatives (assuming binary choice). Getting it wrong isn't a bug as such, they're probability models and a dirty sign with sun glare could be mistakenly read by both computers or humans. Ask 100 people to read it would result in a split of results, some will be wrong. Sticking with the human anology, give all those humans sunglasses to cut down on the sun reflections is likely to result in a different set of results, some people might now be able to read that sign, some who were fine before may now get it wrong, but lets say results are better. The next sign you look at might be at night and poorly illuminated, those sunglasses may have made it overall worse, maybe a lot worse, but the things that are worse may have nothing to do with the reason the change was made. Magnify this up to self driving, and every change they make moves that edge, some things better, some things worse. Testing on a simulation just means you're tuning the models to the smulation, not to the real world. I have no idea how they are going to prove this sufficiently to regulators when any and every update could result in an unintend consequence somewhere.

So for me, the worry is less about Agile as such although they're struggling to cover that one at times, long term it's about the hardware restrictions and how you test and demonstrate probability based AI models still work well enough for people to trust their lives with them.
 
We use Agile. We have automated regression testing, very strict defensive-programming approach / conventions, peer review of all code changes, and I can't remember the last time we had a bug in production - and there is blood-on-the-carpet if we do ...

We are a small company of less than 50 people and 2 developers ...

Doesn't seem like Rocket Science to me ... but "throw it together and chuck it out the door" seems to be most company's axioms for software development these days, particularly in companies where Marketing has managed to wrestle the responsibility away from IT Department and has changed the acceptance requirement from proven-usability to pixel-perfect - most especially for all customer-facing web-based projects.
I shudder at the thought of their regression test/automation pack. Must be huge.
 

WannabeOwner

Well-Known Member
Nov 2, 2015
6,933
3,690
Suffolk, UK
So for me, the worry is less about Agile as such although they're struggling to cover that one at times, long term it's about the hardware restrictions and how you test and demonstrate probability based AI models still work well enough for people to trust their lives with them.

Interesting points George, thanks. I think on the "We never thought to put indicator on wing mirrors" they could use a Beep of some sort - if you signal and an object is detected in blindspot - just need enough of a nudge so the driver does a second-look. But. :) There are plenty of driver aids where BMW clearly does a better job and for some reason Tesla is not bothering. I've always assumed it was Musk saying "Don't waste any effort on making the wipers work well, we need FSD" ... but as Wifee told me "Surely a middle manager has enough clout to get a couple of Summer interns in and get that sorted".

Tesla's back office system is dreadful. Dunno if that is because they have implemented a package (SalesForce or something like that) and struggle to get it configured - I don't have much time for those systems "Completely rebuilt by Middle-Man Implementation Vendor". First time we had to pull data from Great Plains (now Microsoft Dynamics) I stupidly assumed that the data would be of decent quality. Country Code contained Postcode and all sorts of address-related-rubbish in it. So there was clearly no lookup / data validation ... and a field big enough to accommodate it. I presume that was not the Great Plains Default! and that the implementer just made a rubbish job of it (company was far too small for Great Plains IMHO so probably bought-high-end and only able to pay-cheap-implementation). But my view of how back office software should be built is not conventional ... maybe I should invite Musk over and give him a pitch over Dinner :)

I shudder at the thought of their regression test/automation pack. Must be huge.

My view would be that it is essential. But, yeah, "tricky" :) and as George said the cornucopia of legacy hardware / combinations would add to the challenge.

When we built regression test (wishing we had had it from the start, but we hadn't, so we had the "Initial build" effort for all the systems we already had, by that time, in production) it took us 3 months solid. That's the whole DEV team, the related Service / Support people, the QA team and all the Users who were required to share their UAT plan with us (most had laughable plans of course, so plenty of work for them to consider corner-cases, but they were easily sold on "This is the last UAT effort you will have to make")

Regression Cost initial build cost is huge. Do it from Day One to avoid that. Add a corner case every time you trip over one (e.g. in production). Then reap the benefit ... a software change I make today goes through regression test tonight and I know if there is a new / collateral problem first thing tomorrow. So I can fix it before I build a complete edifice on top of it - wrongly assuming the foundations are solid!

I have long considered that the regression bugs that actually turn up in OTA updates are pretty low (open to other opinions on that though), but many/most? new features arrive poorly implemented, and need alteration later, or are buggy (of course that isn't regression testing fault, that's QA or Design - or intentional i.e. Release-and-Improve).
 
I’m amazed when updates like the blind spot camera viewer get to go live.

There are so many steps in the software development lifecycle (whatever methodology being used) where somebody would have the chance to have said “why are we going to put it there on the display, it’ll be more distracting to look at that than look over your shoulder?”.

It’s such a shame because I thought it was a great bit of functionally to be gifted and yet I never use it.
 
Interesting points George, thanks. I think on the "We never thought to put indicator on wing mirrors" they could use a Beep of some sort - if you signal and an object is detected in blindspot - just need enough of a nudge so the driver does a second-look.
Given how many bongs and beeps the car does, as well as pop up small messages in the centre of the display you have no chance of ever reading, I'm a little surprised there arent any voice messages. Perhaps I've watched a bit too much Star Trek TNG but some helpful voice callouts would go miles.

Just not bloody "On the roundabout, take the first exit" :D
 
Given how many bongs and beeps the car does, as well as pop up small messages in the centre of the display you have no chance of ever reading, I'm a little surprised there arent any voice messages. Perhaps I've watched a bit too much Star Trek TNG but some helpful voice callouts would go miles.

Just not bloody "On the roundabout, take the first exit" :D
"Coolant leak. Core breach imminent...."

No, sorry, it was just the shadow of a bridge.
 

WannabeOwner

Well-Known Member
Nov 2, 2015
6,933
3,690
Suffolk, UK
I thought it was a great bit of functionally to be gifted and yet I never use it.

Me too, except that on NoA when I get a nudge to chain lanes coming up to an exit the blindspot camera comes on (I have to confirm the lane-change, so I'm not actually in the process of doing that at that point), and I think that might be useful ...

... but I'm not sure I can see enough in the mini-TV view to be useful - or maybe I've never actually had anything in my blindspot and, if I did, I would notice a great hunk of metal in that spot ...

... which I suppose if never going to happen as NoA "Change lanes" isn't going to offer (I presume?!) if that lane is blocked

So I've talked myself into that being moot
 
Me too, except that on NoA when I get a nudge to chain lanes coming up to an exit the blindspot camera comes on (I have to confirm the lane-change, so I'm not actually in the process of doing that at that point), and I think that might be useful ...

... but I'm not sure I can see enough in the mini-TV view to be useful - or maybe I've never actually had anything in my blindspot and, if I did, I would notice a great hunk of metal in that spot ...

... which I suppose if never going to happen as NoA "Change lanes" isn't going to offer (I presume?!) if that lane is blocked

So I've talked myself into that being moot
It offers the change when the lane is blocked but there's a red border
 

Products we're discussing on TMC...

About Us

Formed in 2006, Tesla Motors Club (TMC) was the first independent online Tesla community. Today it remains the largest and most dynamic community of Tesla enthusiasts. Learn more.

Do you value your experience at TMC? Consider becoming a Supporting Member of Tesla Motors Club. As a thank you for your contribution, you'll get nearly no ads in the Community and Groups sections. Additional perks are available depending on the level of contribution. Please visit the Account Upgrades page for more details.


SUPPORT TMC
Top