Code Freezes during the holidays?

[ahkahn]

Historically Tesla releases some of their best releases just shy of (and even on) Christmas. Often they spin it as a "Christmas present". I believe they did the Model X dance release a day or two before Xmas, then there was the Ho Ho Ho release, and last year it was the more extensive battery prewarming, seat warmers, and other cold weather enhancements.

 

[arghx7]

They will probably try to release something so they can claim more EAP/FSD deferred revenue by quarter's end, similar to the V10 Smart Summon push.

 

[ahkahn]

They will probably try to release something so they can claim more EAP/FSD deferred revenue by quarter's end, similar to the V10 Smart Summon push.

Probably the stop light/stop sign recognition for a "feature complete" FSD that they've been promising by the end of the year. It will be buggy, and it will suck.

 

[arghx7]

Probably the stop light/stop sign recognition for a "feature complete" FSD that they've been promising by the end of the year. It will be buggy, and it will suck.

They've already released that, they just haven't rolled out visualization yet. Under the right circumstances it will throw up a warning if you are close to going through a stoplight or a stop sign when on AP. There's no way it's going to actually do a normal stop when on AP, not yet, although maybe an emergency stop or something.

 

[boonedocks]

They will probably try to release something so they can claim more EAP/FSD deferred revenue by quarter's end, similar to the V10 Smart Summon push.

I am sure Tesla will release a "statement" that FSD is feature complete.

 

[DrDabbles]

Unless/until someone from Tesla posts what their practices are this thread is useless.

Good thing you took the time to come here and comment.

 

[hugh_jassol]

Christmas eve software update... guess shows how much everyone knows about the Tesla process.

 

[ahkahn]

Historically Tesla releases some of their best releases just shy of (and even on) Christmas. Often they spin it as a "Christmas present". I believe they did the Model X dance release a day or two before Xmas, then there was the Ho Ho Ho release, and last year it was the more extensive battery prewarming, seat warmers, and other cold weather enhancements.

Bumping my crystal ball post from before.... Tooting my own horn, and not even using the app!

 

[cwerdna]

The discussion is about the period before holidays, not during the holidays (where the devs should be home). Most changes can be be unwinded/reverted just as easily as they are deployed as long as the CD is well under control.

I do agree that not-easily-unwindable upgrades to 3rd party software that you don't have 100% control can wait until a time where most people can be alert. Luckily 99% of changes are unwindable.

I'm referring to Christmas and New Year's Day with some period of time before and after it.

In some companies, software the end user uses/sees has dependencies on numerous upstream teams. Something that goes awry can lead to "outages" and bad publicity, even making mainstream media. I cannot really elaborate further.

Also, in some cases, software needs to be submitted to a 3rd party company for approval. And, that 3rd party company actually shuts down and doesn't accept submissions during a several day period around Christmas. So, if something critically wrong is found in the app that requires the app to be patched, there's no way to even submit a fix to that 3rd party until they re-open, let alone get it accepted/rejected and picked up by customers.

Seriously, if you and folks like mswlogo worked at my company and tried to be cowboy by making unnecessary production changes against company policy/best practices for these holidays, you'd be getting talked to pretty quickly. If your unnecessary changes broke something critical in production during such a period, that could be a career limiting move.

 

[JBT66]

The Tesla app and the functionality in the car is not revenue generating. No ones worried. If they break something they’ll get it on working on it Thursday.

Agile development is all about quickly responding to bugs.

 

[Fredneck]

I'm not sure Tesla will do anything about code "freezing". They consider the mirrors "freezing" to be adverse conditions and not under warranty.

 

[ChrML]

I'm referring to Christmas and New Year's Day with some period of time before and after it.

In some companies, software the end user uses/sees has dependencies on numerous upstream teams. Something that goes awry can lead to "outages" and bad publicity, even making mainstream media. I cannot really elaborate further.

Also, in some cases, software needs to be submitted to a 3rd party company for approval. And, that 3rd party company actually shuts down and doesn't accept submissions during a several day period around Christmas. So, if something critically wrong is found in the app that requires the app to be patched, there's no way to even submit a fix to that 3rd party until they re-open, let alone get it accepted/rejected and picked up by customers.

Seriously, if you and folks like mswlogo worked at my company and tried to be cowboy by making unnecessary production changes against company policy/best practices for these holidays, you'd be getting talked to pretty quickly. If your unnecessary changes broke something critical in production during such a period, that could be a career limiting move.

Clearly Tesla does not seem to think this is necessary. Referring to the Christmas update with a substantial amount of changes.

 

[ChrML]

Seriously, if you and folks like mswlogo worked at my company and tried to be cowboy by making unnecessary production changes against company policy/best practices for these holidays, you'd be getting talked to pretty quickly. If your unnecessary changes broke something critical in production during such a period, that could be a career limiting move.

This is probably culture and company dependent. Our company has no such policy, hence there is no policy to break.

There is never one person behind a deployment, there is always a team, hence no cowboy problem. The team's confidence in a patch is evaluated the exact same way any time of year, based on a total evaluation. I strongly believe in finding a good way to work which in general has high productivity and not likely to break things. Then follow that, regardless of holidays.

I'm not saying any patch should be applied at any time. Example if a patch should be closely monitored in production for weeks, then the team will wait until after holidays. What I'm saying is I don't believe in introducing company policies for this like "full code-freeze between X and Y". Because the teams are perfectly capable of applying their standard way of working. If they break things that cause customer issues, any time of year, then that's what should be worked on for improvements.

I don't believe in top-down management and unnecessary policies. I believe in distributed responsibility, and trust the teams to make good decisions and common sense. The management's (my) job is only to make sure the team's coordinate well between, and be the invisible second goalkeeper that intervienes if something is about to go completely wrong.

 

[cwerdna]

^^^
My company actually has very few policies, by design.

I cannot elaborate but I doubt your company has anywhere near the number of customers we do and where outages can and do make mainstream media news (in my country and those where we have lots of users). If you were at my company involved in anything that eventually makes it to production, you'd be in agreement w/one of the few policies/practices we have around holidays. Seriously.

 

[Fredneck]

I'm referring to Christmas and New Year's Day with some period of time before and after it.

In some companies, software the end user uses/sees has dependencies on numerous upstream teams. Something that goes awry can lead to "outages" and bad publicity, even making mainstream media. I cannot really elaborate further.

Also, in some cases, software needs to be submitted to a 3rd party company for approval. And, that 3rd party company actually shuts down and doesn't accept submissions during a several day period around Christmas. So, if something critically wrong is found in the app that requires the app to be patched, there's no way to even submit a fix to that 3rd party until they re-open, let alone get it accepted/rejected and picked up by customers.

Seriously, if you and folks like mswlogo worked at my company and tried to be cowboy by making unnecessary production changes against company policy/best practices for these holidays, you'd be getting talked to pretty quickly. If your unnecessary changes broke something critical in production during such a period, that could be a career limiting move.

If a third party company is reviewing software for approval and they let a release slip through with a "critical" problem, you had better bet they will be open 24/7 to get a patch released.

 

[ChrML]

If a third party company is reviewing software for approval and they let a release slip through with a "critical" problem, you had better bet they will be open 24/7 to get a patch released.

Not necessarily. We use 3rd party certification for small parts of our ecosystem (embedded systems), because that software follows a standard that requires certification (code-review + testing).

However they only care about testing that we are following the standard. There are plenty of cases that they don't test, and things not covered by their tests, that could affect our customers negatively.

This is not a real problem though. Things like this is planned for by the team responsible for that part, and the software can be rolled back just as easily as they are deployed (the old version is still certified even if you make a new version). So we don't need the 3rd party available to solve the problem.

 

[cwerdna]

If a third party company is reviewing software for approval and they let a release slip through with a "critical" problem, you had better bet they will be open 24/7 to get a patch released.

Haha. Here's an example of a 3rd party that is closed for submissions Dec 23 to 27: App Store Connect Holiday Schedule - News - Apple Developer.

Not necessarily. We use 3rd party certification for small parts of our ecosystem (embedded systems), because that software follows a standard that requires certification (code-review + testing).

However they only care about testing that we are following the standard. There are plenty of cases that they don't test, and things not covered by their tests, that could affect our customers negatively.

There's no way in hell Apple does comprehensive testing of apps submitted to their Mac and iOS app stores. They seem to care more about whether the app launches, conforms to their rules (App Store Review Guidelines - Apple Developer) and doesn't break their rules.

There are MANY MANY things that could be hit that end up being serious enough for us to want to fix quickly that cannot be caught by a 3rd party trying to achieve the above. Every time we've had to release a quick fix version or fix something on the server side, it was never ever caught by that 3rd party.

This is not a real problem though. Things like this is planned for by the team responsible for that part, and the software can be rolled back just as easily as they are deployed (the old version is still certified even if you make a new version). So we don't need the 3rd party available to solve the problem.

Yes it is. Apple doesn't allow rolling back. One has to submit a "new" version: either a build with the fix or an old build w/a newer version number. It has to go thru the approval process again. None of this happens instantaneously and cannot happen during the above closure time.

 

[ChrML]

Yes it is. Apple doesn't allow rolling back. One has to submit a "new" version: either a build with the fix or an old build w/a newer version number. It has to go thru the approval process again. None of this happens instantaneously and cannot happen during the above closure time.

Yeah, with Apple it's a concern. We don't have iOS software yet, but when we do I guess it should trail the Android release.

 

[EVNow]

The Tesla app and the functionality in the car is not revenue generating. No ones worried. If they break something they’ll get it on working on it Thursday.

Agile development is all about quickly responding to bugs.

There are generally 2 reasons for code freeze
1. Some critical business function that is too risky if it gets affected. A good example would be internal financial s/w around ER. Or companies with retail e-commerce around Black Friday / Christmas etc.
2. Not releasing just before holidays because they will be short on staff to quickly debug & fix. This may not be a formal code freeze. We just take this into account when planning releases.

Since nobody (apparently) gets vacation in Tesla, they don't do any code freezes

These don't apply to critical bug fixes, obviously. Only planned releases.

ps : Tesla might still have actual code freezes for some internal software (1 above). Just that they don't care about (2).

 

[boonedocks]

Apparently Tesla doesn't care about freezing (or letting people take time off during holidays).


Since nobody (apparently) gets vacation in Tesla, they don't do any code freezes

Vacation or not has anyone tried to actually get someone on the phone at Tesla as of the last year or so....I don't think software freezing is even in Tesla's vernacular because you can't hardly report a problem if you wanted to on a good day.