Gregory's ServerA lot of vendors make it intentionally difficult to even do manual validation and deployments these days. Windows, Chrome, Edge, Adobe, etc. all really want to auto update. One of the odd parts of this particular outage is that CrowdStrike updates are SUPPOSED to go out in stages where your test machines are on update N, staging machines are on update N-1, and prod machines are on N-2. So somehow they not only made a bad update, but they also violated their own release cadence by pushing it out to all machines no matter what version they're scheduled to be on.
|
12 comments
 You can do similar with Windows. They have "update rings". It lets you keep your systems on auto-update so IT doesn't have to manually faf with it, but you can have canaries before prod borks. https://learn.microsoft.com/en-us/mem/intune/protect/windows-10-update-rings  @JessTheUnstill @Aphrodite @calamari Excellent! However, that is for the software (incl drivers) that Microsoft supply. What about all the other random software you need to install? I can't remember about Office or Adobe or Chrome whether they have things like that. It's been a few years since I worked at a Windows corp and interacted with endpoint engineering. @JessTheUnstill @Aphrodite @calamari Fair enough, I don't work in one either (thankfully). Interesting discussion though! Thank you. @puck My management didn't like the idea as they were happy to transfer responsibility to zscaler as then it wouldn't be their fault if it broke. I won though... @puck @hmhackmaster Excellent to hear about your success, and that you've been vindicated (yeah, different tool, but same context)! And agreed, many orgs will try to transfer responsibility. Will be interesting to see how well that goes. @puck I care more about uptime and reliability than the blame game. But I am also the kind of person who has a reputation for making reasonable decisions and assuming responsibility when this things go wrong. If taking responsibility (and not dodging accountability) costs me my job then that's clearly a sign the org has lost confidence in me and it was time to move on anyways. Hasn't happened to me yet though, and I have made some pretty big mistakes! @hmhackmaster @puck @JessTheUnstill @Aphrodite @calamari @nonnihil I think you are completely right from a business point of view, but from some upper-management person's viewpoint the "it's the vendors responsibility" is the path to ensure their decision can't come back to bite them. |
@JessTheUnstill @Aphrodite @calamari Agree and understand the vendors want auto update. They need to be told where to stick that idea. My experience is a lot of that is due to the updates on MS Windows being hard to manage.
Interesting to hear about the CrowdStrike release cadence. I've never used it.
In my world, we manage what is released to servers and when.