Email or username:

Password:

Forgot your password?
Infrogmation

Ian Muir explains the context for the Crowdstrike Microsoft disaster.

Facebook post by Ian Muir 19 July 2024:

OK, I will answer here in a group since a lot of non tech friends have asked me. Lots of stuff broke today because a company called Crowdstrike released a bad piece of software that a lot of systems rely on. Here are some answers to common questions people have asked me.
Edit: One note for people who think I'm being dramatic here and that this was all just an accident. This whole thing happened before at a smaller scale with McAffee in 2010. The CTO of McAffee at the time was Crowdstrike's current CEO. This isn't a coincidence, it's a pattern.
Why wasn't the update tested better? Because tech people are expensive and Crowdstrike laid a bunch of them off over last 2 years to save money including testers. Testing is expensive and it's hard to convince a business leader that testing is important because it doesn't "make them money".
Why didn't big tech companies have a way to function without this one piece of software?  Because tech people are expensive and big tech comapnies laid a bunch of them off over last 2 years to save money including the people who help make things stable and flexible. Flexibility is expensive and it's hard to convince a business leader that flexibility is important because it doesn't "make them money".
facebook post by Ian Muir text continues:

Why didn't airlines, governments, telecom companies, and other business have a backup plan?  Because tech people are expensive and many companies laid a bunch of them off over last 2 years to save money including the people who research and implement backup plans. Backup plans are expensive and it's hard to convince a business leader that backup plans are important because they doesn't "make them money".
Wait a minute. How many of these critical systems in tech, transportation, government, and infrastructure are short staffed right now? Pretty much all of them. Good luck.
Wait a minute, all of these people who laid off all the tech workers said they were replaced by AI. What happened? AI was just a bullshit justification to lay off expensive tech people. AI hasn't really replaced many people in tech because making major changes like shifting to AI is expensive and it's hard to convince a business leader to spend money on making changes when they can just fake it because their compensation is affected more by how many tech people they can lay off than how well the companies actually function. Someday, AI might replace a lot of people in tech, but right now most of the jobs were not replaced by AI and were likely just closed or sent overseas.
45 comments
Tinker ☀️

@wendynather @Infrogmation @ReticentTurnip - I absolutely hate being right. I hate seeing these patterns. I hate seeing where they go.

That said, right now is a good time to build actual community and mutual aid.

I dont want to be right about this either.

Michael Fisher

@Infrogmation @TerryHancock Perhaps replacing some of these gigantically-expensive CEOs with an AI model would help the companies save money, hire their productive staff back, and deliver a better and more reliable product? Heck, we know general AI can spout as much bullshit as a real live CEO can, so what does corporate America have to lose?

remote procedure chris

@Infrogmation oo i tried to see if they'd done layoffs too. Amazing how all those people everyone fired were probably doing something

Paul Wilde :blobcatnim: :dontpanic_nobg:

@Infrogmation i remember the mcafee blooper well. Had no idea the same CEO is in play, that's interesting

Professor_Stevens

@Infrogmation

I have been a working computer programmer for nearly 50 years. Over, and over, and over again, I have seen snake-oil products sold with the pitch (sometimes utterly literally), "Fire the programmers!"

Yet here we still are, writing the code that makes the world run. Until, in its latest round of log-rolling, management decides to "cut out the deadwood." This always works in the short term, because our code runs well by itself.

For a while...

Orion (he/him)

@Professor_Stevens @Infrogmation One of the things non-techs (such as myself!) have to have painfully explained to us is: machines need maintenance. They don't run forever by themselves. As you say, they run "for a while," but then when they stop working, we're like WTF? But that's just what happens. It's predictable. It ought to be written into budgets and regarded as an immutable operating cost. But like the OP says, that "doesn't make money," so shit breaks down. All. The. Time.

Orion (he/him)

@Professor_Stevens @Infrogmation There's a directly parallel problem of "it doesn't get votes," too.

Professor_Stevens

@orionkidder @Infrogmation

A badly misguided management paradigm divides operational units into "profit centers" and "cost centers." This, in turn, leads to badly misinformed managers thinking, "Hey, let's increase profits by getting rid of all the cost centers!"

If a center isn't part of why your business is profitable, yes, get rid of it. But it was probably created in the first place because you needed it to make a profit.

Tech support? Yes, if you use tech, then that's a profit center.

Orion (he/him)

@Professor_Stevens @Infrogmation And they will NEVER regard management itself as a "loss centre," but it very often is. At best, they're paid disproportionately to how much they (allegedly) generate, but class solidarity keeps them in high-paying jobs bc once you're A Boss, that's your identity (to them), so you'll get saved from your own incompetence over and over again.

Orion (he/him)

@Professor_Stevens @Infrogmation Fired from one firm for a huge fuck up? No problem, another will hire you and convince themselves it's a steal based on how much you got paid at that last job... that you fucked up.

Archimage

@Professor_Stevens @orionkidder @Infrogmation that assumes the primary purpose of all/most business is to make a monetary ptofit.

Orion (he/him)

@SergKoren @Professor_Stevens @Infrogmation I think it is the purpose of most business, but I also think it shouldn't be.

Archimage

@Professor_Stevens either way, you have not stated your assumptions. Your argument is, therefore, specious.

Korma Chameleon

@orionkidder @Professor_Stevens @Infrogmation there's a saying "if you don't schedule time for maintenance, the machine will schedule it for you"

Orion (he/him)

@KormaChameleon @Professor_Stevens @Infrogmation Oh, that's very good. That said, the machine won't schedule it at all. It's be more like an unannounced emergency meeting.

Ben Rosengart

@Infrogmation @trochee This explanation treats “replacing skilled tech workers with AI” as something that is actually possible with some effort

Which makes me wonder what else it gets wrong

Jeremy Kahn

@fivetonsflax @Infrogmation

It says that the managers **think they can** get away with replacing workers with AI but they haven't found a way to do it yet

Ben Rosengart

@trochee @Infrogmation In the last paragraph he explicitly says the reason is cost.

Jeremy Kahn

@fivetonsflax @Infrogmation

I'm willing to give the benefit of the doubt to the symmetry of all the other arguments.

Making AI "work" isn't possible because the current crop of AI tech actually isn't ever going to functionally replace skilled programming labor.

But this is beyond the horizon of the management consultants who imagine that it _might_ be "good enough"

Jeremy Kahn

@fivetonsflax @Infrogmation

And I agree that there was no need to say "someday AI might replace labor" at all here

Orion (he/him)

@trochee @fivetonsflax @Infrogmation Friendly amendment, it says they *desperately want to* bc by and large, bosses HATE labour with a white-hot gut-level rage that is hard to believe even when you see it directly (I have; I was a union negotiator for a bit). They will jump at any chance, not matter how hair-brained, to eliminate workers. It's become pathological, but it started as ideological. They *believe* workers merely reduce profits rather than *producing* wealth, which is what we do.

Rusted

@fivetonsflax @Infrogmation @trochee
For "AI" read "Indians with runbooks". After all most AI is basically overworked underpaid 3rd world "trainers"

Management has been using underpaid offshore runbook-readers instead of onshore support for years. And it works for a while.

Ann H

@Infrogmation @akosma Tech people are not only expensive, they are also difficult and obstructive the more expensive/senior they are. An industry with quality motivation would call this the value of experience and listen to signals.

Grant

@Infrogmation

Reading that felt like I was in an echo-chamber.

Michael Potter

@Infrogmation This describes my pain. It doesn't mention how the people who warned management about the dangers of austerity were ignored or shouted down, and are now probably being blamed for the results.

UkeBLCatboy

@Infrogmation Why does he go back to 2010? It literally happened with the LINUX version of CROWDSTRIKE

3 MONTHS (THREE) AGO!

neowin.net/news/crowdstrike-br

THREE. MONTHS. AGO. Same software, same problem, just with the linux version...

Vladimir Tarasov

@Infrogmation gosh, the author forgot to mention that 95% of tech people think that they're qualified professionals, but they are not. And it happened more than a decade ago, because of IT bubble.

Listens to Baroque while coding murder.exe :newt:
@Infrogmation funny how he doesn't mention compliance requirements that caused this anti-virus or whatever it is to be installed in the first place
mattscheurich

@Infrogmation maybe us tech people should start to frame information along the lines of preventing losing more money (and marketshare) than it costs

LisPi
@Infrogmation @rysiek For several of those services, it should be mandatory to have adequate fault tolerance, isolation & redundancy (and criminal to fail that).

It doesn't make you money, but if you fail to implement it you are spending the next decade in prison while your company gets reposessed as a public utility or liquidated.
Harris👍Trump👎

@Infrogmation

Sounds right.

What could go wrong?

Note: this also applies in higher-ed and probably any other place that “has computers” … which is everything.

Nazo

@Infrogmation As a side note, since LLMs are not actually AI, "AI" is not actually a replacement for people generally speaking.

So really the plan was to just pretend away the need for humans to fill those positions.

paul

@Infrogmation

they don't pay the price for their mess.

society does.

also see:

data breaches.

regulatory capture.

climate change.

disaster capitalism.

move fast and break things.

Short-termism.

too big to fail.

Sasha

@Infrogmation the missing piece of the puzzle for me is the risk to shareholders.

Shareholders don't care about systems being down if the lost revenue and repair costs are less than the money they made through the layoffs. CEOs mostly spend their time trying to delight their real customers, the shareholders.

Andreas K

@Infrogmation
Well at least 85% correct.

Buying a 3rd party product that claims to fulfil requirement X on your checklist, so you can tick off that item is almost always cheaper than building the capabilities, or even investigating if requirement X on that list even makes sense for your company, and properly documenting that X does not make sense for your company and hence you skipped it.

And if that 3rd party product happens to kill your company, 🤷, you have taken home your bonus already.

@Infrogmation
Well at least 85% correct.

Buying a 3rd party product that claims to fulfil requirement X on your checklist, so you can tick off that item is almost always cheaper than building the capabilities, or even investigating if requirement X on that list even makes sense for your company, and properly documenting that X does not make sense for your company and hence you skipped it.

Go Up