Email or username:

Password:

Forgot your password?
Kevin's posts Post Back to profile
Top-level
Kevin Beaumont

For the people thinking ‘shouldn’t testing catch this?’, the answer is yes. Clearly something went wrong.

This isn’t CrowdStrike’s first rodeo on this, although it is the most severe incident so far.

Eg just last month they had an issue where a content update pushed CPU to 100% on one core: thestack.technology/crowdstrik

Truthfully these issues happen across all vendors - I’ve had my orgs totalled twice now by AV vendors, one while I was on holiday abroad and had to suspend said holiday.

20 July at 10:46 | Open on cyberplace.social
44 comments
Kevin Beaumont replied to Kevin

Btw, that isn’t to excuse it or any vendor. CrowdStrike have gotta be better at this stuff. And they’ll have to, as if they aren’t transparent customers will flee.

It’s a warning shot to all AV/EDR/XDR vendors that if you fuck up availability, your brand will become failure. It’s harsh but that’s the media cycle and modern world.

20 July at 10:51 | Open on cyberplace.social
Kevin Beaumont replied to Kevin

Microsoft estimate almost 9 million Windows devices are impacted by the CrowdStrike incident (likely from crash telemetry). blogs.microsoft.com/blog/2024/

20 July at 15:35 | Open on cyberplace.social
Kevin Beaumont replied to Kevin

Hackers reboot announced for 2025, trailer dropped
20 July at 17:41 | Open on cyberplace.social
Kevin Beaumont replied to Kevin

The Verge has a quick look at the orgs trying to recover from the Crowdstrike incident.

If you’re wondering why it’s dropped off the radar of most press, they think it’s over as Down Detector looks okay (which, to be clear, is not good logic).

theverge.com/2024/7/21/2420296

21 July at 17:15 | Open on cyberplace.social
Kevin Beaumont replied to Kevin

How much is a significant number?
22 July at 17:41 | Open on cyberplace.social
Kevin Beaumont replied to Kevin

Interesting - did anybody keep a list of tweets by CrowdStrike staff during the start of the incident? This one has been deleted. x.com/brody_n77/status/1814186
22 July at 17:52 | Open on cyberplace.social
Kevin Beaumont replied to Kevin
Kevin Beaumont replied to Kevin

Crowdstrike are touting auto remediation of blue screen as an opt in feature.

However, I just tried it - it’s not very successful, most boots still blue screen of death. I think CS need to be careful on messaging about this as it sounds like they’re offering it as a silver bullet. It only works if networking kicks in and the agent updates before Windows finishes booting.

reddit.com/r/sysadmin/comments

22 July at 23:54 | Open on cyberplace.social
Kevin Beaumont replied to Kevin

Delta cancelled another 20% of US flights yesterday as they struggle to recover from CrowdStrike incident bankinfosecurity.com/blogs/cro
23 July at 9:32 | Open on cyberplace.social
Kevin Beaumont replied to Kevin

CrowdStrike have published a video on YouTube about how to remediate PCs: youtube.com/watch?v=Bn5eRUaMZX

(Despite the name, Self-Remediation, it is manual).

23 July at 11:08 | Open on cyberplace.social
Kevin Beaumont replied to Kevin

Delta are still struggling, suspending additional services.
23 July at 14:08 | Open on cyberplace.social
Kevin Beaumont replied to Kevin

Upguard have published a list of companies they say are impacted by the CrowdStrike 'Global IT Outage', based on public reporting.

upguard.com/crowdstrike-outage

Edit: obviously it’s missing most companies as most companies aren’t disclosing publicly.

23 July at 14:12 | Open on cyberplace.social
Kevin Beaumont replied to Kevin

If anybody wonders what the file that took down 8.5 million Windows systems looks like.. it was 41kb in size. The only validity checking I can see CrowdStrike driver does is to check the first few bytes match the pattern seen in the screenshot before loading and executing.
23 July at 16:41 | Open on cyberplace.social
Kevin Beaumont replied to Kevin

The US Department of Transport has opened an investigation into Delta over the disruption related to CrowdStrike incident.

Good luck to the CrowdStrike account manager for Delta.
23 July at 16:57 | Open on cyberplace.social
Kevin Beaumont replied to Kevin

The initial Post Incident Review is out from CrowdStrike. It’s good and really honest.

There’s some wordsmithing (eg channel updates aren’t code - their parameters control code).

The key take away - channel updates are currently deployed globally, instantly. They plan to change this at a later date to operate in waves. This is smart (and what Microsoft do for similar EPP updates).

crowdstrike.com/falcon-content
24 July at 7:49 | Open on cyberplace.social
Kevin Beaumont replied to Kevin

By ‘this is smart’ I mean ‘this is smart… now’. Obviously they shouldn’t have been globally, simultaneously deploying kernel driver parameter changes across all customers: it was waiting to go wrong.

They still are btw, as it will take a while to engineer the correct way of doing it.

24 July at 8:26 | Open on cyberplace.social
Kevin Beaumont replied to Kevin

On insurance and CrowdStrike, Parametrix claim amongst just the Fortune 500 companies, they are facing $5.4bn in losses, of which around 10% will be covered by insurance.
theguardian.com/technology/art

24 July at 17:33 | Open on cyberplace.social
Kevin Beaumont replied to Kevin

CrowdStrike have won this year's Pwnie Award for Epic Fail, which will please @qwertyoruiop.
24 July at 17:51 | Open on cyberplace.social
Kevin Beaumont replied to Kevin
Kevin Beaumont replied to Kevin

If you want to know something crazy:

- This year TCS migrated their EDR to CrowdStrike
- Then they announced a strategic partnership with CrowdStrike
- Then they lost all their systems
- They’re just finishing recovery today, 6 days in
- Then they got a $10 Uber Eats voucher
- …which got cancelled due to Uber flagging CrowdStrike’s account as fraudulent
24 July at 23:20 | Open on cyberplace.social
Kevin Beaumont replied to Kevin

CrowdStrike are… having a week.
25 July at 8:40 | Open on cyberplace.social
Kevin Beaumont replied to Kevin

Questions for your EDR providers (do not assume they are experts in availability):

- What are your different update processes?
- How do you test them?
- Do you dogfood test them?
- Do you roll them out in waves? What are the details, eg what percentages and when?
- Do you monitor failures and roll back?

25 July at 9:13 | Open on cyberplace.social
Kevin Beaumont replied to Kevin

CrowdStrike staff members are selling CrowdStrike monopoly sets they were given on eBay.
25 July at 13:58 | Open on cyberplace.social
Kevin Beaumont replied to Kevin
Kevin Beaumont replied to Kevin
Kevin Beaumont replied to Kevin
Kevin Beaumont replied to Kevin

There’s a really good discussion on @riskybusiness’s YouTube show about the CrowdStrike incident.

About the 3 minute mark @alex made me realise I was far too kind to CrowdStrike. He rightly rips them apart.

youtu.be/EGRqtscp4eE

29 July at 23:00 | Open on cyberplace.social
Kevin Beaumont replied to Kevin
Kevin Beaumont replied to Kevin

Re the Delta case - the lawyer they’ve hired successfully sued Microsoft previously on behalf of the US government, and the decision was upheld on appeal too. The ruling almost lead to the breaking up of Microsoft.

The following US government backed out of the case.

Bill Gates said at the time the lawyer was “out to destroy Microsoft”.

So there’s a chance here the CrowdStrike incident may end up having implications across vendor industry around warranties etc, we’ll see.

30 July at 12:14 | Open on cyberplace.social
Kevin Beaumont replied to Kevin
Kevin Beaumont replied to Kevin

Replacing an XDR platform at scale takes some time, so if you’re wondering what the translation of Elon’s tweet about Crowdstrike is:

Elon: can we replace Crowdstrike?
Somebody: yes, we’ll begin looking into it but..
Elon: job done

Of course.. given how the Twitter takeover happened maybe he just got them to uninstall it and #yolosec
31 July at 10:15 | Open on cyberplace.social
Kevin Beaumont replied to Kevin

Delta’s CEO has confirmed they plan to take legal action against CrowdStrike after incurring a $500m loss

6 minute video interview: cnbc.com/2024/07/31/delta-ceo-
31 July at 14:01 | Open on cyberplace.social
Kevin Beaumont replied to Kevin
Kevin Beaumont replied to Kevin

CrowdStrike made a net loss of $845m between 2018 until this year, and has taken on $743m of debt during this period.

1 August at 7:41 | Open on cyberplace.social
Kevin Beaumont replied to Kevin

Spirit Airlines in the US anticipates a $7.2 million hit to its third-quarter operating income due to operational disruptions caused by the CrowdStrike incident, which forced the carrier to cancel 470 flights.

1 August at 12:24 | Open on cyberplace.social
Kevin Beaumont replied to Kevin

Here's the Delta boss on his thoughts about the CrowdStrike incident.

They had 40k Windows Server boxes alone, all with BitLocker full disk encryption enabled, all of which wouldn't boot and weren't fixable without manually unlocking BitLocker. That had gone all in with CrowdStrike + Microsoft's most premium offerings.

He has a really good point about how tech companies have become obsessed with growth as their only metric of success, and customer satisfaction is not on the radar.

1 August at 18:12 | Open on cyberplace.social
Kevin Beaumont replied to Kevin

There's a really mad moment in that interview where they ask them what assistance CrowdStrike have offered, and he essentially says nothing, not even a lunch voucher.

What a time to be alive.

1 August at 18:27 | Open on cyberplace.social
Kevin Beaumont replied to Kevin

CrowdStrike’s website then vs now
2 August at 15:28 | Open on cyberplace.social
Kevin Beaumont replied to Kevin

CrowdStrike complained to Cloudflare about a CrowdStrike parody site… and Cloudflare took it down. Without a court order. clownstrike.lol/crowdmad/

Cloudflare recently announced they have become a strategic partner with CrowdStrike: cloudflare.com/en-gb/press-rel
2 August at 20:52 | Open on cyberplace.social
Kevin Beaumont replied to Kevin

Additionally to loop this in, CrowdStrike submitted a takedown for a parody label (they’ve since rescinded it after being called out).
2 August at 20:57 | Open on cyberplace.social
Kevin Beaumont replied to Kevin

We’ve reached the part of the brand cycle where people are using CrowdStrike as an excuse theverge.com/2024/8/2/24212298

3 August at 6:15 | Open on cyberplace.social
Kevin Beaumont replied to Kevin

360 takes a look at the Crowdstrike kernel drivers - finds they implement an eBPF like system, contain a wide attack surface, don’t check validity of update files (eg no signing of updates) and claim they contain conditions for LPE and RCE vulnerabilities. mp.weixin.qq.com/s/uD7mhzyRSX1

Before people write this off as ‘the Chinese’, I’ll give you a hint: there really, really should be security research about the security of security products across all vendors. I’ve seen things.

360 takes a look at the Crowdstrike kernel drivers - finds they implement an eBPF like system, contain a wide attack surface, don’t check validity of update files (eg no signing of updates) and claim they contain conditions for LPE and RCE vulnerabilities. mp.weixin.qq.com/s/uD7mhzyRSX1

3 August at 6:47 | Open on cyberplace.social
Kevin Beaumont replied to Kevin

Previously on Crowdstrike Falcon vulnerability research, check out this timeline where they tried to use NDAs to avoid disclosure, then fixed it without telling anybody. modzero.com/modlog/archives/20

3 August at 21:21 | Open on cyberplace.social
VessOnSecurity replied to Kevin

@GossiTheDog I had a similar experience with Microsoft.

A junior colleague found a 1-click exploit in Skype for Linux. We reported it. We didn't want any bounty money - just to be assigned a CVE that we could include in our paper. Microsoft's response was essentially "it's not an RCE, go away".

Then they silently fixed it, without crediting us.

Never every doing the "responsible disclosure" dance with Microsoft ever again.

4 August at 5:43 | Open on infosec.exchange
Go Up