Email or username:

Password:

Forgot your password?
Kevin Beaumont

Crowdstrike published a faulty update. Causes Windows to bluescreen. Driver is C-00000291*.sys. Will cause worldwide outages. Thread follows, I suspect. 🧵

71 comments
Kevin Beaumont

I am obtaining a copy of the driver to see if malicious or bad coding, if anybody else checking let me know.

Kevin Beaumont

If anybody is wondering the impact of the Crowdstrike thing - it’s really bad. Machines don’t boot.

The recovery is boot in safe mode, log in as local admin and delete things - which isn’t automateable. Basically Crowdstrike will be in very hot water.

Kevin Beaumont

You know it was coming...

Crowdstrike's BSOP theme tune

Kevin Beaumont

Sky News has gone off air in the UK.

Kevin Beaumont

Favour to IT folks fixing - could you please copy the C-00000291*.sys file to somewhere and upload it to Virustotal, and reply with the Virustotal link or file hash? It's still unclear if the update was malicious or just a bug.

Kevin Beaumont

I've obtained copies of the .sys driver files Crowdstrike customers have. They're garbage. Each customer appears to have a different one.

They trigger an issue that causes Windows to blue screen.

I am unsure how these got pushed to customers. I think Crowdstrike might have a problem.

For any orgs in recovery mode, I'd suspend auto updates of CS for now.

Kevin Beaumont

If anybody is wondering, the update was delivered via channel file updates in Crowdstrike.

Kevin Beaumont

BBC tracker (they mix up an earlier Microsoft outage, what they're actually tracking is the Crowdstrike issue) bbc.co.uk/news/live/cnk4jdwp49

Kevin Beaumont

The .sys files causing the issue are channel update files, they cause the top level CS driver to crash as they're invalidly formatted. It's unclear how/why Crowdstrike delivered the files and I'd pause all Crowdstrikes updates temporarily until they can explain.

This is going to turn out to be the biggest 'cyber' incident ever in terms of impact, just a spoiler, as recovery is so difficult.

Kevin Beaumont

CrowdStrike's shares are down 20% in pre-market.

Kevin Beaumont replied to Kevin

I'm seeing people posting scripts for automated recovery.. Scripts don't work if the machine won't boot (it causes instant BSOD) -- you still need to manually boot the system in safe mode, get through BitLocker recovery (needs per system key), then execute anything.

Crowdstrike are huge, at a global scale that's going to take.. some time.

Kevin Beaumont replied to Kevin

Crowdstrike statement: bbc.co.uk/news/live/cnk4jdwp49

Basically 'it's not a security incident... we just bricked a million systems'

Kevin Beaumont replied to Kevin

For anybody wondering why Microsoft keep ending up in the frame, they had an Azure outage and- this may be news to some people- a lot of Microsoft support staff are actually external vendors, eg TCS, Mindtree, Accenture etc.

Some of those vendors use Crowdstrike, and so those support staff have no systems.

But MS isn’t the outage cause today.

Kevin Beaumont replied to Kevin

Crowdstrike publishes updated CIA triad

Kevin Beaumont replied to Kevin

By far my fave thing with the Crowdstrike thing is Microsoft saying to try turning impacted PCs off and on again in a loop until you get the magic reboot where CrowdStrike updates before it blue screens.

Kevin Beaumont replied to Kevin

lol Microsoft have put ‘reboot each box 15 times’ on its website

Kevin Beaumont replied to Kevin

The chuckle brothers at NoName attempting to claim they caused the incident. To be super clear, NoName can barely DDoS a bike shed website, and once asked me to make their logo in Minecraft.

Kevin Beaumont replied to Kevin

Probably the funniest BBC news update so far (they’ve cleared the airways for this incident).

Kevin Beaumont replied to Kevin

BBC News at 6 is leading the entire show with this. (They asked me to appear but I was slightly busy).

For the record I spent much of the day trying to tell people it isn’t a Microsoft issue.

Kevin Beaumont replied to Kevin

When I get successfully DDoS’d it’s both a security incident and I’m not protected…

Kevin Beaumont replied to Kevin

Billboards in Times Square blue screen of deathing. Nice way to find out which orgs use Crowdstrike, this 🤣

Source is BBC News, if anybody wondering.

Kevin Beaumont replied to Kevin

Crazy video of flights being ground stopped across the US earlier today, due to the CrowdStrike issue. bbc.co.uk/news/live/cnk4jdwp49

Kevin Beaumont replied to Kevin

*whispers* They work remotely on Friday

Kevin Beaumont replied to Kevin

CrowdStrike have effectively a mini root cause analysis out

Pretty much as everybody knows, they did a channel update and it caused the driver to crash.

If they blame the person who did the update.. they shouldn’t, as it sounds like an engine defect.

crowdstrike.com/blog/technical

Kevin Beaumont replied to Kevin

For the people thinking ‘shouldn’t testing catch this?’, the answer is yes. Clearly something went wrong.

This isn’t CrowdStrike’s first rodeo on this, although it is the most severe incident so far.

Eg just last month they had an issue where a content update pushed CPU to 100% on one core: thestack.technology/crowdstrik

Truthfully these issues happen across all vendors - I’ve had my orgs totalled twice now by AV vendors, one while I was on holiday abroad and had to suspend said holiday.

For the people thinking ‘shouldn’t testing catch this?’, the answer is yes. Clearly something went wrong.

This isn’t CrowdStrike’s first rodeo on this, although it is the most severe incident so far.

Eg just last month they had an issue where a content update pushed CPU to 100% on one core: thestack.technology/crowdstrik

Kevin Beaumont replied to Kevin

Btw, that isn’t to excuse it or any vendor. CrowdStrike have gotta be better at this stuff. And they’ll have to, as if they aren’t transparent customers will flee.

It’s a warning shot to all AV/EDR/XDR vendors that if you fuck up availability, your brand will become failure. It’s harsh but that’s the media cycle and modern world.

Kevin Beaumont replied to Kevin

Microsoft estimate almost 9 million Windows devices are impacted by the CrowdStrike incident (likely from crash telemetry). blogs.microsoft.com/blog/2024/

Kevin Beaumont replied to Kevin

Hackers reboot announced for 2025, trailer dropped

Kevin Beaumont replied to Kevin

The Verge has a quick look at the orgs trying to recover from the Crowdstrike incident.

If you’re wondering why it’s dropped off the radar of most press, they think it’s over as Down Detector looks okay (which, to be clear, is not good logic).

theverge.com/2024/7/21/2420296

Kevin Beaumont replied to Kevin

How much is a significant number?

Kevin Beaumont replied to Kevin

Interesting - did anybody keep a list of tweets by CrowdStrike staff during the start of the incident? This one has been deleted. x.com/brody_n77/status/1814186

Kevin Beaumont replied to Kevin

Crowdstrike are touting auto remediation of blue screen as an opt in feature.

However, I just tried it - it’s not very successful, most boots still blue screen of death. I think CS need to be careful on messaging about this as it sounds like they’re offering it as a silver bullet. It only works if networking kicks in and the agent updates before Windows finishes booting.

reddit.com/r/sysadmin/comments

Kevin Beaumont replied to Kevin

Delta cancelled another 20% of US flights yesterday as they struggle to recover from CrowdStrike incident bankinfosecurity.com/blogs/cro

Kevin Beaumont replied to Kevin

CrowdStrike have published a video on YouTube about how to remediate PCs: youtube.com/watch?v=Bn5eRUaMZX

(Despite the name, Self-Remediation, it is manual).

Kevin Beaumont replied to Kevin

Delta are still struggling, suspending additional services.

Kevin Beaumont replied to Kevin

Upguard have published a list of companies they say are impacted by the CrowdStrike 'Global IT Outage', based on public reporting.

upguard.com/crowdstrike-outage

Edit: obviously it’s missing most companies as most companies aren’t disclosing publicly.

Kevin Beaumont replied to Kevin

If anybody wonders what the file that took down 8.5 million Windows systems looks like.. it was 41kb in size. The only validity checking I can see CrowdStrike driver does is to check the first few bytes match the pattern seen in the screenshot before loading and executing.

Kevin Beaumont replied to Kevin

The US Department of Transport has opened an investigation into Delta over the disruption related to CrowdStrike incident.

Good luck to the CrowdStrike account manager for Delta.

Kevin Beaumont replied to Kevin

The initial Post Incident Review is out from CrowdStrike. It’s good and really honest.

There’s some wordsmithing (eg channel updates aren’t code - their parameters control code).

The key take away - channel updates are currently deployed globally, instantly. They plan to change this at a later date to operate in waves. This is smart (and what Microsoft do for similar EPP updates).

crowdstrike.com/falcon-content

Kevin Beaumont replied to Kevin

By ‘this is smart’ I mean ‘this is smart… now’. Obviously they shouldn’t have been globally, simultaneously deploying kernel driver parameter changes across all customers: it was waiting to go wrong.

They still are btw, as it will take a while to engineer the correct way of doing it.

Kevin Beaumont replied to Kevin

On insurance and CrowdStrike, Parametrix claim amongst just the Fortune 500 companies, they are facing $5.4bn in losses, of which around 10% will be covered by insurance.
theguardian.com/technology/art

Kevin Beaumont replied to Kevin

CrowdStrike have won this year's Pwnie Award for Epic Fail, which will please @qwertyoruiop.

Kevin Beaumont replied to Kevin

If you want to know something crazy:

- This year TCS migrated their EDR to CrowdStrike
- Then they announced a strategic partnership with CrowdStrike
- Then they lost all their systems
- They’re just finishing recovery today, 6 days in
- Then they got a $10 Uber Eats voucher
- …which got cancelled due to Uber flagging CrowdStrike’s account as fraudulent

Kevin Beaumont replied to Kevin

CrowdStrike are… having a week.

Kevin Beaumont replied to Kevin

Questions for your EDR providers (do not assume they are experts in availability):

- What are your different update processes?
- How do you test them?
- Do you dogfood test them?
- Do you roll them out in waves? What are the details, eg what percentages and when?
- Do you monitor failures and roll back?

Kevin Beaumont replied to Kevin

CrowdStrike staff members are selling CrowdStrike monopoly sets they were given on eBay.

Go Up