Gregory's ServerIn an unexpected turn of events, a sensible take on #Crowdstrike from the Orange Site.
 112 comments
 @donaldball @calamari I don't know SOC2 that well but I do work on critical infrastructure that is certified (with 0 findings!) to a similar German standard despite not using any of these scary products. Yes you need to explain to your auditor how you intend to meet your security objectives despite not having bought the proprietary appliance that claims to magically make you do that. But you'll manage. @donaldball @calamari "If you don't buy XYZ you'll surely fail your audit" is repeated across the industry as a truism but barely ever put to the test. @muvlon @donaldball @calamari "it tixs a box". As long as it easier to deploy a software which tix a box as to discuss with your auditor each and every year why you insist do do it by yourself we will see incidents like this today 🤷 @muvlon @donaldball @calamari exactly. you can work with the regulatory entities & auditors, but you have to know what you do. This is far more about the companies doing the implementations than the compliance frameworks themselves. Companies will do the bare minimum to pass audit and then completely ignore the ongoing audits, assessments, and improvement cycles demanded by the compliance framework. With SOC2 you can get away with some of this, less so in FedRamp, and even less so in ISO but companies don't want to spend money to mitigate risks - all they want is tech magic they can ignore. @gaysteve @donaldball @calamari this sounds like the most accurate description to me. I’ve been through the SOC process a few times, I can see how companies want to take some mostly reasonable norms on what they’re supposed to audit and try to abstract it to a software package. I have always found the anti-malware norms both reasonable in principle and vexing in implementation myself. This is where invasive endpoint software shows up.  Checklists are only as useful as the knowledge necessary to know why the checklist exists. Pilots and surgeons train for extensive periods so they learn why they need to go through their checklists. What happens far too often is checklists turning into ritual disconnected from the rationale. Religion often has this problem. Many of the rituals of religion have roots in Something Deep From Back In The Day, but that link, with time, has since worn away.  @Aphrodite @calamari we've progressed from "cargo cult" to "checklist cult" At least with the former, we got to build cool bamboo models of planes and control towers. tbh the Adeptus Mechanicus of 40K make too much sense in that framing they don’t know why tech works, they just know to do the rituals and they can make a thing @Aphrodite @cjust @calamari the last paragraph seems to apply to modern day youth using AI to do homework as well. @cjust @Aphrodite @calamari That's because the reviewing the checklists can then be - no offence intended - offloaded to cheap workers in 2nd and 3rd world countries who are judged by the checklists they sign off. There is no room for critical thinking or adapting to the particular situation. I see this happening daily. @Aphrodite @calamari They found out of many dozens of entries in the list that it was rare for any of them to actually be done, including the one to double check the patient's name prior to anaesthetic, to make sure they're about to operate on the right person! @Twirrim @Aphrodite @calamari As someone who's had a lot of surgeries & procedures involving anesthetic in the past 5 years, I'm quite thankful that my identity has been checked and re-checked every time.  @Aphrodite @calamari an even more important check box is change management. How can you have effective change management when updates are applied automatically? If compliance frameworks require automatic updates, then they're broken, and given what has just happened, I really hope they'll be fixed. Sure, have EDR etc, but the updates need to be validated, then rolled out by the organisations. Sadly, as the world just discovered, there is no silver bullet when it comes to security. @JessTheUnstill @Aphrodite @calamari Agree and understand the vendors want auto update. They need to be told where to stick that idea. My experience is a lot of that is due to the updates on MS Windows being hard to manage. Interesting to hear about the CrowdStrike release cadence. I've never used it. In my world, we manage what is released to servers and when.  You can do similar with Windows. They have "update rings". It lets you keep your systems on auto-update so IT doesn't have to manually faf with it, but you can have canaries before prod borks. https://learn.microsoft.com/en-us/mem/intune/protect/windows-10-update-rings @JessTheUnstill @Aphrodite @calamari Excellent! However, that is for the software (incl drivers) that Microsoft supply. What about all the other random software you need to install? I can't remember about Office or Adobe or Chrome whether they have things like that. It's been a few years since I worked at a Windows corp and interacted with endpoint engineering. @JessTheUnstill @Aphrodite @calamari Fair enough, I don't work in one either (thankfully). Interesting discussion though! Thank you. @puck My management didn't like the idea as they were happy to transfer responsibility to zscaler as then it wouldn't be their fault if it broke. I won though... @puck @hmhackmaster Excellent to hear about your success, and that you've been vindicated (yeah, different tool, but same context)! And agreed, many orgs will try to transfer responsibility. Will be interesting to see how well that goes. @puck I care more about uptime and reliability than the blame game. But I am also the kind of person who has a reputation for making reasonable decisions and assuming responsibility when this things go wrong. If taking responsibility (and not dodging accountability) costs me my job then that's clearly a sign the org has lost confidence in me and it was time to move on anyways. Hasn't happened to me yet though, and I have made some pretty big mistakes!  @Aphrodite @calamari When a company gets hacked and sued, they have to answer to "were you negligent in protecting against this or were you just unlucky?". Courts are incompetent in determining this, and companies are mostly actually negligent (because they don't want to pay for it), so we get these "best practices" checklists instead. How do you legislate competence? Most companies can't even determine if the people they hire are competent! @calamari If I had a dollar for every hour that Zscaler disrupted someone on my team or one of our stakeholders internally because of how aggressive it got, I’d be able to retire. @calamari What's interesting about this is that best practice is to be on n-2, or two versions behind on driver updates. Which we are. But this was a policy update, or a channel update where they modified modified the detections such that it borked ALL versions of the driver. TL,DR senior leadership assumed we would be covered to prevent this, but n-2 doesn't mean what everyone thought it meant. @hudsoncress @calamari They don’t like to be behind on security updates though. These were definition files so being n-2 would mean exposure to 1 and 2 day critical security vulnerabilities. This isn’t the first major crisis caused by rapid fire security updates. It won’t be the last. @LogicalApex @calamari. It's just wild that Crowdstrike pushes that apparently untested definition file globally, and was able to hit hundreds of millions of endpoints before anyone saw it was literally breaking every computer it touched? I mean... WFT @LogicalApex @calamari also, quoth the vendor, "There is no way at this time" to turn off channel updates... SLT is gonna love that.  @hudsoncress @LogicalApex @calamari Because if not, if it were malware, there would be at least hope that the responsible would be prosecuted at some point. @yacc143 @hudsoncress @calamari Companies put boiler plate language in their contracts. They either absolve themselves of any liability for damages or limit their liability to your license fee. Probably also includes a mandatory arbitration clause to further limit liability fallout. I bet that’s the case here too. 😬 @LogicalApex @yacc143 @calamari what’s interesting is how we all assumed n-2 would save us from this but nobody was clear beforehand that the real risk was a policy update, not a driver version. @LogicalApex @hudsoncress @calamari Interestingly, so they sell you a product that does something, on most days what the sales prospectus says, and on some days destroys your IT, and say enjoy, you cannot sue us, and the IT crime laws don't apply to us, as you voluntary provided us with access to all your IT. Now purely as the IT guy, that is GREAT. @LogicalApex @hudsoncress @calamari Admittedly he crossed out this paragraph when I explained to him the issues ;)   @calamari Not wrong, but I still feel like pushing out a critical update that breaks global commerce isn't entirely on the airlines and banks. Delta Airlines has to check that box. CrowdStrike sold them a product saying that they can safely check that box with their product. @vincent @calamari It's still at least somewhat on Delta and others for allowing CrowdStrike to blindly update their entire fleet without using progressive rollouts and canaries. There is a lot of fault at play here that a more thoughtful approach to compliance would have made this bad, but not catastrophic. @vincent @calamari (Then CrowdStrike perhaps would not be so willing to claim so many things.) The whole thing of companies working without accountability mega-sucks. So I broke the Internet today, but there will be 0 feedback to prevent this in the future.  @yacc143 @vincent @calamari I always recall a discussion I had with an aerospace engineer about liability for compiler bugs, and he said what was the point, why would they want to end up owning Cray Systems if their planes started falling out of the air. Of course that was back in the 90s when we didn't expect planes to fall out of the sky. @ianturton @vincent @calamari And in our "capitalist" world, this happens by assigning costs. As long, there is no direct cost for bugs, even catastrophic ones, companies will ship products with catastrophic bugs just to meet the schedule some marketing egg head invented for artificial reasons.   @calamari what the difference between compliance and checkbox compliance… they are almost always checkbox exercise!!! Agree with everything you said! Number of times I have fought compliance auditors about outdated checkbox compliance requirements… sigh!!! @calamari funny thing is I don’t think SOC2 can be termed regulation precisely. The norms of what you put in SOC2 reports are, unless working with government, an emergent phenomenon of private industry expectations. The basic framework of SOC2 is “you say you do these things, audit firm proves it to some extent.”  @calamari I'm willing to blame CrowdStrike for building a huge business around exploiting that organisational checklist dysfunction and then not bothering to take even basic precautions to avoid bringing all their customers down. After all, they're the ones who pitched themselves as the experts; their customers' sin was ignorance and misplaced trust.  
100% on the money with that bit.     @calamari 100% on this. A big problem is, the people with the technical know how to say "this will fail in a big way" are either ignored or so jaded they don't speak up. The people who didn't scream "this is shit" are as much to blame as the idiots in governments pushing through ham fisted legislation that's supposed to stop white collar crime, but never seems to    @calamari@mastodon.social this kind of thinking, is how disasters like Chernobyl happened. While this isn't as destructive ad a failed nuclear reactor, it keeps happening. At some point, some things should change.  LB 👆🏻 Even our little factory has been getting pressure to deploy endpoint surveillance onto every user device, because some of our customers want us to have “cyber insurance” in order to do business, and the insurer lists endpoint threat detection along the things we should buy. Classic box-ticking behaviour.   @calamari I’ve run into people complaining about HIPAA not prescribing solutions, and this articulates exactly why that is such a bad idea.  @Qbitzerre @calamari "Nobody ever got fired for buying CrowdStrike" ... because nobody's fixed the HR team's computers yet.  @calamari Given two paths, one that reduces risk and the other that reduces liability, the system always rewards the latter decision, even when it increases overall risk. @calamari @calamari checkbox compliance attitudes kinda tick me off, there are opportunities to effect thoughtful change but not if you don't respect the process or have the confidence to make decisions. You can comply with the letter of the rule and pass off responsibility or comply with the *spirit* of the rule and _take_ responsibility.  @calamari Tonight on the PBS News Hour, they had Bruce Schneier as a guest to weigh in. His summary was, "It's all economics", companies want to grow fast and break things, and the markets want that. And companies that buy those products,... as you point out, are trying to tick a box on an audit form. If we did this the `right way` it would cost more money,...  @calamari link? What is the context? Is ZScaler the third-party software? Do you know how it was involved in this failure?  @calamari oh man, I hate ZScaler with a passion. It makes our development machines so slow, and we can't even do something basic like checking if a deployed site runs a correct SSL certificate  @calamari ehhhhhhh from being lightly involved in compliance, I never saw a, "oh no suddenly there's new compliance!". It was *always* "we need to push all these features first because an exec has a bonus tied to it. (actual, literal months pass) Oh crap we're late now and have to rush out a fix". @calamari IS 27001 is a perfect example of the bureaucratic way of ensuring system security. @calamari @kushal Not sure this take is as sensible as it might appear. Regulations exist for a reason. Regulators are not responsible for an organisation's flawed attempt to satisfy the regulation. A business is responsible for managing its own risks and meeting regulatory requirements. No one has forced these businesses to operate with disregard for their own business risks.  @calamari why unexpected, though? The orange site has most often a few sensible and nuanced takes on hot topics. They then bubble up and push the tech bro nonsense away almost always. Not on the < 30 comment threads obv. But hot topics? I keep getting positively surprised there. @calamari One thing I would add to this, which I have witnessed too often first hand, is the incompetence in evaluating those risks. When the consultants say that this should be done, there is no further probing into whether or not this actually mitigates or even introduces risks.  @calamari@mastodon.social Heard of a company that would rather lock up account (pam faillock or sth) instead of using properly configured fail2ban/denyhosts to block password bruteforce attack based on IP because WOOHOO COMPLIANCE SAID SO smh :blobcatshrug2:   @calamari Compliance is meant to be a tool, to expose businesses to robust thinking around problems. IMO anyone, who knowingly does check-box compliance, (we'll just buy in / defer compliance) for any reason, should be shown the door.  @calamari At Amazon in 2019 I argued against the Falcon sensor on the hosts I was responsible for. I agreed their service was good, but the lack of transparency, control and even ability to test changes made it a no go for me. I asked why we weren't either building our own solution or just buying them. I was overruled and I'm sure those systems, if they still exist, had a fun past few days. Again, I like CS, but major tech companies are not short on resources or expertise. |
@calamari I have a whole theory about how the development processes that SOC2, Fedramp, etc. all but mandate in order to survive an audit freeze the design of covered systems, often prematurely, and actively impede the evolved practices that might otherwise have improved their quality and reliability.