Email or username:

Password:

Forgot your password?
Security Writer :verified: :donor:'s posts Post Back to profile
Top-level
Security Writer :verified: :donor:

Service desk is now aware. By ‘now aware’ I mean they’re getting hundreds of tickets opened of something they’ll be unable to fix or diagnose.

30 Jan 2023 at 12:24 | Open on infosec.exchange
5 comments
Security Writer :verified: :donor:

Service Desk is now aware that everyone else except them was aware, and now IT is absolutely incandescent.

I think these machines were largely for retained consultants, field workers, regional managers and such.

30 Jan 2023 at 13:02 | Open on infosec.exchange
Security Writer :verified: :donor:

Alright, things are moving faster now, so I might condense-toot to avoid pollution. C-Level is now aware that all the other departments involved were also aware, and that nothing was done, despite being… entirely aware.

Brief call from the CTO to vent first, and second apologise. Finance approved, expedited, a deployment plan is in place and the works are booked for two weeks today.

The fun part is what happens with the best part of 2000 users unable to do ANYTHING relating to the organisation.

The more fun part is the CTO is aware that in 8 days another batch of 400 machines will have the same fate, that everyone else is… wait for it… already aware.

I think this is the first time in my career I’ve been this close to a monumental clusterfuck of this scale… and not have any responsibility for it or to fix it. All we can do is observe.

It’s not Schadenfreude, it’s a bad situation and it’s not fun. I’m not sure there’s a word for this.

Alright, things are moving faster now, so I might condense-toot to avoid pollution. C-Level is now aware that all the other departments involved were also aware, and that nothing was done, despite being… entirely aware.

Brief call from the CTO to vent first, and second apologise. Finance approved, expedited, a deployment plan is in place and the works are booked for two weeks today.

30 Jan 2023 at 13:31 | Open on infosec.exchange
Security Writer :verified: :donor:

So, things have calmed down a bit. There’s currently a longer lead time than expected on the hardware order (I did chuckle a bit, I’ll admit).

There’s only so much shouting that can be done before everyone runs out of steam.

Many mumblings of “lessons learned” and “post mortem” - a bit early for that methinks!

For anyone wondering about the actual politics of this. The machines came to end of life about 12 months ago, and the company being a multi-billion dollar operation managed to eke out another year of manufacturer support. Mostly symbolic as they’re not exactly going to release custom firmware for a handful of devices. They then put a set-in-stone tombstone date on support. 12pm today.

The idea is that it allowed the org to stay compliant with its own (admittedly fantastic) security and compliance policies. As well as the audit req from some of its customers is for hardware to fall under manufacturer support/updates etc. This satisfied both Legal and Compliance.

So for a whole year, they knew this was coming.

But nobody wants all that additional spend, so close to year end. Departments bickering over who’s responsibility it was, who’s budget it came out of, and so on. So everyone dug their heels in, and we continued to shout “iceberg!” from the sidelines.

C-level delegated as they should and middle managers also did so in turn, as they should.

And everyone under them went silent. Not wanting to look bad, have higher spend, rock the boat etc. not realising the cost burden was about the same on all departments, as they had roughly the same share of old devices.

And here we are!

So, things have calmed down a bit. There’s currently a longer lead time than expected on the hardware order (I did chuckle a bit, I’ll admit).

There’s only so much shouting that can be done before everyone runs out of steam.

Many mumblings of “lessons learned” and “post mortem” - a bit early for that methinks!

30 Jan 2023 at 15:51 | Open on infosec.exchange
Security Writer :verified: :donor:

If you want a bit of the financial spice, they’re estimating for this time of year (low turnover) and the burden rate of those effected, plus the loss of revenue generated by them at about $9.2mil/ day. Not sure how accurate that is, and seems a touch high, but it’ll still sting either way.

My money is on Compliance and Legal being bullied into softening their stance and allowing access. But all those devices are pretty much BYOD now they’re purged from Intune, so it’s probably going to need RTB or some very clunky remote AAD joins.

I’d imagine their counter offer will be “so long as we can inform customers” and the response will be “no”.

If you want a bit of the financial spice, they’re estimating for this time of year (low turnover) and the burden rate of those effected, plus the loss of revenue generated by them at about $9.2mil/ day. Not sure how accurate that is, and seems a touch high, but it’ll still sting either way.

My money is on Compliance and Legal being bullied into softening their stance and allowing access. But all those devices are pretty much BYOD now they’re purged from Intune, so it’s probably going to need RTB...

30 Jan 2023 at 16:03 | Open on infosec.exchange
Security Writer :verified: :donor:

If you all want a laugh, this is the second time this has happened in 12 months at 2 different companies. Smaller last time, but still. Last time it was our ‘fault’ for implementing *their* spec, but the person that knew what that spec was left. Which left us as the only people knowing their spec.

So they wrote new procedures, but didn’t actually get anyone to implement them technically.

By comparison this is the Fires of Mt. Doom as far as ostrich-management goes.

“Best practice, please”

“No, not like that”

If you all want a laugh, this is the second time this has happened in 12 months at 2 different companies. Smaller last time, but still. Last time it was our ‘fault’ for implementing *their* spec, but the person that knew what that spec was left. Which left us as the only people knowing their spec.

So they wrote new procedures, but didn’t actually get anyone to implement them technically.

30 Jan 2023 at 16:51 | Open on infosec.exchange
Go Up