Email or username:

Password:

Forgot your password?
Stefano Marinelli

This morning, I took my wife to the hospital for routine blood tests that had been scheduled for some time. Everything was going smoothly: check-in, number, waiting room. Suddenly, everything came to a halt and shut down. I was connected to the hospital’s public Wi-Fi and noticed that my connection also went down.

Having managed a couple of similar facilities, I immediately understood what had happened. I saw the staff panicking and calling the technicians, but they quickly reorganized within 10 minutes. They managed to process everyone who already had a number and then proceeded with the others in the order of their arrival. Despite the ten-minute delay (even though people started complaining right away), they were extremely efficient.

I later confirmed that the entire booking, check-in, and queue system is “in the cloud.” The hospital experienced a connectivity interruption, and all related services stopped. The staff no longer had access to anything, so a technician sent the lists to a manager via another channel, and everything resumed manually.

For years, I’ve insisted that certain things MUST be local. The healthcare facilities I manage have all the necessary systems for the operation of the facility internally, including patient records. External services like websites, emails, etc., are secondary.

Everything essential must always be accessible locally and, in special cases, it should be possible to physically access the servers and connect directly to them, bypassing any network/switch failures.

There has been only one interruption in the past, due to human error. Today, we have redundant servers (not HA on virtualizers, but two machines running the same software with replicated databases - on separate power lines) so such an issue shouldn’t happen anymore.

Not everything can be anticipated, but history is a great teacher. The Internet connection will eventually be interrupted :-)

When it comes to the health and survival of people, there are no compromises.

#IT #Internet #Networking #Outage #Health #HA #Cloud #CloudComputing #OwnYourData

32 comments
TomAoki

@stefano
Exactly.
All critical data should be held locally as primary, and clouds should be for backups and shares. If making cloud as primary, multiple physical connections with different routes should be always kept, if the data there is "critical".

Kevin Russell

@TomAoki @stefano

There are rules that can be followed, but medical records for 51 million people do not need to be stored at every hospital.

Your point is a start.

TomAoki

@kevinrns @stefano
Of course, single hospital doesn't need to host and manage records of other hospitals. But keeping their own patients' records locally as primary AND backing up in real time to (shared governmental) cloud would be wanted.
With this, they can access to their records even when external connections are lost and safe keep their records for hazards.

Kevin Russell

@TomAoki @stefano

Yep. Good rules can be made, they have not been made.

Seeing military organisations running Windows 95 is terrifying.

Evannakita

@TomAoki @kevinrns @stefano A shared governmental cloud does worry me considering the recent efforts by some Republican governors to obtain the medical records of trans people, though.

Stefano Marinelli

@kevinrns @TomAoki of course, there's no need to keep the records for all the people. But at least the ones being treated by that specific ward

Kevin Russell

@stefano @TomAoki

My main point is the complex protections and needs have NOT been accounted for, the continuation of providing service despite computational sctewups has not been met, or studied.

Its a complete dogs breakfast.

Epistatacadam

@stefano To support you, when cloud services first arrived, we frontline workers received explicit instructions: no clinical data was to be put on the cloud.
The director of IT changed, the new one had no experience of running the system having gone on an IT course after attaining his management qualifications.
Within weeks a new directive came out, all vital data was to be stored on the cloud as it was more secure.
It was but at peak times it just wasn't available!=> Copies on USB drives!

@anniemo71@tilde.zone

@stefano
Exactly! Great post. I am a retired nurse and I remember well, if a patient's paper notes went missing, it felt like we were working down a coal mine without a light!

Then, I took my son who was 9 years old for his appointment. He is fine but was born with a heart condition. He hated the doctors looking at him, and talking about him but had been seen 3 monthly and then 6 monthly from birth. Well, on that day all his records were missing. They never ever turned up, which meant that many of his past tests had to be repeated. Expensive for our NHS and distressing for my son.
Anyhow, the moral of this story is, ever since then I asked for a copy of all my families investigations and results. Nowdays, the GP's and hospitals should be happy to do that if a patient wants to keep their own copy.

@stefano
Exactly! Great post. I am a retired nurse and I remember well, if a patient's paper notes went missing, it felt like we were working down a coal mine without a light!

Then, I took my son who was 9 years old for his appointment. He is fine but was born with a heart condition. He hated the doctors looking at him, and talking about him but had been seen 3 monthly and then 6 monthly from birth. Well, on that day all his records were missing. They never ever turned up, which meant that many of...

UkeleleEric

@stefano So many people are so reliant on things that they ultimately have no control over. I know how to do things the 'old-fashioned' way. I also eschew devices or systems that are needlessly complicated or reliant on technology. Many years ago, I was a programmer - I am no technophile, just realistic. Nothing wrong with following a Satnav, as long as there is a map in the car for when something goes wrong...

kikebenlloch

@UkeleleEric @stefano Exactly, previous technologies can be reliable backup options in adverse circumstances, that's the same reason why I find it absurd to completely depend on wireless solutions, for instance landlines can save your day if everything goes south.

Bob 🇺🇲♒🐧🪖

@stefano

Putting any kind of stuff on the "cloud" is just setting up fails. The "cloud" is NOT safe. Not only does governments have access, hackers do to.

MeidasRomi

@stefano thei should have redundancy for the internet access (from two different providers) like most good offices have now.

Stefano Marinelli

@MeidasRomi they have a direct public fiber connection to a regional backbone. But sometimes bad things happen.

Arqtec

@stefano

Just another reason to STOP ELON MUSK

Doerk

@stefano These are the blessings of the cloud. Everyone seems to believe that the cloud is the only salvation. But actually the cloud is only someone else’s server. Sure, you don’t have to buy new servers every few years, but you have to pay for the flexibility, even if you don’t use or need it. It’s not really cheaper and you give up control.

fedops 💙💛

@doerk but it fulfills your short-term incentive: reduce costs. Sure, it's more expensive to run stuff in "the cloud" when comparing apples to apples, but you can create shareholder value by firing all your qualified local IT staff.
@stefano

Doerk

@fedops @stefano Yes, they are telling you that you can reduce costs and need less staff. On first glance, this is true, but it leads straight to total dependence to the cloud provider. Vendor Lock-in included. And after five years you figure out that you didn’t really save any money.

fedops 💙💛

@doerk the concept in a nutshell. Which is why I insist on claiming it is a failure of IT management.

They get their bonuses by clear-cutting their IT workforce. Knowing full well that the consequences are to be borne by whoever comes after them when their 5 year stint expires and they have moved on to greener pastures.

If you want to make sure never to be stuck in a meeting again ask your IT manglement for their exit strategy from the cloud. There isn't one and they will hate you.
@stefano

@doerk the concept in a nutshell. Which is why I insist on claiming it is a failure of IT management.

They get their bonuses by clear-cutting their IT workforce. Knowing full well that the consequences are to be borne by whoever comes after them when their 5 year stint expires and they have moved on to greener pastures.

GhostOnTheHalfShell

@stefano

It’s almost like a decentralized federated system is resilient.

OrSomething ™️

They should used that in places, ‘cause. Maybe even for communications. :ablobgrin:

Alex White-Robinson

@stefano hospitals I've worked with usually have back-to-paper drills semi regularly so that a network or software failure won't dramatically impact patient care.

osfa_2030

@depereo @stefano The digital record at the public hospital I go to is rather bad and messy. No doctor knows well how to use it. Patients are supposed to have access to their own records but that is rather a joke. Always incomplete, recent info not showing up... It's the typical thing that in theory should be efficient, but in reality never works well enough, costly to maintain, to keep updated with external changes... It's the false utopia of digital administration.

David Palk #FBPE #RejoinEU

@stefano Absolutely. Not as life critical as your experience but I was the IT manager of a broking house in the City of London. Absolutely everything was backed up, both on site behind fire walls, literal & programmed. We also had off-site secure back ups. We were bombed out by the IRA twice in the eighties, up and running in two hours. Just goes to show & privately, I wouldn't put a damn thing up to the cloud. An 'accident' waiting to happen IMO

Mark vW

@stefano Storing HIPAA stuff on somebody else's computer...who has explicitly disclaimed any liability for the loss or misuse of that information.... What could possibly go wrong?

mkj

@stefano Regarding "The Internet connection will eventually be interrupted", I think the litmus test is: ANY ONE THING WILL fail at a random time.

Design your systems and procedures such that you can handle that. Preferably by not even noticing; failing that, without significant disruption by having a tested and practiced alternative.

Electricity. Phone lines. Internet uplink. Computer security software vendor pushes a bad update. Whatever.

If you can handle more than one, that's far better.

Kris Gainsforth

@stefano Local also means that you need to have more FTE IT staff in order to deploy, secure, update and monitor. I understand why SaaS vendors like Epic are being used more and more. Budgets vs. redundancy.

Do I like it? No.
Am I justifying it? No.

Just looking at it from a patient that works in security and sees budget problems all the time.

Debra Shannon For Harris

@stefano about 6 years ago a doctor asked me the best way to set up his new office and patients files. I recommended a server that was not connected to the internet to keep his patients records. The program on the web he wanted to use we could basically restyle to suit his needs. He found that idea unacceptable and went with a cloud based company. I often wonder how many times he goes down or needs to tell patients their files have been compromised.

kikebenlloch

@stefano I couldn't agree more. Choose whichever architecture you wish (IDK -even periodical local copies of cloud repositories if nothing else is available our supported by management) but ask yourself if critical services are really autonomous or else can be brought down by not-so-extreme circumstances and render basic vital information inaccessible. If I depend on a single wire (literally or figuratively speaking) I'm likely to run into trouble.

nigel
Yah my work moved their accounting/job/invoicing system to the cloud. It is annoyingly slow and I wish they just done hybrid but apparently they were never given that option.

But yeah, people aren't generally aware of how fragile infrastructure is.
Go Up