Nightmare server crash has a silver lining

One Saturday morning in July 2022, the server at Dallas Clinic in Morrinsville unexpectedly crashed, including the backup system. The practice lost all its data from the previous two days. 

The response from the clinic team and their IT provider was phenomenal. Together they pulled out all the stops to make sure the impact on patients was minimal and information gaps were filled, while putting the entire IT system back together in time for Monday morning.   

Dallas Clinic practice owners and general practitioners, Dr Roger Smith and Dr Saif Sulaiman, have generously shared their experience and learnings from that weekend and the days and weeks after. The story of how the practice managed, responded and recovered is a great resource for any other practices who may (unfortunately) experience a similar situation.  

“It was the most stressful time either of us has ever had. It was a nightmare and I never want to do it again,” says Roger when asked about the server crash and the weeks that followed.   

He was the one to discover what had happened, alerted by not being able to remotely login to the system from home early on Saturday morning. “I thought that’s a bit odd, and came down here. That’s when I discovered the server had failed.”  

Thankfully, Roger says, they managed to get hold of their technology provider – a new team they had just switched to. They arrived fast to work across the weekend and get the clinic back up and running so it could open on Monday morning.   

“Having really good IT support helped,” says Roger. “They came in and said ‘we can sort this’ and off they went. Saif and I spent most of the weekend in the practice helping them and explaining things while they set up the whole system within two days.”   

They knew it was likely that information from the Friday wouldn’t be recovered. On the Monday once everything was restored, it was discovered that all the data from Thursday the was also lost. Two whole days of information from a busy practice, gone.  

Noone could understand or explain what had happened, except for the possibility that the server died on the Friday night, and the non cloud-based back up system also failed at the same time. Either way, the problem remained: how to make sure the impact on patients was minimal, and find all the important information that was lost so that nothing critical got missed.  

“The thing I thought was not to panic,” says Roger. “Straight away on the Saturday Saif and I had made a list of patients we could remember seeing the day before, because we knew even then that it would be likely we’d lost information from Friday.   

“From there, we just had to find ways to get the information back.”

Starting with the list Roger and Saif made on the Saturday, everyone in the practice pitched in to brainstorm ways of finding out who had been in on the Thursday and Friday, and what they could do to recover as much information as possible.   

Bookings in the system made prior to those two days were still visible, but there was nothing in the book for anyone who rang up on the Thursday or Friday. The team trawled through the appointment information they had and double checked banking and EFTPOS payment records to further identify people from both days. Any paper-based records held at the clinic from time around the lost data days were re-scanned into the system.  

The team also reached out to other services. Pathlab was asked to resend all the results for Wednesday, Thursday and Friday so Roger, Saif and the nurses could look at them and remember some of the consultations. Similarly, they reached out to ask Healthlink to send any information they had to the clinic and got ManageMyHealth re-send any messages.   

All patients got a text message advising them of the IT failure and loss of information, asking anyone who had had an appointment on that Thursday or Friday to get in touch with reception. As the lists grew, phone calls were being made to everyone identified as having had an appointment on those days. All up, around 80-90 people needed to be phoned to run through their consultation, make sure the right information was recorded and that any follow ups were done.   

“Most of that phone call workload fell to me and Saif,” said Roger. “It’s not something we could delegate to anyone.   

“Because we’re so busy during the day it needed to be in the evening, ringing at least 30-40 people each at 7pm or 7.30pm, over two or three weeks. Thankfully we didn’t get any angry responses. The patients felt incredibly sorry for us and were pretty grateful I think to have that phone call.” 

Roger says it was probably two to three weeks before they were confident they had tracked everyone down.  

“We were all relieved when it was all finished, and felt confident we got everyone covered. We annotated the notes of anyone affected by the server data loss issue and so far haven’t had any feedback that we’ve missed results, or any referrals that I’m aware of. While we agree there will be some information lost, I don’t think it’s critical.”  

In hindsight he says it could have been worse. “I know a practice a few years ago that lost two or three weeks of information. We’re a small practice, and we were onto it straight away. We did the best we could.”

From a technology perspective, Roger says the clinic has learned a lot from the experience, particularly when it comes to timely server upgrades and working with a good IT company.  

“We knew the current server at the time was five years old and we needed to upgrade the whole system. Plans were in place to make that happen, alongside moving technology services to our new provider (Huston Technology).   

“Unfortunately we delayed our changeover – mainly because we were stressed with the COVID-19 panedmic and workload. It was easier to say ‘We’re tired, we’ll talk about it next week.’   

“We’ve learned a valuable lesson: when the advice is to get a new server or upgrade the system, do it – don’t wait.” Thankfully, he says, when the worst happened the team at Houston were onto it.   

“They resurrected and rebuilt some of the old server to keep us going, then installed a whole new system a few weeks later. What we have now is newer, faster, much better than anything we’ve had before. We’re really pleased with it.”  

He says there’s a little bit of peace of mind knowing the new system has multiple back ups, one in the cloud and two other different systems, so the clinic is covered even if there’s a power cut.   

“In three years when this system gets to the end of its warranty we’ll be sitting down with the IT people and saying: what are we doing now?” said Roger. “If they say it’s okay to extend out another year, fine but when they say it needs replacing, we’ll get it done.”

• Have really good IT support, people you can talk to easily and work alongside, and that have expertise working with medical practices. 
• Don’t let server upgrades fall down the list of things to keep on top of, talk to your IT support before it reaches the end of its warranty and replace promptly.  
• Peace of mind is multiple backup systems. Use the Cloud as well as other systems so that you’re covered for back ups even if there’s a power cut. 
• Keeping any loose paperwork that comes in for a good period of time after it has been scanned is a useful practice, and provides a good back up if the technology fails.