The Impact of One of New Zealand's Greatest Breaches

The true cost of a security breach is nebulous and difficult to define. There’s the monetary cost – 8,160 incidents cost over $4.5 million per quarter – and all the other costs, from a demoralised workforce to public scrutiny that extends well beyond the initial reputational damage.

The Waikato District Health Board knows the breadth of breach impact only too well after a vicious ransomware attack in 2021.

This attack was so significant to public discourse in New Zealand and abroad that it has its own Wikipedia entry and has since become the subject of study around the management of security and risk.

How long did it take to restore the systems following the attack, and how many people were involved?

I had some 80-odd people indirectly reporting to me. That included the service desk, all engineering staff, software development staff and application support staff. We had all our major systems back online within six months, but we were still doing small amounts of recovery work 18 months later.

I understand that the human impact of the breach was particularly significant. Can you describe why?

The cost to our staff was extreme. While some of it could probably be chalked up to “The Great Resignation,” many left directly because the experience was demoralising. We would have lost 30 per cent of our workforce in the months following the breach.

For example, the engineer on call at the time was very senior and was quite traumatised by the experience. He’d been with the company for 20-odd years and left the business shortly after the breach to join a company with a much lower public profile, which he thought meant a much lower chance of being targeted.

Externally, the impact was significant. Everybody thinks that when you use the term
"Waikato District Hospital”, you’re referencing the hospital in Hamilton, but that isn’t the case. There are five other hospitals all using the same systems. So it significantly disrupted the entire Waikato region's health services. For example, patient flows were affected because, from a patient management perspective, no clinical records were available, as were ancillary applications like radiology.

Can you describe the experience when it first became clear that something had happened?

At first around 180 IT staff turned up on a Tuesday morning at 8:00 for work and quickly realised that there was nothing for them to do. They could not access the critical tools needed to support the hospital they believed in.

So they’re standing around and coming to terms with the impact this would have on the business. It's a pretty hopeless and disempowering feeling because there was nothing anyone could do to help.

That was an incredibly high-impact and demoralising experience. One of the positive things that we figured out by day three in managing the response was to deploy staff to support and focus on the hospital's business continuity. We hatched a plan that enabled us to reuse and refocus our PC fleet so staff could attach them to Microsoft Teams, SharePoint, their email systems, which we had migrated to Office 365, and a few other things that were in there. This enabled them to keep electronic notes and feel like they were doing something positive and contributing to the organisation.

Did the organisation have an incident response plan to cover a breach?

We didn’t have one, and that was certainly a huge gap. But the thing is, there aren’t many businesses in New Zealand that would have a plan. It’s something everyone talks about but rarely follows through with.

It's also worth noting that with ransomware, even with a plan, we wouldn’t have been able to access it.

Following recovery from the breach what did Waikato District Health Board do to improve security?

We adopted a Zero Trust approach to security. Never waste a good crisis, right? As an organisation, we decided to come back stronger. We employed Microsoft to show us the best practices approach to architecture that we should bring our systems back with. We then followed it very closely.

Now whilst that architecture and the processes and the behaviours behind Zero Trust were accepted while we were in emergency mode, as we moved into business as usual, the staff wanted us to remove the controls. They were objecting to the restrictions and the processes with Zero Trust, arguing that it was becoming difficult to do their jobs because they no longer could simply log on as the domain administrator and that even after going through the process to get the login, there were limitations on what that login could do, and so on.  

This was fascinating because these were people who had been traumatised by the event and recognised the need for Zero Trust in response to it. However, they still didn't recognise its necessity once regaining business as usual.

This is important for others to understand because to successfully implement Zero Trust, it speaks to the need for a change management programme.

Would Zero Trust have prevented this breach had it been in place beforehand?

It would have been enough to prevent the breach in the way it occurred. The breach happened because passwords were broken, and some passwords' were old-fashioned domain administration passwords. This allowed the criminals to be very lateral and do whatever they wanted within the entire domain. A Zero Trust architecture would have prevented that from happening or made it harder and provided us more response time.

What should we take from your experience?

Firstly, you should always have a plan and practice it. And, as part of that plan, you need a team you can bring in that hasn’t been impacted by the breach in the first place. In other words, have you got a “bench” team? One of the things we considered was to contract an organisation such as Softsource vBridge to be the team on the “bench.” Not all disasters are cyber attacks – there are natural disasters and other risks. For business continuity, you need to consider what would happen if the local team needed to focus on their own lives first.

Secondly, you need to practice dealing with the impacts on your team on their mental and physical health and well-being. For one example, there was some effort to get a “blame game” going, in that there was a lot of speculation in the team about whose domain admins got compromised by the cybercriminals. Through our investigation, we determined what had happened. I could rattle off the five names, but I won’t ever do that, and I'm the only person who knows them apart from the investigator who found them.

Thirdly, communication is critical. In these situations, there is no such thing as under or overcommunication. People needed to know what was going on and what was planned, even if we didn’t know what the plan was. When that happened, I needed to be honest and at the daily, stand up and bluntly say “Look, I have no idea what I'm doing here, but I'm figuring it out. If any of you have some good ideas, let me know.”

_{Image from Waikato District Health Board notice of outage of systems from cyber attack.  RNZ / Andrew McRae}