A new era for Pinnacle data and information

We're making some important changes in the way we store and use data to increase security, decrease costs and administration, plus open up new possibilities for understanding the health status and needs of the people in our region. 

Health data is collected and used for a variety of reasons, from a practitioner tracking the individual health challenges and progress of a patient through to using aggregated, anonymised data in order to plan and provide relevant services.  

Pinnacle has been safeguarding the health records of nearly half a million patients across the Midlands region for over 30 years. But with demand for data driven decision making on the rise, and new tools and technologies available, it was time to re-evaluate and upgrade our current ways of working. 

As part of designing our new storage and use processes we've completed a privacy impact assessment (PIA) with the Office of the Privacy Commissioner. This process helps to identify and mitigate any potential risks arising from the new ways proposed to collect, use and handle patient's personal information, and gives us assurance we are meeting all our legal obligations. The feedback on our PIA we received was very favourable. In addition to the PIA everything complies with regulations, including the Health Information Privacy Code. 

Cloud vs cloud: updating our storage solution 

We're moving our data from a 'local' mini-cloud based in a Hamilton data centre to a 'virtual' cloud, powered by global experts, Amazon Web Services.

This new storage solution enhances the excellent protections we've always had. Our new data cloud comes with extra security measures in place, including tracking every single interaction with the data which allows for more robust audit processes.  

As well as being highly secure, the virtual AWS cloud is far more cost effective. The new technology frees us from costs such as licensing, hardware, operating systems, backup and support/maintenance contracts - our AWS solution will cost less than 5 per cent of the current data warehouse costs. 

Taking the plunge: introducing data lakes 

We're also moving from a traditional data warehouse to a more modern 'data lake' concept - Pinnacle is one of the first health organisations in New Zealand to make this move - and the potential is enormous. 

Like most traditional databases, our data warehouse has a fairly rigid structure which means there are lots of manual steps involved in moving data from one source to another - it's very rare that data moves automatically. Lots of steps are involved in transforming data into useable formats. This rigid structure also means it is difficult to adapt to a changing landscape - in fact most of our data team's time is involved in getting the data into the warehouse in a format that can then be used.

Given the way the health system constantly moves and changes, rigid structures are no longer really fit for purpose. More flexibility and speed are required to transform data into information that can be acted on. With moving to the cloud we are moving away from a traditional data warehouse structure into a much more flexible 'data lake'. 

Data warehouse

• A data warehouse needs data in a highly processed and structured format. 
• Often a lot of steps are involved in 'cleaning' data to make it fit the current structure in place in the data warehouse.  
• The end use of the data (it's purpose) really needs to be rigidly described and understood before the data can be processed into information that can be fed out as meaningful reports.  
• If you want to add in something new, or try use something existing for a new purpose, things can get complicated... requiring much thinking and re-engineering to match data feeds to the predetermined field formats the data based was built on.

Data lake

• A data lake is raw and unprocessed data - literally you take a data feed (be it from a practice management system, DHB or MOH) and "chuck it in the lake" - nothing has to be adapted to fit.  
• You don't need a predetermined purpose for the data. 
• All data is highly accessible and easily updatable. 
• With no rigid structure the data is incredibly malleable - which means it can be quickly queried and analysed for any purpose. 
• Is ideal for machine learning. 

Transforming data: information for action 

In addition to saving money and enhancing protection, the AWS cloud comes with new tools which allow us to do the things we want to do with greater ease.

We'll be enjoying these benefits alongside many other AWS customer's such as Air NZ, Qantas, Fairfax Media, Orion Health, Xero, Finance One, GE Healthcare, Novartis, US Centers for Disease Control and Prevention and the US Food and Drug Administration. 

AWS are committed to the health sector and have released specific health-related products. One of these - Comprehend Medical - can understand free text and offers the possibility to automatically code from a consultation record; other voice-activated services would initiate this without using a keyboard. 

Working with modern tools available from AWS will allow us to 'crawl' the data in the lake and create virtual catalogues, automatically and unbelievably quickly. Once catalogued, data can be queried and analysed. Other new tools can automate the creation of dashboards and reports, or even detect anomalies and trends.  

Essentially these changes will provide a more efficient and effective data service for Pinnacle. Our data team will shift gears from building, maintaining and running a data warehouse to analysing data, providing meaningful information for Pinnacle to gain a deeper understanding of the health needs of the region, allowing the organisation to respond accordingly through service design.  

If you have any questions about the changes we're making please get in touch with our delivery partner Ventures.