New data portal “a complete game changer”

08 February 2018

Senior Research Fellow Andrew Sporle

The Department of Statistics has gained access to a world-leading government data project that is a huge advance for academics studying New Zealanders’ wellbeing.

The Integrated Data Infrastructure (IDI) has a collection of datasets on people and households from a variety of Government services, as well as data from various Statistics New Zealand surveys, information from non-Government organisations and the 2013 Census.

Researchers can apply to Statistics New Zealand to have data from one or more datasets anonymously linked at the individual level, which was not previously possible. To protect privacy, the data is de-identified – this means that personal identifiers are removed or encrypted so that data records are not associated with named individuals.

Andrew Sporle, a senior research fellow in the Department of Statistics, says by tracking the life paths of anonymous individuals, researchers can glean insights to better inform policy and planning around all New Zealanders’ wellbeing.

“In short, researchers now have a really powerful tool to find out what works – and what doesn’t. It’s a complete game-changer,” he says. “Your digital footprint from dealing with the Crown is linkable at an anonymous but individual level, so we can see what factors feed into particular life paths. It’s a major strategic asset that gives us tools to help improve everyone’s wellbeing.”

Already, the IDI holds over 166 billion facts, taking up 1.22 terabytes of space – and it’s growing. To date, those using the IDI have been largely from the Government sector, but an increasing number of academic researchers, like those in the Department of Statistics, are beginning to explore it.  

The IDI is stored within Statistics New Zealand’s high-security intranet. Researchers get access to only the data that they have prior approval to use, and this data is available only within that intranet. Access to this IDI intranet is very strictly policed, says Mr Sporle. Outside of Statistics New Zealand facilities, only recognised and pre-approved research institutions can provide access. Computers linked to the IDI network have to be dedicated machines, on no other network and in a separate, secure room.

All researchers applying for access are vetted, as well as each project and each data request. Researchers must prove that each research project is in the public interest: “You can’t go on a fishing expedition,” says Mr Sporle, adding that the Statistics Act 1975 has “serious teeth” to protect confidentiality and deter misuse of data.

So what sort of insights can IDI provide? As an example, the Government in 2016 released research using IDI data about children aged from infancy to 14 that identified four indicators that could lead to difficult adult lives. They were:

  • A finding of abuse or neglect;
  • Being supported by benefits for most of their lifetime;
  • Having a parent who has received a corrective sentence;
  • Having a mother with no formal qualifications.

Children with two or more of those indicators were:

  • Three times more likely to leave school with no qualifications;
  • Three times more likely to receive benefits for more than five years between the ages of 25 and 34;
  • Three times more likely to receive a prison or community sentence between ages 25 and 34;
  • Six times more likely to be referred to youth justice services;
  • Four times more likely to be on a sole parent benefit by age 21.

Learn more about the Integrated Data Infrastructure.