DNSSEC chain validation issue for .nz domains
Currently, some .nz domains may face resolution problems. The issue could occur for domains at .nz, .ac.nz, .co.nz, .geek.nz, .gen.nz, .kiwi.nz, .maori.nz, .net.nz, .org.nz, .school.nz, .cri.nz, govt.nz, .health.nz, .iwi.nz, .mil.nz and .parliament.nz.
InternetNZ and the Domain Name Commission are aware of the situation, monitoring it and working on resolving it as soon as possible.
The .nz domain name system remains intact, the system is secure and functioning.
The fault occurred during a routine InternetNZ procedure. We sincerely apologise for the problems it caused New Zealanders who couldn’t access some web applications and sites today. We are working urgently to help and resolve this.
The latest technical information about the issue can be found on the InternetNZ System Status Page. This page will be updated as new information is available.
The problem relates to a Domain Name System Security Extensions (DNSSEC) key-signing-key rollover, this can cause validation issues for some users. A Key Signing Key (KSK) rollover, in the context of DNSSEC (Domain Name System Security Extensions), refers to the process of changing or replacing the key pair (private and public) used to secure the DNS infrastructure for a given domain. It's a necessary procedure to maintain the integrity and security of DNSSEC.
The issue can affect any provider or user operating a DNSSEC-validating recursive DNS server, including Internet service providers, companies, or individuals.
How to resolve the issue — for ISPs
Nameserver operators (eg ISPs) can resolve the issue by flushing the validating recursive server cache for the relevant zone to pick up fresh DNSSEC records and return to a functioning state. This can manually clear the issue and we have reached out to the industry to accelerate this. This can be done preemptively. If no action is done from the ISPs, the issue will also resolve itself some time over the next 48 hours.
How to resolve the issue — for individual users
Please contact your DNS provider, usually this is your Internet Service Provider (ISP) and ask them to follow the instructions on the Status Page to flush the validating recursive server cache.
How many websites are down right now?
There’s no way to know how many websites people cannot access as the issue impacts some end users' ability to access websites rather than the websites themselves. Someone may be able to access a particular website while someone else may not be able to access the same website.
What caused this issue?
The issue occurred during a standard annual procedure, run by InternetNZ, of rolling out new cryptographic keys used to secure the DNS (Domain Name System) infrastructure operated by InternetNZ. We have been running these updates for more than 10 years without incident. The difference this year was that we ran this process for the first time since the new registry system was implemented in late 2022.
InternetNZ has now identified that the new system has slightly different outputs compared to the old one which wasn’t detected during the testing and validation phase of the new platform and its integration with the DNS system.
The change we implemented can’t be rolled back. Therefore difficulty in accessing some websites can’t be resolved by InternetNZ as we don’t operate the servers where the problem manifests. We have notified the industry and we are providing guidance to the ISPs in flushing caches, which should resolve the issue for users.
The issue occurred during what is known as a key rollover. This is when our team starts an automated process that rolls cryptographic keys used to secure the DNS (Domain Name System) infrastructure operated by InternetNZ, that is .nz and the 15 second level domains, co.nz net.nz etc.
To simplify, think of these keys like the keys to a house. Every so often, for security reasons, you might change your house's locks and get new keys. Similarly, in DNSSEC, InternetNZ must periodically change keys to maintain the system's security.
While we've fixed the issue at InternetNZ's end, the roll-on effect means that the problems people are experiencing result from cached data further downstream, most likely at their Internet Service Provider.
We sincerely apologise
We sincerely apologise for the problems it caused New Zealanders who couldn’t access some web applications and sites today and to the industry partners working hard on resolving it.
We will thoroughly investigate what exactly has happened to prevent it from happening again. We will review our processes and systems to ensure a robust service to the Internet users of Aotearoa. We’ll be engaging with the industry and stakeholders to communicate any potentially impactful updates to the system in the future.
When did the issues start?
This issue started with ac.nz sites at approximately 1pm on 29 May. This was reported to us and investigated, but we were unable to detect the issue in any of our monitoring or diagnostic processes. Other .nz domains started to be affected from 10:45pm on 29 May.