Zone and web scanning
InternetNZ regularly conducts zone scans and web scans. This page sets our why we do it, how we do it, when we do it and other details. If you have any questions then please contact us.
The scan periodically iterates through the .nz zones and for each domain within those zones conducts a data collection scan for the following purposes:
- Research and analysis into DNS, how it works and how it is used. e.g. What is the usage of less common DNS Resource Records such as SRV and how that changes over time.
- Gathering data that can help with the maintenance and development of the DNS infrastructure; e.g. How much of our traffic originates from different networks and does that match our server placement to provide the optimal response times.
- Providing useful information for our customers, including registrars and DNS providers; e.g. Identifying lame delegations and reporting these to the registrar.
- Providing public information on the use of the Internet in NZ. e.g. Reporting on the uptake of IPv6 as measured by the number of IPv6 enabled domain names.
This practice statement sets out how InternetNZ conducts this scanning.
The use of zone data by InternetNZ in this way is explicitly authorised in the .nz Zone Transfer Policy. Additionally, DNCL may within the scope of the same policy, authorise access to zone data to third parties who intend to use it for scans.
Types of scan
InternetNZ conducts four types of scans:
- Zone scanning. The collector looks up public DNS records within the delegated zones. To be clear, this means that InternetNZ data collectors send DNS queries to the authoritative DNS servers for each delegated zone.
- Web scanning. The collector connects to publicly accessible web servers and retrieves the home page only, unless the home page has very little text on it in which case another page may be tried.
- HTTP scanning. The collector runs a variety of SSL tests against publicly visible websites under the .nz namespace, checking for protocols supported, certificate information and other TLS options.
- DNS Extended usage, where the collector tests for a variety of newly defined DNS records for email security, DNSSEC adoption, and other security-focused records.
The four types of scan are conducted independently.
For zone scanning, aggregate query rates are controlled so as not to exceed 50 queries per second to any individual nameserver. The determination of what is a single nameserver is made by using the domain name of the nameservers specified in the .nz zones and not the IP addresses.
For host scanning, aggregate connection rates are controlled so as to not exceed 40 simultaneous connections to any individual host. The determination of what is a single host is made by using the IP address of the host.
The scan frequency is limited to a maximum of 3 scans per day. Ordinarily the number of scans is much lower than this but this figure is set sufficiently high to enable restarts of failed scans.
Notifications and audit trail
InternetNZ maintains on its website a schedule of planned scans, giving no less than two days notice of the scans. This list details:
- The type of scan being conducted (zone, host, HTTPS or DNS Extended);
- The approximate time and anticipated duration of the scan;
- The source addresses of the scan.
InternetNZ also maintains an audit trail of all historical and active scans with the same information as above.
IP addresses used
The source addresses used for the scans are published on the schedule.
Where possible, IP addresses used for scanning contain a correctly configured reverse entry that provides information about the role of this host is scanning, which is:
This domain has a web redirect to this page.
It is recognised that some infrastructure operators may object to this activity for a variety of reasons and do not wish their infrastructure to be included in any scan. To address this concern the following safeguards are provided:
- The scans respect robots.txt and equivalents.
- The source addresses of the scanning hosts are published to enable a network operator to block access to the scans within their own infrastructure and with minimal disruption to other activity. Where possible these IP addresses are not used for any other operational purpose so that a network operator blocking these IP addresses should not experience any other operational impact.
If an infrastructure operator believes that our scanning is causing problems for their infrastructure then they should contact us and we will take appropriate action. If an operator believes that any such scanning as authorised by DNCL causes problems for their infrastructure then they should contact DNCL as well.
Coordinated scanning and third parties
We recognise the issues that could arise from multiple uncoordinated scans and are committed to cooperating with DNCL to coordinate with anyone else authorised by them to conduct scans.
No data is collected just because it can. Data is only collected where it can be used to further the purposes above.
The following is a non-exhaustive list of data that is retrieved along with an example of use that illustrates the purpose behind the collection:
For zone scanning:
- Authoritative NS records These are then compared against the delegations in the .nz zones and issues such as lame delegations detected.
- Authoritative TTL. This supports analysis into how TTL affects traffic and how a .nz service failure is felt over time.
- Record type survey. To identify the presence and number of novel DNS records as a signal of new technology adoption. This includes, for example, DNSSEC records, SPF, DKIM, SRV, LOC, etc.
We also test nameservers to see if they support the following undesirable and desirable service/behaviour:
- Public zone transfer (if this is successful then we discard the data received)
- TCP support for regular DNS queries
For host scanning:
- SSL certificates. To identify the number of X.509 certs in .nz and various features, such as how many are self-signed and so would benefit from DANE.
- META statement keywords and outbound links. To identify semantic links between domain name labels.
- Trust seals and SSL certificates. To help identify the overall security posture of .nz.
- Text. To provide a better understanding of the use of .nz.
For HTTPS scanning:
- Details about SSL certificates including Issuing Certificate Authority, Public Key Strength, Signature algorithm, etc.
- Support for different SSL/TLS protocols, including deprecated SSL protocols and new TLS versions.
- Support for different protocols options, including deprecated and new ones such as HSTS
For DNS Extended scanning:
- Records related to email security such as SPF, DKIM and DMARC
- Records related to DNSSEC including DS, DNSKEY, CDS and CDNSKEY
- Records related to Certificate security such as CAA
- Records related to Email Transport Security such as SMTP TLS
- Records related to the adoption of DANE, such as TLSA
All data collected is kept as retrieved indefinitely.
It is extremely unlikely that any data collected through this scanning process contains personal information about an identifiable individual. While this means that no legal constraints exist, there are expectations of confidentiality and commercial integrity for which a voluntary commitment to privacy is appropriate. This means that:
- Only anonymised and aggregated data is available as open data, where ‘open’ is taken to mean publicly available, freely licensed and free of charge.
- Non-anonymised and non-aggregated data is only available under restrictions, generally limiting access to the registrants, registrars and DNS providers that the data applies to.
This data is published as described above on our Internet Data Portal.