Domain popularity across countries

As the DNS operator of .nz, we manage 4 of the 7 .nz nameservers ourselves and we've been collecting their DNS traffic. The other 3 are hosted by overseas providers and we only have access to the DNS traffic data from 2 of them. In total, our accessible data covers 6/7 .nz nameservers and about 80% of the total traffic.

Through this DNS traffic, we hope to find out which .nz domains are more popular compared with others and the change of domain popularity across time. With the data from multiple locations around the world, we're able to analyze and compare .nz domain popularity in different countries, and how specific events affect that popularity, which will be explored in this post.

Algorithm

The original algorithm we invented for domain popularity ranking was introduced here. As it's still being tested and improved, we'll use a simplified version and some adjustment has been made for the calculation according to country. We use MaxMind GeoIP to map the traffic from an address to a country. By extracting the daily traffic from a specific country c, we can calculate a popularity score for a domain d as follows:

A domain with a high popularity score means it's popular in the DNS traffic. By ranking the popularity score in different countries, we'll see which .nz domains are popular in each country.

There are many factors impacting the traffic we observe for a domain. This simplified version of the algorithm doesn’t account for TTL values, resolver behaviours, and user population. The popularity score is then a proxy of popularity.

Next， I'm going to exemplify the analysis for web domains and email domains using data from Dec 2017 to Jan 2018.

Web domain popularity (Dec 2017 - Jan 2018)

We calculated the popularity score for each .nz domain in the total web traffic (A or AAAA queries for a .nz domain itself or with the hostname of 'www' ) and extracted the top domains as listed on the vertical axis of the heat map below. We want to compare the popularity ranking of these domains in the 10 countries that sent the most significant traffic to us during the period in our analysis. The number in a cell represents the ranking value for the domain of that row in the country of that column, and we use the shade of the purple to visualize the numerical magnitude.

We can see that many of these world popular .nz domains, which include cryptocurrency exchanging website, social media platform, online travel booking and hospitality service, are not ranked as high in NZ as in other countries. Then which domains are most popular in NZ? Let's look at the heat map below.

We see some kiwi favorite radio and news media websites, popular online local marketplace and ISP website on the top list of NZ. We can also find that these NZ popular domains are more popular in AU than in other foreign countries.

The difference of popular domains in NZ and other countries shows that .nz domains have different recognition locally and internationally.

By drawing the bump chart below, we can inspect the top 10 NZ web domains' popularity ranking across time.

neighbourly.co.nz is a New Zealand neighborhood website founded in 2014, and till 2017 it has 460k members, less than 9% of the country population. Despite having a smaller user population than trademe.co.nz, neighbourly.co.nz still achieved a higher popularity score in the DNS traffic. We found that neighbourly.co.nz has a much smaller TTL value which makes its DNS records in resolvers' cache expire faster, thus need to query us more frequently, which would explain the confusion. There're some interesting patterns in the chart. For example, the ranking changes show a weekly cyclicity and a holiday impact for some domains like newstalkzb.co.nz and thehits.co.nz.

Email domain popularity (Dec 2017 - Jan 2018)

The analysis of domain popularity in email traffic (MX queries for a .nz domain) is more elusive due to the prevalence of spam. We explored the most popular email domains queried from NZ and US. For each country, we use a box plot to show the most popular email domains they queried and the ranking distribution of each domain across the time in analysis.

Per NZ, half of the list is email service provided by some ISP, such as vodafone.co.nz, kinect.co.nz (Trustpower), xtra.co.nz (Spark), and inspire.net.nz. The rest are from Trademe and government. From the plot, we can see that ird.govt.nz (Inland Revenue Department) shows a big fluctuation due to it's used in communications only happened on a business day.

Popular email domains queried from US contain some .nz domains of Yahoo, Hotmail and Outlook email services. I'm not sure how many people use these email domains today, but I guess it's possible that they were targeted by spammers.

Impact of events

Internet traffic is affected by human activities. From the changes in domain popularity ranking, we expect to find some interesting stories about the human world. Here're some examples of domains' popularity in New Zealand impacted by big events.

Above is the trend of popularity ranking of radiosport.co.nz website across Dec 2017 and Jan 2018 in five countries. All five lines went up in the same period across late December and early January. During that time, the West Indies cricket team was on tour in NZ and they played a number of games against the Black Caps. The other impacting event could be a well-known sports journalist for Radio Sports resigned on Dec 19 and subsequently they changed their weekend programme line-up. So the jump could have been due to an increased advertising campaign as a result of that.

The New Zealand Qualifications Authority, nzqa.govt.nz, spiked on the day when New Zealand's secondary school examination result was published. People went to the website to check their performance thus made the web domain more popular than usual.

The email domain of Inland Revenue Department, ird.govt.nz, dipped at Christmas and New Year holiday, well matched their working time.

Conclusion

In this post, we used our DNS traffic to analyze .nz domain popularity across different countries based on a simple algorithm. We compared the most popular domains in New Zealand and some foreign countries, and demonstrated how specific events affect the domain popularity ranking. Domain popularity analysis provides us with insight into the use and trend of the .nz namespace. The algorithm still has space to improve to generate a more accurate result that can provide even more valuable insights. Finally, we want to thank Dave Baker for the information about real-life events to better explain the plots.