Frequently when people ask “is anyone else seeing this” questions on the New Zealand Network Operators Group (NZNOG) list they get a large number of replies, but in a case over the last week, a company at the centre of the concern has demonstrated to the industry how to handle these sorts of technical queries.
When faced with a question asking if any other UFB customers on 20 megabit upload plans were seeing speeds of only 2 megabit, Brent Marquis, Layer 2 Network Specialist at Chorus took it upon himself to respond with an offer to take responsibility and personally help. Brent enlisted the support of a fellow Layer 2 Network Specialist, Adam Ng. Together, with some of the affected RSPs, debugging of the issue quickly uncovered the likely cause.
It’s not unusual to see large telecommunication companies hide behind lines such as: “Please log a call with our helpdesk” or "We have determined that this is a problem outside the company's network. Affected parties are encouraged to talk directly with their hardware vendors".
What you don’t expect was Brent’s public response of:
“I’m investigating at the moment – instead of me running back via account managers to track down technical peoples details, can RSPs experiencing this email me please?” He then followed up the next day with “Having been inundated with replies yesterday (which was great!) we’ve narrowed this issue down to a probable cause… Collaborative debug/information sharing with Chorus, Fastcom, Callplus, Inspire and DTS has shown that Chorus is, in fact, forwarding all traffic from our handover as expected, but traffic marked with DEI=1 is lost/discarded in RSP networks.”
Over the next 8 days, Brent has worked to keep the entire industry engaged in conversation about this issue and worked through a way Chorus can work with the RSPs to address an issue which on the face of it looks to be a vendor issue with hardware in RSP networks.
Well Done Brent and Adam, well done Chorus for empowering them to engage with the technical community in such an open way. Give them a day off after this is all over, and hire more like them. Other industry players should take a leaf out of Chorus' book.