Telstra system behind outage understaffed, underfunded
Duncan Murray |
Telstra failed to sufficiently fund the system behind a major nationwide network failure, despite its potential to catastrophically impact services.
As a result, two key technicians who implemented the change behind the failure were sent home on a mandatory rest break and were not called back to help rectify the outage until hours later.
An independent investigation into the telco’s network failure on July 8, which left thousands of Australians unable to make calls or access data, was released on Wednesday.
The outage sparked chaos with triple-zero calls and businesses affected, along with failures across train networks and electronic payment services.
The outage began at 2.50am – nearly an hour earlier than what Telstra had claimed – following a planned change to a timing chassis in Melbourne.
When the chassis came back online, its internal GPS card, which had reset during the change, began sending the incorrect date as 2006 into the network causing rolling failures.
The report found the change to the chassis wasn’t tested to confirm key functions were working.
By the time the impact became known, the two primary system engineers involved were on mandatory standdown, leaving support personnel dealing with the outage without any clear point of contact.
The engineers not being present in the early stages of the outage likely reflected deeper systemic issues with Telstra’s governance, according to the University of Canberra research director Faraz Hasan.
“If these people had been around, it probably would have saved a bit of time in identifying what happened and putting measures in place to prevent further problems,” Dr Hasan told AAP.
“If governance is not right, you can’t expect the engineers or technicians to work their magic and keep things running smoothly.”
A Telstra spokesperson said such rest breaks were standard practice for those working overnight.
“Our team members completed their assigned work and began a rest break, in line with our enterprise agreement and our health and safety commitments to our people,” the spokesperson said.
“As the outage became clearer in the morning, they were called back in.”
A lack of ownership and oversight of the functions that led to the outage were identified as key contributing factors by Technology Audit Partners in the report.
The investigation found a further eight triple-zero calls did not go through to emergency services during the outage.
The report clearly showed preventable failures and tasked the industry with learning from its mistakes and rebuilding trust, Federal Communications Minister Anika Wells said.
Telstra chief executive Vicki Brady told media on Wednesday the outage was not the result of issues with staffing numbers or network expenditure, but with where they had been allocated.
“It’s not about not enough resources. It’s not about not enough money,” she said.
“It’s about the prioritisation, which meant we didn’t allocate the right level of resourcing and expertise to this specific area.”
Ms Brady said additional resources and expertise had since been allocated to the area.
Since the outage, just over 30,000 customers have contacted the telco which has processed credits of almost $1 million.
Any customers who felt they were impacted and had not been in contact were encouraged to reach out to the company.
An ongoing Australian Communications and Media Authority investigation into the outage could result in fines of up to $30 million per breach for the telco and may apply to the more than 600 failed triple-zero calls.
AAP