[Q&A] KT Outsourcing A-Class Facility Management "Manager Absent Due to Other Duties"

Ministry of Science and ICT Holds Emergency Briefing on KT Network Outage on 29th
Issues Raised Over Notification Only via Website
Measures to Address User Damage Planned

On the 29th at 3 PM, the Ministry of Science and ICT held an emergency briefing at the Government Seoul Office regarding the KT wired and wireless communication outage. They announced the investigation and analysis results of the cause of the KT network failure and measures to prevent recurrence. Photo by Jo Kyung-sik, 2nd Vice Minister of the Ministry of Science and ICT

[Asia Economy Reporter Cha Min-young] KT, which caused a nationwide wired and wireless communication outage on the 25th, entrusted the management of A-grade critical communication facilities to a subcontractor, while the actual manager was absent due to other tasks during the work process.

On the 29th at 3 p.m., the Ministry of Science and ICT held an emergency briefing at the Government Seoul Office to announce the investigation and analysis results of the KT network outage incident, along with measures to prevent recurrence.

Analysis of the accident logs revealed that during the replacement work of the enterprise network router at KT’s Busan exchange, the worker input an incorrect configuration command, which subsequently caused routing errors leading to a nationwide internet network outage. It was determined that this was not a Distributed Denial of Service (DDoS) attack.

Cho Kyung-sik, the 2nd Vice Minister of the Ministry of Science and ICT, stated, "This accident occurred due to KT’s poor management. The Ministry of Science and ICT recognizes this seriously and will establish a strengthened response system to prevent similar cases from recurring."

Below is a Q&A session with government officials.

Q. Please disclose the detailed timeline related to the accident.

▲(Hong Jin-bae, Director of Information Security Network Policy) The timeline is as follows: ▲11:16 a.m. traffic surge ▲11:20 a.m. KT recognized internet outage ▲Around 11:20 a.m. suspicion of DDoS attack ▲11:40 a.m. reported to Ministry of Science and ICT ▲11:44 a.m. KT corrected the notice to routing error, not DDoS attack ▲11:56 a.m. Ministry of Science and ICT issued level 2 alert ▲11:57 a.m. recovery started ▲12:45 p.m. recovery completed.

Q. Why did KT workers perform the work during daytime instead of at dawn?

▲(Hong Jin-bae) Although no formal investigation was conducted, we directly confirmed with the worker, the party involved, and the manager. It was understood that daytime work was preferred because no one actually likes night work.

Q. What were the qualifications of the worker involved?

▲(Hong Jin-bae) Additional verification is needed regarding the worker’s detailed personal information and qualification requirements.

Q. Why were KT managers absent?

▲(Choi Sung-joon, Head of Network Policy Division) The manager was absent due to other tasks.

Q. Could KT blame the subcontractor?

▲(Hong Jin-bae) We confirmed with both KT managers and the subcontractor, and the work was carried out with mutual agreement between both parties.

Q. KT initially announced it was a DDoS attack, which increased public anxiety. Was there no process within the Ministry of Science and ICT to identify the cause more quickly?

▲(Hong Jin-bae) We were also monitoring the traffic increase. After dispatching and conducting a primary investigation including packet analysis, we judged it was unlikely to be a DDoS attack.

Q. The configuration script had an error missing the 'exit' command. Who writes this script, and is the primary contractor responsible for review while the subcontractor writes it?

▲(Hong Jin-bae) Basically, the script writing is done jointly by subcontractors and KT employees. KT employees conducted first and second-level reviews but failed to detect the error.

Q. Is it possible for incorrect information from Busan to spread nationwide within 30 minutes? Why did the firewall not operate?

▲(Hong Jin-bae) Firewalls are designed to block external intrusions. They are not systems to prevent internal configuration errors. Since this was not an external intrusion, the firewall equipment did not activate.

Q. The nationwide network was paralyzed due to missing a single 'exit' command. Was the recovery process simply re-entering the correct command to restore the system at once?

▲(Na Sung-wook, Director of Future Network Center, Korea Intelligent Information Society Agency) Because the command was entered without the 'exit', incorrect routing information was propagated to all routers. During recovery, since the command was already entered, the 'exit' command became irrelevant. The first step was to isolate the problematic router. Next, with the network no longer critically affected, each regional router was accessed to verify and delete the problematic processes and then restore them. This procedure normalized the system.

Q. Three years ago, after the KT fire, the Ministry of Science and ICT asserted that roaming service measures would prevent further outages. This incident seems beyond common sense. Is proactive government response difficult?

▲(Hong Jin-bae) Daytime work is against protocol. The work plan specifies it should be done at dawn. There was a violation of protocol in this work. We agree that there is a need to enhance response measures. We will reflect this in safety reinforcement plans. The Ahyeon exchange incident three years ago was a fire, a physical event focusing on physical and localized response. This time, it is a system-level issue, a new type of outage we have not experienced before. Proper response is crucial.

Q. I heard the Busan facility is A-grade, so it is hard to understand outsourcing the backbone network to subcontractors. Will there be institutional improvements to ensure telecom companies directly manage critical facilities?

▲(Hong Jin-bae) Facility grades are A, B, C, and D. The router replaced was located at the C-grade Busan exchange, and the work was remotely conducted from the Busan Communication Center, an A-grade facility. Regardless of whether it is A or C grade, the current practice of remote work by subcontractor staff without a manager present, and connecting the network during daytime, is inappropriate.

Q. Do other telecom operators have safety devices or test beds for such errors, or is KT the only one?

▲(Hong Jin-bae) On the day of the accident, we immediately requested urgent inspections from major Internet Service Providers (ISPs). While strengthening network stability measures, we will investigate their work management guidelines, simulation methods, simulator operation, and structure to identify and improve shortcomings.

Q. Which major telecom companies are targeted in the network stability reinforcement plan?

▲(Hong Jin-bae) The network stability targets will mainly be ISPs.

Q. There are criticisms that the government’s communication process was insufficient.

▲(Hong Jin-bae) We disseminated information through the media and social network services (SNS) to ensure sufficient awareness.

Q. Could the alert level have been raised to 'serious' instead of 'caution'?

▲(Hong Jin-bae) After review, the 'caution' level was deemed appropriate. We have four alert levels in total. We issued a level 2 alert and were reviewing escalation when recovery began, so no further escalation was made.

Q. KT announced the incident on its website. Was it impossible to notify via text messages?

▲(Hong Jin-bae) At the time of the incident, basic notifications were made via press releases, and text message systems should have operated, but KT was delayed. (Choi Sung-joon, Ministry of Science and ICT Network Policy Division) Currently, notification is required but no specific method is mandated. KT only posted on their website, but going forward, we plan to improve so that notifications can be made via SNS, text messages, and various other means. (Lee So-ra, Korea Communications Commission User Protection Division) After the Ahyeon exchange fire three years ago, the need for user notifications was raised. The Telecommunications Business Act was amended, and the enforcement decree was revised accordingly. Under Article 37-11, Paragraph 3 of the current enforcement decree, in case of communication outages, methods such as email, mobile phone, text messages, and internet homepage are listed, but only one method is required. This time, KT notified via their website, but we will review more convenient notification methods for users during institutional improvement discussions.

Q. KT held an emergency board meeting. When will compensation measures be announced?

▲(Lee So-ra) The most important thing regarding compensation is for KT to accurately assess user damages. They are currently receiving complaints through a reception window and will soon conduct a more detailed survey through a separate channel. We expect KT to prepare appropriate compensation measures. The Korea Communications Commission will monitor whether these measures are properly implemented.

Q. Regarding user compensation

▲(Hong Jin-bae) If users are not properly notified or reports are not properly made, responsibility can be imposed. However, legally, there is no basis to impose sanctions solely for damages caused by such outages.

Q. What responsibilities will the government hold KT accountable for?

▲(Hong Jin-bae) Regarding KT’s responsibility, the Telecommunications Business Act provides a framework to hold them accountable for user compensation. If they fail to properly notify users, compensate, or report to us, responsibility can be imposed.

Q. According to the amended Telecommunications Business Act, in case of service interruption due to communication equipment failure, the facts, causes, response measures, and contact information for consultation must be notified. Was this properly done, and if not, what penalties apply?

▲(Lee So-ra) This facility qualifies as a critical communication facility under the enforcement decree, so users should have been promptly notified of the interruption. Currently, it is judged that all obligations specified in the regulations were fulfilled.

Q. The amendment requires notifying users within 30 days about compensation standards and claim procedures. Was this done?

▲(Lee So-ra) The 30-day notification about compensation standards is based on contracts between users and providers, so it is appropriate to judge based on terms of service. Nevertheless, KT has expressed willingness to provide compensation beyond this obligation. We expect KT to prepare appropriate compensation measures based on accurate damage assessments within 30 days.

Q. Direction for revising telecommunications terms and conditions

▲(Lee So-ra) Yesterday, members of the National Assembly’s Science, ICT, Broadcasting and Communications Committee pointed out the need to revise telecommunications terms. We will prepare appropriate improvement measures.