[Reading Science] Korea Hacker Develops AI to Prevent AI Attacks

Team Atlanta Led by Samsung Executives Advances to Finals of US Government AI Security Competition
DARPA, Creator of Internet and GPS, Hosts the Event... White House Also Anticipates
Aiming to Develop AI That Detects and Fixes Open Source Program Vulnerabilities Independently

Team Atlanta members are cheering after securing their spot in the AIxCC finals. Photo by Team Atlanta Blog

The word "Hacker" is based on the concept of a person. Until recently, both hackers who find and exploit vulnerabilities in computer networks or software and white-hat hackers who focus on identifying vulnerabilities were all humans. The advancement of artificial intelligence (AI) demands a shift in this perception. Now, as AI performance rapidly improves and the emergence of AI-driven hacking becomes a reality, related countermeasures are accelerating.

Recently, the semi-final of the AI hacking system competition called the "AI Cyber Challenge (AIxCC)," led by the U.S. Defense Advanced Research Projects Agency (DARPA), was held in Las Vegas. This competition took place one year after DARPA abruptly announced its sponsorship last year. The seven teams selected for the finals will compete next year. The scale of the competition is vast, with the semi-finals and finals spanning two years.

Professor Kim Tae-su at Georgia Tech and Executive Director at Samsung Research

Hacking researchers reportedly were surprised by the prize money of this competition. The total prize pool reached $18.5 million (approximately 24.7 billion KRW).

DARPA is a U.S. military-related technology research and development agency under the Department of Defense. The U.S. government and Department of Defense, sensitive to advanced technology, have fostered technology through DARPA. It is widely acknowledged that Silicon Valley, the center of IT, would have been difficult to develop into its current form without DARPA's support.

Representative technologies born with DARPA's support include the Internet and the Global Positioning System (GPS). Numerous research projects transitioned to the private sector, changing the way humanity lives.

Logo of the U.S. Defense Advanced Research Projects Agency (DARPA)

DARPA hosting a hacking competition using AI can be read as a warning that AI-related risks could threaten the safety of networks and computers. This competition can be seen as an attempt to detect and prepare countermeasures against the risks of AI-driven hacking in advance. Considering DARPA's mission to "identify anticipated strategic problems early, find solutions, and demonstrate them," this is a plausible scenario.

The significance of the competition can also be understood from its sponsors. The event was supported by leading AI companies such as Google, Microsoft, OpenAI, and Anthropic, who publicly showcased their AI technologies. The fact that leading AI companies are taking the lead indicates a high evaluation of the potential for both the development of AI-driven hacking and its defense.

◇ Create AI that finds and fixes vulnerabilities = The task of AIxCC is to design a new AI system that protects open-source software. In other words, it is the development of an artificial intelligence "Cyber Reasoning System (CRS)" that finds and fixes program vulnerabilities.

In this competition, vulnerabilities were intentionally embedded in open-source programs such as Jenkins, Linux kernel, Nginx, SQLite3, and Apache Tika, and the AI systems were tasked with finding and fixing them.

Team Atlanta, which advanced to the finals of the AIxCC competition. Team Atlanta is led by Professor Taesu Kim of Georgia Tech and Executive Director at Samsung Research. Photo by Team Atlanta

As a result, a total of 22 vulnerabilities were discovered, and 15 were fixed. An unintended real bug was also found in SQLite3. The team that discovered and reported this bug was Team Atlanta, which included Korean participants. The organizers were surprised by the results of this competition. Andrew Karney, AIxCC program manager, said, "When we started this competition in the DARPA way, we did not know if the hypothesis would be proven, but now we know that AI systems can not only identify vulnerabilities but also apply patches to protect computer code supporting critical infrastructure."

◇ A Korean-led team becomes a milestone in AI hacking defense = Team Atlanta is a coalition team from Georgia Tech, Samsung Research, POSTECH, and KAIST. It was mainly composed of members from Professor Taesoo Kim’s lab at Georgia Tech, who is currently serving as an executive director at Samsung Research. Professor Kim is an information security expert who was appointed the youngest executive at Samsung Research in 2021. Most of the team members are of Korean descent. Professor Kim revealed on the team blog that they had been preparing for the competition since October last year and that Samsung Research employees played a significant role in this competition.

Team Atlanta’s CRS, Atlantis, found five bugs and received praise for discovering a new vulnerability in the database program SQLite3 that the competition organizers had not intended.

List of Teams Advancing to the AIxCC Competition Finals

Team Atlanta aims to pioneer next-generation security research and develop Security Artificial General Intelligence (Security-AGI) by using Large Language Models (LLMs). They presented the goal of revolutionizing the security paradigm by leveraging AI and LLMs in this competition as well.

According to Team Atlanta, Atlantis is a next-generation automated bug-finding and fixing system with intelligence beyond machines. Based on LLMs, it detects and fixes bugs in real software and automatically patches bugs through human-like reasoning. It supports various programming languages such as C/C++ and Java, handles complex systems like the Linux kernel, and evolves autonomously over time. It performs reasoning independently without human intervention.

Team Atlanta will receive $2 million (approximately 2.7 billion KRW) in research funding based on these results. They will advance to the finals held at DEF CON in August 2025, aiming for the championship. The grand prize is $4 million. The White House reportedly plans to release the CRS developed by the winners for free to help defend against hacking attempts. If Team Atlanta wins, they could become a supplier of programs distributed by the U.S. government.

Professor Insoo Yoon of KAIST, a former white-hat hacker and participant in Team Atlanta, said, "Compared to past competitions, the level of problems in this competition has significantly increased. It seems DARPA wants to test AI that hacks like a human." He added, "In the finals, problems are expected to be presented in more diverse programming languages and operating systems, so we will do our best to combine AI and security to win."