"Reading AI's Thoughts": KAIST Visualizes the Internal Structure of AI Decision-Making

Deep learning-based image recognition technology, which independently trains artificial neural networks to extract general rules from example data during the learning process, is rapidly advancing. However, it remains difficult to clearly explain the criteria by which artificial intelligence (AI) internally recognizes and judges images. In particular, analyzing how large-scale models combine objects into concepts to reach conclusions remains a challenge that cannot be solved in the short term.

A domestic research team has presented a clue toward finding an answer to this issue.

(From left) Dahee Kwon, PhD candidate at KAIST; Sehyun Lee, PhD candidate at KAIST; (top) Jae-sik Choi, Professor. Courtesy of KAIST

KAIST announced on November 26 that the research team led by Professor Choi Jaesik at the Kim Jaechul Graduate School of AI has developed an explainable AI (XAI) technology that visualizes the process of concept formation within the model at the circuit level, making the basis for AI decisions understandable to humans.

Within deep learning models, there are basic computational units called "neurons," similar to those in the human brain. Neurons detect small features in images (such as ear shapes, specific colors, outlines, etc.), calculate their values (signals), and transmit them to the next stage.

In contrast, a circuit refers to a structure in which multiple neurons are interconnected to collectively recognize a single meaning (concept). For example, to recognize the concept of "cat ear," neurons that detect the outline of an ear, the triangular shape, and the fur color pattern must operate in sequence. These together form a functional unit (circuit).

Until recently, most explanation technologies have focused on the approach that "a specific neuron detects a specific concept," centering on individual neurons. However, actual deep learning models form concepts through circuit structures in which multiple neurons cooperate. Inspired by this, the research team proposed a technique that interprets the concept representation units of AI not as individual neurons, but as circuits.

The "Granular Concept Circuits" (GCC) technology developed by the research team is a new method that analyzes and visualizes the process by which image classification models form concepts internally at the circuit level.

GCC automatically tracks circuits by calculating neuron sensitivity and semantic flow scores.

Neuron sensitivity measures how strongly a particular neuron responds to certain features, while the semantic flow score indicates how strongly that feature is transmitted to the next concept. Using these indicators, the technology can visually demonstrate, step by step, how basic features such as color and texture are assembled into higher-level concepts.

Overview of the conceptual circuit proposed by the research team. Provided by KAIST

The research team conducted experiments temporarily deactivating (ablation) specific circuits, and found that the disappearance of the concept handled by the circuit led to actual changes in AI predictions. This directly proves that the deactivated circuit was indeed responsible for recognizing the concept.

This study is significant as the first research to reveal the actual structure of concept formation within complex deep learning models at the detailed circuit level. It is evaluated as a breakthrough that allows a structural look into "how AI thinks."

Through this, the research team demonstrated practical applicability across the entire field of XAI, including enhancing the transparency of AI decision-making, analyzing the causes of misclassification, detecting bias, debugging and improving model structures, and improving safety and accountability.

Professor Choi stated, "Our team has presented the first approach to precisely interpret the internal structure of models at the detailed circuit level, unlike previous methods that simplified and explained complex models. Through this, we have proven that it is possible to automatically track and visualize the concepts learned by AI."

Meanwhile, Dahee Kwon and Sehyun Lee, PhD candidates at the KAIST Kim Jaechul Graduate School of AI, participated as co-first authors of this study. The research results were recently presented at the International Conference on Computer Vision (ICCV).