Naver Cloud announced on the 27th that HyperCLOVA X scored higher than generative AIs from OpenAI and Google in the Korean AI performance evaluation system 'KMMLU (Measuring Massive Multitask Language Understanding in Korean)'.
KMMLU is an AI performance evaluation metric project led by 'HAE-RAE', a leading domestic open-source language model research team. It consists of 35,030 questions across 45 fields including humanities, social sciences, and science & technology, designed to assess expert-level knowledge. Approximately 80% of the questions cover broad knowledge applicable worldwide, such as mathematical reasoning skills, while 20% focus on Korea-specific problem-solving abilities, such as Korean Peninsula geography and domestic law, thereby evenly measuring AI’s universal capabilities and local knowledge. It is evaluated as a comprehensive tool to judge AI usefulness for Korean users.
When North American tech companies like OpenAI and Google translate the 'MMLU' metric, which they primarily use to verify their AI performance, into Korean, there have been limitations in accurately gauging AI models’ Korean language abilities due to inaccurate translations of questions and cultural contexts unique to English-speaking countries embedded in the problems. KMMLU, composed of original Korean test questions, allows for a more precise evaluation of both domestic and international AIs’ Korean language understanding capabilities.
According to the KMMLU research paper, HyperCLOVA X scored higher than OpenAI’s 'GPT-3.5 Turbo' and Google’s 'Gemini Pro'. Its overall performance, combining General Knowledge and Korea-Specific Knowledge, is competitive with AI from global big tech companies. Notably, in terms of Korea-specific knowledge, it scored higher than OpenAI’s GPT-4. It is analyzed that HyperCLOVA X could be useful in industries where local information such as education and law is highly important.
Sung Nak-ho, Head of Hyperscale AI Technology at Naver Cloud, said, "HyperCLOVA X is a sovereign AI that combines universal global knowledge with Korea-specific problem-solving abilities, and it is being adopted across domestic industries with excellent performance and strong security solutions. As global demand for native language-centered AI is observed, we will accelerate our entry into the global market based on the competitiveness of sovereign AI confirmed in Korea."
© The Asia Business Daily(www.asiae.co.kr). All rights reserved.
