How Skilled Is AI at Gaming?... Krafton Develops World's First "Game Benchmark"

'Orak' Evaluates AI's Game Performance
Quantitative Analysis of Situational Awareness, Decision-Making, and Action Selection
"Competition Planned to Test Game AI Design Capabilities"

Krafton announced on the 18th that it has unveiled "Orak," a benchmark for evaluating artificial intelligence (AI) performance in games. This is the world's first game benchmark based on actual video games from various genres.

Orak is an evaluation system that quantitatively analyzes AI's situational awareness, decision-making, and behavior selection processes during gameplay. It is designed based on six popular game genres: action, adventure, role-playing games (RPG), simulation, strategy, and puzzle.

The core technology is the Model Context Protocol (MCP). MCP delivers text-based information so that language models can understand games and converts the model's responses into actions applicable within the game. For example, in an action game, the model recognizes the location of obstacles and then makes decisions such as moving or jumping.

Krafton stated that it developed its own benchmark as part of its efforts to create gaming experiences where players enjoy games alongside AI. By leveraging Orak, the company has secured AI evaluation capabilities across various game genres, which enables further advancement of AI agents. In addition, Krafton has collaborated with NVIDIA to develop a new type of character called the "Co-Playable Character" (CPC), which can interact with players. Unlike traditional Non-Player Characters (NPCs), CPCs can converse and cooperate with users, recognize situations like humans, and respond flexibly. Furthermore, ten Krafton research papers have been accepted at ICLR and ICML, the world's most prestigious AI conferences, this year, with three of them selected for the "ICLR Spotlight," representing the top 5% of submissions.

Lee Kangwook, Head of Deep Learning at Krafton, said, "Orak is a game-specialized large language model (LLM) benchmark that encapsulates Krafton's advanced research and expertise," adding, "Based on this, we are also planning a competition in the future to test LLM agent design capabilities."