[Tech Talk] Upgraded ChatGPT, Has It Actually Become Dumber?

"Response Quality Has Declined," Industry Voices Complaints
Math and Coding Answer Quality Dropped Compared to 3 Months Ago
Possibility of AI Model Architecture Change Raised

OpenAI's 'ChatGPT,' which sparked the generative artificial intelligence (AI) craze earlier this year, is now under suspicion for producing inaccurate answers more frequently after a major upgrade. Although this suspicion has not yet been clearly verified, the academic community is already unsettled, suggesting that the impact could be significant.

Has ChatGPT Become Less Intelligent After the Upgrade?

ChatGPT Photo by Yonhap News

ChatGPT was developed based on OpenAI's large language generation model (LLM) series 'GPT.' Currently, the free version of ChatGPT available on the OpenAI website uses 'GPT-3.5,' while the premium version, accessible through a paid subscription, employs the much more advanced 'GPT-4.0.'

However, since the introduction of GPT-4.0, claims that ChatGPT has become 'less intelligent' have been continuously raised. In May, Peter Yang, a developer of the US metaverse platform 'Roblox,' pointed out on Twitter that "after the upgrade, ChatGPT's response speed became faster, but the quality of answers declined."

Previously, OpenAI directly refuted these claims, stating that "ChatGPT has actually become smarter." They explained that "due to the explosive increase in users, shortcomings that were previously unnoticed are now being found more frequently."

"Decline in Functions Such as Math Problem Solving and Code Generation" Paper Published

Peter Yang, Head of Product at the famous American platform company 'Roblox,' introduced a paper on the performance degradation of ChatGPT.
[Image source=Twitter]

However, the controversy is ongoing. Recently, computer science researchers at Stanford University in the US analyzed ChatGPT's response data and published a research paper estimating how much its actual performance has declined.

In this paper, the researchers concluded that "in four areas?math problem solving, code generation, visual reasoning, and others?overall functionality has decreased compared to the March version." However, this paper has not yet undergone peer review, so it should be taken with some caution.

Assessing AI 'intelligence' is actually very difficult because there is currently no metric to accurately gauge the real performance of AI models. While the AI industry is increasingly attempting to develop 'AI performance benchmark tests' composed of data from various questions and tasks, no standard test has yet emerged.

Nevertheless, it seems difficult to simply dismiss claims that "ChatGPT's answer quality has declined compared to the past" as false. Many users, including developers, have voiced dissatisfaction, and ChatGPT's usage rate has been on a downward trend, decreasing by 9.7% month-over-month since last month.

OpenAI May Have Changed AI Architecture

There is speculation that OpenAI may have radically changed the structure of the AI model that constitutes ChatGPT.

If ChatGPT has indeed become less intelligent than it was three months ago, why might that be? The reason remains a mystery. Although machine learning technology is rapidly advancing, academia still does not understand 'how' computer intelligence exercises insight. The deep learning technology underlying ChatGPT remains largely a black box, making it difficult to pinpoint exactly which factors have caused the performance decline.

Some experts speculate that OpenAI may have radically changed the model that constitutes ChatGPT. Up to GPT-3.5, ChatGPT was based on a single massive language model.

However, starting with GPT-4.0, it is possible that multiple smaller language models, each trained intensively on specific expert knowledge, were combined to operate as a single AI. This approach is called 'Mixture of Experts (MOE).'

If ChatGPT has shifted from a large model to MOE, this could explain why response speed has increased compared to the past. However, the smaller size of the model answering each question might have led to a decline in actual reasoning ability.

Could the New Model Have Caused the Quality Decline?

OpenAI treats most information related to the models of its AI products as confidential.
[Photo by Yonhap News]

However, these explanations remain speculative within parts of academia. This confusion arises because OpenAI has not disclosed almost any details about the actual AI model behind ChatGPT.

Oren Etzioni, CEO of the US nonprofit AI research institute Allen Institute for AI, told the US media outlet Insider that the possibility of ChatGPT switching to an MOE model is a "well-informed guess," but he added, "I cannot be certain."

Etzioni emphasized that the MOE approach "increases response speed and allows AI to operate more cheaply," but does not necessarily degrade performance. However, he added that OpenAI might have sacrificed some quality in the process of transitioning ChatGPT to MOE.

The controversy over ChatGPT's performance decline also introduces a new issue in the AI business. Currently, OpenAI charges subscription fees to companies or individuals for ChatGPT premium services and also licenses API (application programming interface) usage for a fee.

However, if AI performance can significantly deteriorate depending on the operating capabilities of the company, it could be a major disadvantage for consumers. Transparent disclosure of information to guarantee AI product performance may become necessary.