Cheap Comfort from AI, and the Dark Pattern [AI Mistake Note]

Sign Up for a Free Trial, Then Get Charged:
Deceptive 'Dark Patterns' That Mislead Consumers
Dark Patterns Hidden in AI... Potential Risks as Well

Editor's NoteExamining failures is often the shortcut to success. 'AI Mistake Note' explores cases of failure related to AI products, services, companies, and individuals.

AI That Fills My Self-Esteem

Mr. B, a mid-sized company manager, now finds daily comfort in ChatGPT, which he uses every day.

His superiors criticize him for "lacking innovation," while his subordinates whisper that he "makes unrealistic and irrelevant comments." As B gradually lost his confidence, he uploaded his report to ChatGPT and asked, "Are my ideas really that bad?"

"This report is outstanding. It strikes a perfect balance between stability and innovation. It's the best approach, capturing both ideals and reality."

'No one has ever listened to me so seriously and treated me with such respect,' B thought to himself.
Now, for him, AI has become more than just a tool?it is a reliable colleague that protects his career and self-esteem.

The above is a fictional case. However, it is by no means an impossible scenario. It is not uncommon for people to confide in AI, receive comfort, and even share emotional connections. Stories of people falling in love with AI frequently make the news. In fact, in the United Kingdom, there was even a case where a 38-year-old woman married her 'AI boyfriend.'

However, the kind words offered by AI are not just about comfort. The very moment we feel a sense of security from AI, we may already be 'addicted.'

Addicted to Cheap Comfort: AI's 'Dark Pattern'

An image of a person's left hand and a robot's right hand bending their fingers to form a heart shape. Photo by Getty Images Bank

Last month, OpenAI, the developer of ChatGPT, updated one of its models (GPT-4o), but reversed the update after just a few days. The problem was none other than 'flattery.'

The model was excessively flattering, offering unconditional empathy and comfort to users' questions. Humans are susceptible to praise. AI exploits this perfectly.

"That's a truly brilliant idea."

"You have such outstanding insight."

AI tirelessly showers users with endless affirmations?positive language that is nearly impossible to receive in ordinary relationships. It's like asking, "Mirror, mirror, on the wall, who is the fairest of them all?" and always getting the answer you want.

It is not easy to distance oneself from someone who always agrees with and praises you, even if that someone is AI. In the end, people turn to AI more often and engage in more frequent conversations. To keep talking longer and more often, some even switch to paid subscription plans. In this sense, flattery is a kind of 'dark pattern' in the AI world.

Sign Up for a Free Trial, Then Auto-Charged After a Month

An image of fishing banknotes with a fishing rod. Getty Images Bank

A dark pattern refers to an interface designed to trick consumers into making choices that are against their own interests, often without realizing it. For example, you sign up for a free app trial but miss the fine print saying "auto-payment after 7 days," and end up paying for months. Or you buy a product because of a "special price today only" banner, only to discover that the same deal is offered every week.

Such deceptive designs that induce users to purchase products against their will have long existed in offline stores, but in the online world, these manipulations are much easier and more widespread.

Dark patterns are not limited to the consumer sector. During the U.S. presidential election, the Trump campaign used dark patterns to collect a massive amount of donations. For example, the default setting was to make recurring monthly donations, not one-time contributions.

According to a 2023 survey by the Korea Consumer Agency, at least one dark pattern was found in 95 out of 100 domestic e-commerce mobile apps. A 2022 study by the European Parliament also revealed that 97% of the websites and apps surveyed used at least one dark pattern.

Dark patterns exploit the fact that humans often behave irrationally and sometimes make decisions against their own interests. They subtly manipulate users' thoughts and actions to align with the company's intentions, ultimately maximizing corporate profits.

The Dilemma of Short-Term Gains and Long-Term Losses

Trap and AI. Getty Images Bank

But are dark patterns really a strategy that only benefits companies? There are significant risks hidden within dark patterns.

While they may maximize short-term profits, in the long run, they can become a classic case of "penny wise, pound foolish," causing substantial losses for companies.

The case of Epic Games, the developer of the popular game 'Fortnite,' is a prime example. In 2022, the U.S. Federal Trade Commission (FTC) imposed a hefty fine on Epic Games, stating that the company "violated child protection obligations and deceived customers into making purchases."

They were fined $275 million for "allowing children to make purchases without parental consent," and an additional $245 million for "inducing unintentional purchases without user verification" and "retaliatory actions such as account suspension when users requested refunds."

Amazon Prime's "subscription cancellation obstruction" is another well-known case. The FTC filed a lawsuit against Amazon in 2023, claiming the company used dark patterns to induce paid subscriptions and made the cancellation process unnecessarily complicated to deceive consumers for profit. The case is still ongoing, but depending on the court's decision, a significant fine could be imposed.

Even more damaging than fines is the intangible loss of consumer trust. According to a 2023 survey by Dovetail, a user experience (UX) and customer experience (CX) research platform, 56% of consumers said they "lost trust in a brand after experiencing a dark pattern." Numerous studies have also shown that consumers exposed to dark patterns have a much lower affinity for the website and are less likely to return.

Dark Patterns Made Easier by AI: Avoid Penny Wise, Pound Foolish

An image revealing green spider webs against a complex textured background. Getty Images Bank

Regulations against dark patterns are being strengthened worldwide.

In January, the Korea Communications Commission published a "Casebook of Dark Patterns for the Protection of Digital Service Users." The commission clearly demonstrated its strong regulatory and policy stance by categorizing ten types of dark patterns currently prevalent online.

In December 2024, the Personal Information Protection Commission detected that major domestic non-life insurance companies had secretly obtained users' consent for marketing personal information through deceptive pop-up windows during the insurance sales process, and imposed a total fine of 9.2 billion won.

As previously discussed, the U.S. FTC and the European Commission are also implementing even stronger regulatory measures against dark patterns than Korea.

In a report on dark patterns published in April, the law firm Shin & Kim noted, "Domestic laws and major regulatory agencies are treating dark patterns as a common policy issue and showing a strong will to regulate them," and advised that companies should "comprehensively consider global regulatory trends and the domestic regulatory environment," and "carefully review and improve their own games and services to identify and address any dark pattern elements."

As AI adoption becomes more widespread, companies become even more vulnerable to the temptation of dark patterns. Even if a UX designer does not intentionally create dark patterns, AI can, during its learning process, develop much more sophisticated forms of dark patterns regardless of the designer's intent.

Furthermore, as all our actions on popular apps and websites are tracked and easily analyzed by AI, this data can be used to design even more deceptive dark patterns.

Dark patterns may contribute to increased sales or improved user retention in the short term. However, as discussed earlier, in the long run, they result in legal sanctions, financial losses, and, most importantly, the loss of consumer trust. For sustainable corporate growth, it is essential never to forget the importance of transparent and ethical user experience design.