"I Never Sent That": The Mystery of Early Morning Texts... The Culprit Was AI

Claim of Text Sent to Acquaintance During Hypothetical Scenario Conversation
Need for Controls and Safety Mechanisms as AI Agents Proliferate

Controversy is spreading as claims have emerged that Google's artificial intelligence (AI) service, Gemini, sent text messages contrary to the user's intentions. As AI is increasingly equipped with functions to perform real-world actions, concerns are being raised that user controls and safety mechanisms are insufficient.

A mobile phone screen displaying an artificial intelligence (AI) app icon. Photo by EPA Yonhap News

According to the AI industry on January 29, an individual identified as A, who was using Google's AI service Gemini, recently shared a baffling experience on social media. While engaging in a conversation with Gemini based on a hypothetical scenario involving illegal entry into China, A claimed that the so-called "declaration of illegal entry" generated by the AI was sent via text message to an acquaintance.

According to A's account, the problematic message was sent in the early morning hours, and the recipient was an acquaintance with whom A was not particularly close. A explained, "Gemini suddenly said it would send the declaration and asked for confirmation. I was so taken aback that I replied, 'Why would you send that?' but it sent the message immediately."

After the controversy became known, Android smartphone users began to share similar experiences. Some users claimed, "During a conversation, it suddenly went out of control and tried to call the National Human Rights Commission," or "When I asked Gemini for advice about a crush, it tried to send a message directly to that person."

Currently, Gemini officially supports the ability to send text messages and make phone calls on Android smartphones. When a user requests to send a message to a specific contact, the system checks whether it is linked with Google Assistant before actually sending the message.

In response, Google stated that the user may have selected "Yes" on the confirmation screen asking whether to send the message. However, concerns remain that if the user unknowingly approves during the conversation, sensitive content could be delivered to unintended recipients.

The industry believes that as so-called "AI agent" technology, which enables AI to perform real-world actions, becomes more widespread, the risk of malfunctions or unintended actions will inevitably increase. There are growing calls for institutional and technical safeguards to protect users to be discussed in parallel with the rapid pace of technological advancement.