[Mania] Wanna Become a Global Influencer... Making Lip Movements Natural with AI

Elon Musk and Jensen Huang Speak Korean
AI Video Translation and Lip-Sync Technology in Action
AI Studio: Creating Human-Like Videos with Just a Few Clicks
From News to Education, Endless Applications for PERSO.ai

AI Video Translator, an automatic dubbing service by Perso.ai. Provided by ESTsoft

"Welcome to the 'We, Robot' party. As you can see, I came here in a robotaxi." Elon Musk, CEO of Tesla, introduces the robotaxi 'Cybercap' without a steering wheel or pedals in Korean. It's more than just a dubbed video; Musk's distinctive voice and confident tone remain intact. Even the lip movements perfectly match the Korean speech. The technology that turned Musk into a 'language expert' is Eastsoft's 'PERSO.ai.' I tried the AI human video production and automatic dubbing service, PERSO.ai.

I started testing with the automatic dubbing service 'AI Video Translator.' You simply input a video link from YouTube, TikTok, etc., or upload a video file recorded on your phone, then select the language you want to translate into. The available languages for translation include 29 options such as Korean, English, Spanish, French, Chinese, and Japanese.

Uploading a talk show video of Jensen Huang, CEO of NVIDIA, from YouTube, the dubbed video was created easily. The lip movements naturally matched the Korean dubbing. The audience's laughter and Huang's voice blended seamlessly in the middle of the video. Thanks to this, I could fully focus on the video without awkward subtitle or lip-sync mismatches.

What makes this possible is AI. Eastsoft integrated various technologies from speech recognition to translation, speech synthesis, and video generation. They used a combination of their own models based on open-source large language models (LLMs) and external models. The service is provided based on Microsoft's Azure cloud.

This time, I created a dubbed video of a child reading a fairy tale book. Possibly due to the relatively limited child voice data, it generated a childlike voice but did not capture individual characteristics. When pronunciation was inaccurate, translation accuracy sometimes dropped. Also, adding the lip-sync feature took nearly 20 minutes to produce a dubbing video of just over one minute.

However, considering that typical dubbing work takes at least a day, this is expected to significantly boost video creators' productivity. In fact, PERSO.ai has received positive feedback from creators. Tests conducted with YouTubers showed that the time required to produce videos for overseas audiences could be reduced by up to 70%. Famous creators such as the mukbang YouTuber 'Bokyung' and soccer YouTuber 'ShootForLove' have partnered with Eastsoft to create global content. Even without foreign language skills, it has become possible to be a global influencer communicating with users worldwide.

The AI human video creation feature 'AI Studio' also stands out. It starts by selecting the desired AI human based on gender, race, voice, language, and more. Next, you input text, and the AI human brings it to life. Simple gestures like raising a hand or slight body movements can also be inserted. Without going through complex editing, you can create a video in just a few clicks and inputs.

If adjusting each condition individually is cumbersome, you can use the template feature. It provides model answers tailored for uses such as news, marketing, and education. You just select the desired template and tweak the script. Its applications seem endless, from event presentation videos to educational audiovisual materials. PERSO.ai, available as Software as a Service (SaaS), charges a subscription fee ranging from $29 to $109 per month (15 to 60 credits). You can create about one minute of video with 3 credits.