[2024 Future Business Forum] Leol Pinto, NYU Professor, "Robots Rapidly Evolve with AI"

Special Lecture on 'Building the Foundation of General-Purpose Robots'

"The robot mastered a new movement just 20 minutes after being shown. By combining artificial intelligence (AI) with the robot, we were able to rapidly expand its tasks."

Professor Lele Pinto of the Computer Science Department at New York University said this on the 22nd at the Lotte Hotel in Sogong-dong, Jung-gu, Seoul, during the '2024 Asia Future Business Forum.' Professor Pinto gave a special lecture on the topic of 'Building the Foundation for General-Purpose Robots.'

Professor Lettle Pinto of New York University is giving a special lecture via video on the 22nd at the '2024 Asia Future Business Forum' hosted by Asia Economy at Lotte Hotel, Jung-gu, Seoul. Photo by Hyunmin Kim kimhyun81@

Professor Pinto is a young scholar focusing on machine learning for robots. He is developing general-purpose robots that can be used in everyday life by applying AI models. Last year, he was selected as one of MIT Technology Review's 'Innovators Under 35,' and recently gained attention by founding 'Fauna Robotics.'

Professor Pinto explained the logic model underlying behavior using a video of children tidying up toys. To clean up toys, one must first understand and distinguish what objects are scattered in the living room. AI works similarly. When asked to 'draw people relaxing comfortably at the beach,' the AI can produce realistic results because the logic model is operating. It analyzes and understands objects at the beach and generates the output based on that.

For a robot to clean up toys like children, it must undergo learning. This involves inputting large amounts of data into a model that is initially blank like a white sheet of paper. Using this model, more data is generated, and the process of correcting errors is repeated. Professor Pinto explained, "After training the robot system with a lot of data, we observe how it recognizes and acts on objects it has never seen before. Although such research has expanded greatly, it has not yet improved enough for robots to be used at home."

Professor Pinto cited data issues as the reason. The more data AI or robot models learn, the fewer errors occur, but the data itself is insufficient. No matter how much data is collected, it does not fully represent reality.

Professor Pinto’s lab found clues to solving this problem in animal behavior. He gave the example of teaching a cat to open a door by pressing a handle. The cat does not understand the concept of a 'handle.' However, by showing the cat a person pressing it and rewarding imitation, the behavior can be taught through interaction. A person creates data for a specific action, and the cat learns it. Professor Pinto said, "Applying this to robotics means adding new knowledge of opening doors to existing algorithms. We are further researching how to enable robots to perform actions in environments different from those they learned in."

Professor Lettle Pinto of New York University is giving a special lecture via video on the 22nd at the '2024 Asia Future Business Forum' hosted by Asia Economy at Lotte Hotel in Jung-gu, Seoul. Photo by Hyunmin Kim kimhyun81@

Professor Pinto emphasized that AI models are crucial for the advancement of robot technology like this. To operate in unfamiliar environments like humans, AI must support the ability to recognize and judge objects in new surroundings.

Of course, large language models (LLMs) are not万能 problem solvers. For example, if a robot is commanded to 'pick up an orange drink bottle' and fails, it might succeed if told to 'pick up a golden metal drink bottle.' According to Professor Pinto, language-based models fundamentally depend on what language is used and how it is conveyed, which greatly affects the robot’s capabilities.

Professor Pinto is improving this issue through 'OK Robot.' Instead of relying solely on language models, the robot learns from videos. A smartphone is attached to a long gripper, and the gripper’s movements are recorded by the smartphone camera and shown to the robot. It takes only 20 minutes for the robot to watch a video of the gripper opening a drawer and learn the same movement. When this method was tested in 10 households in New York, commanding the robot to perform 110 tasks resulted in an 81% success rate. Professor Pinto emphasized, "The model was able to learn quickly because it had prior knowledge of previously learned skills. It can rapidly adapt to new environments and new tasks."

Professor Pinto expanded these experiments to robot fingers and robot arms. He is advancing robotics technology based on AI models. He added, "There are still many challenges to overcome in robotics technology. We must develop the technology considering reliability, safety, and privacy."