[AI Hanip News] "Small but Strong" The Miniaturization War of Giants

MS·Apple and others unveil sLLM
Competition beyond technology to commercialization
Competing with cost-effective models

The 'small wars' of the giants have begun in earnest. Until now, big tech companies have focused on increasing parameters directly linked to AI performance. Now, the competition has shifted to 'small but strong' models. Instead of huge language models (LLMs) that require enormous costs for training and operation, they aim to grow the market with cost-effective small large language models (sLLMs).

Microsoft (MS) unveiled 'Pi-3 Mini' on the 23rd (local time). It is a small model with only 3.8 billion parameters. Compared to OpenAI's GPT-3.5 (175 billion parameters), which powers ChatGPT, it is just 1/50th the size.

Parameters act like synapses in the brain that learn and remember information, so the larger the number, the better the performance. Generally, models with over 100 billion parameters are classified as LLMs, and those below as sLLMs.

Although Pi-3 Mini is a small model, it possesses various capabilities such as language, reasoning, and coding. If you upload a short report, it can perform Q&A based on it. MS plans to introduce 'Pi-3 Small' with 7 billion parameters and 'Pi-3 Medium' with 14 billion parameters following Mini.

Following MS, Apple introduced the small model 'OpenELM.' It consists of eight models with parameters of 270 million, 450 million, 1.1 billion, 3 billion, and so on. Apple reportedly used a layer-wise scaling strategy to achieve high performance even with small models. This method efficiently allocates parameters across each layer of the model to improve accuracy.

Small models are not limited to these. Meta released the next-generation AI model 'LLaMA 3' on the 18th, including a small model with 8 billion parameters that can be used for chatbots and coding support. Google unveiled Gemma 2B and 7B with 2 billion and 7 billion parameters respectively. Anthropic, known as an 'OpenAI rival,' also announced 'Claude 3' along with the small model 'Claude 3 Haiku.'

Why have they shifted from competing on model size to efficiency? The biggest reason is cost. Increasing parameters improves versatility and performance but also raises costs. OpenAI CEO Sam Altman said, "The cost of running ChatGPT is enough to bring tears to your eyes." It costs a lot not only to train models but also to operate them. Due to their large size, optimization and management are burdensome. High costs also mean lower utilization.

On the other hand, small models save time and cost for training and have relatively low operating expenses. While not universally versatile, they can perform quite well when specialized for specific fields or companies. Their small size also makes integration with other applications easier. Simply put, they offer good cost-effectiveness. S?bastien Bubeck, MS Vice President of Generative AI Research, emphasized that Pi-3 Mini "costs dramatically less" and that "compared to other models with similar functions, the cost is about one-tenth."

Above all, small models are well-suited for on-device AI devices. On-device AI runs AI directly on the device without going through servers or the cloud. To run AI on smartphones with limited performance and space, smaller models are more appropriate. Apple's OpenELM is also designed for on-device AI and could potentially be applied to Apple laptops or smartphones.

To win beyond technological competition and move into commercialization, profitable models are essential. An industry insider predicted, "In the future, the game will be about who can create cost-effective, appropriately sized models and apply them to services. Optimized models tailored to various use cases will divide the market."