Why Nvidia Chose This Startup: The Reason Behind the $20 Billion Investment [Tech Talk]

Groq Uses SRAM Instead of HBM
Faster Data Transmission, but Scalability Limitations
Big Tech Companies Investing Actively in Affordable Chips

Nvidia has invested 20 billion dollars (approximately 29 trillion won) and entered into a technology licensing agreement with the artificial intelligence (AI) semiconductor design startup Groq. Groq's value lies in its unique semiconductor design technology. Unlike existing AI chips such as Graphics Processing Units (GPUs) and Tensor Processing Units (TPUs), Groq has significantly reduced its reliance on High Bandwidth Memory (HBM).

The Core Technology of Groq: LPU and Its Differences from GPUs

Grok's Language Processing Unit (LPU), differentiated from existing artificial intelligence (AI) chips. Grok

On December 24 (local time), Groq announced via its blog that it had signed a non-exclusive licensing agreement with Nvidia. As part of the agreement, key personnel such as Groq founder Jonathan Ross and President Sunny Madra have joined Nvidia. Nvidia will now be able to use Groq's Language Processing Unit (LPU) technology.

Groq is a US-based AI chip design startup that has developed a unique chip called the LPU. What differentiates the LPU from existing AI chips like GPUs and TPUs is its approach to memory. Instead of connecting external memory using HBM like other chips, the LPU utilizes memory embedded inside the chip itself.

Embedding Memory Inside the Chip Instead of Externally

AI accelerators require memory to store vast amounts of AI data. Nvidia and Google have addressed this need by connecting HBM externally to their chips. The surge in demand for HBM used in GPUs and TPUs has led to a boom for domestic memory companies such as Samsung Electronics and SK Hynix.

The main difference between a typical Graphics Processing Unit (GPU) and the Grok LPU is that the Grok LPU connects Static RAM (SRAM) internally, enabling faster data transmission speeds. Inside the Grok LPU, the SRAM is neatly arranged like blocks. Grok

However, some companies have adopted a different memory strategy from Nvidia and Google. Mainly, AI startups including Groq are researching this approach, using Static RAM (SRAM) embedded inside the chip rather than externally. The memory is connected right next to the chip's processing units, or 'cores,' enabling faster data transmission speeds.

The Groq LPU can achieve sufficient data capacity and transmission speed without HBM. According to Groq's LPU technical manual, "Groq's memory bandwidth (transmission speed) using SRAM is 80 terabytes per second," and "the HBM of a typical GPU provides only about 8 terabytes per second of bandwidth, so the LPU guarantees up to 10 times the speed advantage and does not require separate memory chips."

Limitations: Difficult to Handle and Limited Scalability

However, SRAM also has its drawbacks. The biggest issue is that SRAM-based chips are not yet familiar to computer engineers. In contrast, Nvidia has maintained an overwhelming lead in the early AI chip race by offering CUDA, software optimized for AI. Today, Nvidia GPUs account for about 90% of the AI accelerator market.

Another issue lies in semiconductor manufacturing. The latest computer chips are produced using advanced 2 to 3 nanometer (nm) process nodes. However, SRAM's stability is compromised at these advanced nodes, making it difficult to densely arrange SRAM inside the chip. As the process becomes more refined, SRAM density must increase to fully replace external memory, but progress has suddenly stalled.

Thanks to the latest EUV lithography machines, the current production nodes of computer chips have been reduced to the 2 to 3 nanometer (nm) range, but it is difficult to arrange SRAM in this process. ASML

In this regard, Greet Hellings, Director at Belgian semiconductor research facility IMEC, said in an interview with the semiconductor media outlet SemiEngineering last year, "Currently, there is little room to further advance SRAM," and added, "If you shorten the length of the SRAM pins, you must increase their height. Even with advanced process nodes, the density of SRAM cannot be increased sufficiently."

"Potential to Become a Much More Efficient AI Chip Than GPUs"

Despite these limitations, investment in SRAM has been increasing recently. Groq is not the only player. UK-based Graphcore, a pioneer in SRAM, was acquired by SoftBank led by Chairman Masayoshi Son in 2024 and is now designing new AI semiconductors. On December 11, it was reported that Intel was in the final stages of acquiring SambaNova, another SRAM-focused startup.

If SRAM overcomes these obstacles, the LPU could emerge as an AI semiconductor optimized for inference in large language models (LLMs) that underpin generative AI services. US tech media outlet SiliconANGLE emphasized, "Chips like the LPU have the potential to perform AI inference work much more efficiently than GPUs," and explained, "The reason is that AI performance is optimized with SRAM instead of HBM. SRAM consumes much less power than HBM."