Groq Backed by Nvidia: Why It's Gaining Attention... Will an AI Chip Without HBM Become Reality? [Tech Talk]

Groq Uses SRAM Instead of HBM
Faster Data Transmission, but Scalability Limitations
Big Tech Companies Investing Actively in Affordable Chips

Nvidia has invested 20 billion dollars (approximately 29 trillion won) and entered into a technology licensing agreement with the artificial intelligence (AI) semiconductor design startup Groq. The value of Groq lies in its unique semiconductor design technology. Unlike existing AI chips such as Graphics Processing Units (GPUs) and Tensor Processing Units (TPUs), Groq has significantly reduced its reliance on High Bandwidth Memory (HBM).

The Core Technology of Groq: How the LPU Differs from the GPU

Grok's Language Processing Unit (LPU), differentiated from existing artificial intelligence (AI) chips. Grok

On December 24 (local time), Groq announced via its blog that it had signed a non-exclusive license agreement with Nvidia. As part of the agreement, key personnel including Groq founder Jonathan Ross and President Sunny Madra have joined Nvidia. Nvidia will now be able to use Groq's Language Processing Unit (LPU) technology.

Groq is a US-based AI chip design startup that has developed a unique chip called the LPU. What differentiates the LPU from existing AI chips such as GPUs and TPUs is its approach to memory. Instead of connecting external memory using HBM like other chips, the LPU utilizes memory embedded inside the chip itself.

Embedding Memory Inside the Chip, Not Outside

AI accelerators require memory to store massive amounts of AI data. Nvidia and Google have addressed this by connecting HBM externally to their chips. The boom experienced by domestic memory companies such as Samsung Electronics and SK Hynix was also due to the surging demand for HBM used in GPUs and TPUs.

The main difference between a typical Graphics Processing Unit (GPU) and the Grok LPU is that the Grok LPU connects Static RAM (SRAM) internally, enabling faster data transmission speeds. Inside the Grok LPU, the SRAM is neatly arranged like blocks. Grok

However, some companies have adopted a different memory strategy from Nvidia and Google. Primarily, AI startups including Groq are conducting research using Static RAM (SRAM) embedded inside the chip, rather than external memory. This SRAM is connected right next to the chip's processing unit, or "core," enabling faster data transmission speeds.

Groq's LPU can secure sufficient data capacity and transmission speed even without HBM. In its LPU technology manual, Groq claims, "Groq's memory bandwidth (transmission speed) using SRAM is 80 terabytes per second," and "the HBM of a typical GPU only provides about 8 terabytes per second of bandwidth, so the LPU guarantees up to 10 times the speed advantage and eliminates the need for separate memory chips."

Limitations: Difficult to Handle and Limited Scalability

However, SRAM also has its drawbacks. The biggest issue is that SRAM-based chips are not yet familiar to computer engineers. In contrast, Nvidia holds a dominant position in the early AI chip competition thanks to its AI-optimized software, CUDA. Today, Nvidia GPUs account for about 90% of the AI accelerator market.

Another issue lies in semiconductor manufacturing processes. Currently, the latest computer chips are produced using advanced 2 to 3 nanometer (nm) process nodes. However, the stability of SRAM is weaker in these advanced processes, making it difficult to densely arrange SRAM inside the chip. As the process becomes more miniaturized, the density of SRAM must increase to fully replace external memory, but this has suddenly hit a roadblock.

Thanks to the latest EUV lithography machines, the current production nodes of computer chips have been reduced to the 2 to 3 nanometer (nm) range, but it is difficult to arrange SRAM in this process. ASML

On this topic, Greet Hellings, director at Belgium's semiconductor research facility IMEC, said in an interview with the semiconductor media outlet "SemiEngineering" last year, "There is currently little room to further advance SRAM," adding, "If you shorten the length of SRAM pins, you have to increase their height accordingly. Even with advanced process nodes, the density of SRAM cannot be sufficiently increased."

"Potential to Become a Much More Efficient AI Chip Than GPUs"

Despite these limitations, investment in SRAM has been increasing recently. Groq is not alone in this trend. UK-based Graphcore, a pioneer in SRAM technology, was acquired by SoftBank, led by Chairman Masayoshi Son, in 2024 and is now designing a new AI semiconductor. On December 11, it was reported that Intel was in the final stages of acquiring SambaNova, another SRAM-focused startup.

If SRAM overcomes these obstacles, the LPU has the potential to emerge as an AI semiconductor optimized for large language model (LLM) inference, which underpins generative AI services. US tech media outlet SiliconANGLE stated, "Chips like the LPU have the potential to perform AI inference workloads much more efficiently than GPUs," emphasizing, "The reason is that AI performance has been optimized with SRAM instead of HBM. SRAM consumes much less power than HBM."