Nvidia has announced that its Groq 3 LPX rack is in full production, marking a key milestone following its $20 billion acquisition of Groq. The Groq rack will be utilized alongside Nvidia's Vera central processors and Rubin graphics processors at neocloud Nebius, with deployment expected later this year.
This move highlights the increasing demand for low-latency inference technology, which is crucial for creating responsive AI applications, particularly in coding. Nvidia's senior director Dion Harris emphasized that this technology allows cloud companies to offer premium service tiers, catering to customers who require low-latency solutions.
The Groq architecture features 500 megabytes of SRAM on the chip to minimize memory bottlenecks, and Nvidia's LPX racks will house 256 Groq 3 chips, capable of delivering 3,400 tokens per second, according to benchmarks from Artificial Analysis.
The competitive landscape includes Advanced Micro Devices, which plans to integrate its systems with Cerebras chips, focusing on similar low-latency applications. Harris clarified that low-latency chips like Groq do not replace GPUs but serve specific functions within AI workloads.
Nvidia is also ramping up shipments of its Vera Rubin systems, with CEO Jensen Huang projecting $1 trillion in cumulative sales from current and new chip generations by 2027. Nvidia is set to report its earnings soon, which will provide further insights into the impact of these developments