Imagine an AI assistant that goes silent, then spills a page of text onto the screen almost instantly. The stream is impressive. The pause is what you remember.
Nvidia’s August 24 production announcement for Groq 3 LPX puts specialized inference processors beside its Vera Rubin GPUs. The aim is faster token generation: accelerating the fragments from which a language model assembles its output. The interesting question is why one answer benefits from different kinds of chip.
To understand the pairing, follow a request from the moment you send it. There is more than one place for the clock to run.
Three moments to watch
The DistServe research paper separates language-model inference into two phases. Prefill processes the prompt and produces the first token; decode generates the subsequent tokens step by step. The first phase can work across input positions in parallel, while each new output token depends on what came before. Those differences create different hardware demands.
| Moment | What it tells you |
|---|---|
| Request sent | Start the clock when the application submits the input. |
| First token arrives | How long you waited before the response began. |
| Final token arrives | How long the complete response took. |
Artificial Analysis measures output speed after the first token arrives. It records the initial wait separately, and distinguishes the first token from the first answer token for reasoning models that emit thinking tokens. A fast stream can therefore follow a slow start. For an application that requires a finished answer before it can act, the final arrival matters too.
This distinction changes how we read Nvidia’s headline. The company reports roughly 3,400 output tokens per second for Gemma 4 31B with a 100,000-token context, attributing the benchmark to Artificial Analysis. Its technical account identifies a system in Nvidia’s own data centers. That is a particular tested deployment, with particular settings; it is not a promise that every request to every provider will finish at that rate.
Why put another processor next to a GPU?
Separating input processing and output generation predates LPX. DistServe placed them on different GPU resources to reduce interference and allocate each phase appropriately. It also accounted for the transfer of intermediate state between machines. Specialization only pays when the handoff does not consume the gain.
Memory movement is another part of the story. FlashAttention’s original research showed how organizing attention calculations in blocks reduces traffic between larger GPU memory and smaller on-chip SRAM. Its wider lesson is useful here: computing more quickly and moving data more intelligently are different ways to improve the same operation. The paper is background research, not a test of LPX.
Nvidia’s architecture description specifies 256 LPU accelerators and 128GB of SRAM across an LPX rack. That memory total belongs to the rack, not each chip. Keeping storage close to computation is part of the design.
The compiler also schedules computation and chip-to-chip transfers ahead of execution. Think of musicians entering on a rehearsed cue: coordinating each small exchange in advance avoids repeatedly negotiating who moves next. Nvidia describes several GPU–LPU arrangements, including GPU prefill followed by LPX decode. Others divide work inside decoding or let LPX propose tokens for a larger model to verify.
Four times faster can save half the wait
Here is an invented example you can check. Suppose ten seconds pass before the first token, followed by twenty seconds of streaming. Keep the first interval unchanged and make only the streaming stage four times faster.
| Stage | Before | After |
|---|---|---|
| Before first token | 10 seconds | 10 seconds |
| Streaming the rest | 20 seconds | 5 seconds |
| Complete response | 30 seconds | 15 seconds |
The stage improves fourfold; the complete wait improves twofold. If the original stream took only two seconds, the same change would reduce the total from twelve seconds to ten and a half. Faster generation matters most when generation occupies a large share of the experience. These numbers are illustrative arithmetic, not measurements of Nvidia hardware.
You can explore the same relationship in our interactive explanation of IBM’s quantum speedup. The technologies are different; the arithmetic of accelerating only part of a job is shared.
The test an application builder should run
Nebius’s adoption announcement says LPX and Vera Rubin will run on its Token Factory platform, initially for a subset of models. It does not provide a general-access date or a specific LPX tariff. Those details matter to anyone deciding whether to build on it.
For a useful comparison, keep the task, model settings and answer requirements consistent. Record the initial wait and the completion time, repeat with concurrent requests, then inspect whether the outputs meet the task. Count usable responses delivered within the time you can tolerate. DistServe calls the latency-qualified request rate goodput.
For an agent, extend that measurement to the work you actually delegated. A completed response may still need a tool call, a test or a correction. Our guide to delegating work to Astra starts at that acceptance boundary. Faster tokens become valuable when they bring the accepted result closer.
Produced with AI-assisted research, drafting and editorial checks; publication authorized by Vastkind’s publisher. No separate human fact-check or original hardware test was performed.
