Speed matters when a language model generates text. In turn, it shapes cost, latency, and the overall user experience. That is why, in practice, any gain in LLM inference can directly improve AI products, research, and large-scale deployments.
Takeaways 🚀
✅Accelerating LLM inference with DFlash
✅Achieving 3X speed on TPUs
✅Parallel drafting for faster results
✅Innovations by UCSD researchers
✅Open-source milestone with Google
A recent advance on Google TPUs shows that there is still plenty of room to improve LLM inference. More specifically, researchers at UCSD, led by Hao Zhang, integrated DFlash—a diffusion-style speculative decoding method—into the vLLM TPU ecosystem. As a result, the findings stood out. The team achieved an average 3.13x speedup on TPU v5p, with even larger gains on math and coding tasks.
This milestone also matters because of its open-source impact. In fact, Google highlighted the project as a major open-source achievement, and for good reason. Notably, Hao Zhang is known as the co-inventor of paged attention and prefill/decode disaggregated serving, two key ideas that helped make LLM serving more efficient. In this case, his team pushed that progress further by adapting a non-sequential generation method for the TPU environment.
What Problem Does LLM Inference Solve?
Large language models usually generate text in an autoregressive way. In other words, they produce one token, then the next, and then the next. As a result, each step depends on the one before it.
This approach works well, however, it has a clear downside. After all, modern hardware like Google TPUs can handle many tasks at once. Yet token-by-token generation forces the system to move step by step. As a result, it cannot use the accelerator’s full power.
This is where speculative decoding comes in. At its core, the idea is simple. First, a smaller draft model proposes several future tokens, and then the main model checks them in parallel. When that draft is accurate, the system can accept multiple tokens at once and, as a result, cut latency.
However, many speculative methods still hit a bottleneck. Although they verify tokens in parallel, they still draft them one by one. As a result, that limits the overall speedup.
The Key Shift: From Sequential Drafting to Parallel Drafting
This work, more specifically, replaces that step-by-step process with a parallel, diffusion-based approach.
Instead of generating candidates one by one, DFlash proposes a full block of tokens in a single pass. As a result, the technical paper describes this as a shift from O(K) sequential cost to nearly constant O(1) cost when generating a block of size K.
To explain it in simple terms, for example, it helps to compare two ways to write a draft:
- The traditional approach writes one word, waits, writes the next, and then waits again.
- DFlash tries to “paint” the whole block at once.
This matters because, in turn, it removes much of the delay caused by the draft stage. At the same time, it also fits TPU design well, since TPUs excel at running large parallel workloads.
UCSD and Hao Zhang Drive This Open-Source Breakthrough
In particular, Google highlighted this advance as a major open-source milestone from UCSD researchers led by Hao Zhang. More importantly, that matters because the project went far beyond a simple internal test or a closed prototype.
As a result, the team brought block-diffusion speculative decoding to the open vLLM TPU inference stack. In turn, this move makes the technology more accessible to developers, researchers, and organizations that want to build faster TPU inference systems.
This contribution also builds on a strong track record. For example, Hao Zhang has played a key role in earlier work on LLM serving efficiency, especially paged attention and prefill/decode disaggregated serving. Over time, both ideas improved how real-world systems handle memory, compute, and inference flow. In the same way, this new integration follows the same principle: strong models matter, but smarter serving matters too.
What DFlash Is and Why It Stands Out
DFlash is a diffusion-style speculative decoding framework. More importantly, it improves the drafting stage, which remains a weak point in many inference systems.
In classic autoregressive methods, proposing K tokens usually takes K inference steps. As a result, that adds latency and extra coordination between calls. By contrast, DFlash avoids that overhead by generating the full candidate block in a single pass.
As a result, that difference led to clear gains in the reported tests:
- 3.13X de aceleración promedio en tokens por segundo sobre TPU v5p
- Mejoras cercanas a 6X en tareas complejas de matemáticas
- 2.29X de speedup en una comparación directa contra EAGLE-3
- En tareas de código como mbpp, el tiempo por token bajó de 9.81 ms a 3.48 ms
Taken together, these numbers show a clear and meaningful improvement. More importantly, DFlash delivered faster end-to-end performance, not just a small gain.
How DFlash Fits Into the vLLM TPU Ecosystem
Bringing DFlash to the Google TPU and JAX environment, however, took more than a simple code conversion. Because the original system was built for a different stack, the team therefore had to redesign key parts so it could work smoothly with TPU-optimized inference.
This process, in particular, involved several key technical innovations.
Double-Cache Design for Attention
One of the biggest challenges, in particular, involved attention.
In TPU environments, high-performance serving uses paged attention with specialized kernels. In general, this approach stores memory in fixed pages to make better use of the hardware. However, DFlash uses block diffusion, and as a result, that design does not fit naturally with this setup.
To solve this, therefore, the researchers designed a dual-cache architecture:
- The target model continues to use a paged KV cache
- The draft model uses a specialized path with static JAX arrays
As a result, this split kept TPU serving efficient while preserving DFlash’s core design.
Smarter Context Management
At the same time, DFlash does not work in isolation. Instead, the draft model uses internal signals from the main model, including its intermediate hidden states.
That required efficient context storage and transfer. To address this, the team used a power-of-two padding strategy to avoid bottlenecks between the host CPU and the TPU. As a result, the system could move data in more efficient blocks and reduce communication overhead.
The team also tracked exactly how much context the draft model had already used. In turn, this step matters because it prevents repeated processing and keeps the proposal and verification stages aligned.
Fixing the Metadata Gap
Another challenge, meanwhile, came from handling persistent state. Specifically, DFlash relies on elements such as:
- context buffers
- KV cache positions
- RoPE offsets
- true count of accepted tokens
In the TPU-optimized vLLM pipeline, however, the proposer received metadata designed for more common methods, not for a diffusion-style architecture. As a result, this mismatch inflated the sequence length and caused internal misalignment.
The team fixed this issue by first reconfiguring the proposer to sync only with the true number of accepted tokens. As a result, this change kept the block logic aligned and therefore let DFlash run with full mathematical precision on TPU.
Why the 3x Speedup Matters
A 3x improvement in inference does more than speed up the system. In addition, it can lower cost per output, improve response times, and let the same infrastructure serve more users.
For example, in tests on TPU v5p, the team saw an average 3.13x performance gain. Moreover, results were even stronger in demanding areas like math and coding.
For example, the results included:
- In math500, generation time dropped from 8.02 ms per token to 1.40 ms
- In humaneval, improvements exceeded 3.5x
- In direct comparisons, DFlash clearly outperformed EAGLE-3
This advantage comes from more than raw speed. More specifically, DFlash avoids unnecessary sequential steps. By contrast, EAGLE-3 still proposes a small number of tokens in an autoregressive way. Meanwhile, DFlash generates a larger, higher-quality block in a single pass. As a result, this cuts coordination between steps and makes better use of hardware parallelism.
A Closer Look at the “K-Flat” Effect
One of the most striking findings in the study, in particular, involved the cost of verification.
Researchers found that on hardware like TPU v5p, checking many more tokens does not always cost much more. For instance, in some cases, verifying 1024 tokens can cost about the same as verifying 16 tokens.
Researchers called this phenomenon “K-Flat.” As a result,
In these accelerators, loading model weights already takes up much of the runtime. As a result, expanding the verification block does not raise the cost as much as many might expect.
This changes how we think about optimization. In other words, if the system can verify large blocks at almost no extra cost, then draft quality becomes the main limit, not block size.
Why Quality Matters More Than Quantity
Based on that analysis, however, the team concluded that simply increasing K is not enough.
Wider blocks can help, for example, but only up to a point. Beyond that, the gains get smaller. In fact, in the team’s analysis, K=16 already captured more than 90% of the maximum theoretical speedup under their test conditions.
That is why the priority is not just to propose more tokens, but also to propose them more accurately. In practice, therefore, improving the acceptance rate at each position in the block matters more than endlessly increasing block length.
Moreover, this point matters because it shows where research may go next: better draft models, better training, and more precise methods for tasks where structured text makes prediction easier.
Why Math and Coding See Bigger Gains
However, not all tasks behave the same way.
The team found that DFlash gains depend on how predictable the text is. For example, math and coding tasks usually follow stricter rules. As a result, that makes it easier for the model to predict several tokens in a row within the same block.
By contrast, meanwhile, open-ended chat and general conversation tend to be less predictable. As a result, that lower predictability reduces token acceptance as the block moves forward.
As a result, the findings can be summarized as follows:
- math sees the biggest gains
- coding also shows strong improvements
- conversational chat improves too, but more moderately
Taken together, this pattern suggests that faster inference with parallel drafting may work especially well for systems built around structured reasoning.
A Technical Advance with Open-Source Impact
Another key part of this announcement, meanwhile, is its open-source focus.
The team submitted the implementation to the vLLM tpu-inference repository. Specifically, it includes DFlash model components, integration into the speculative decoding pipeline, and end-to-end tests. As a result, this makes it easier for the community to reproduce, study, and build on the work.
This matters because, in many cases, inference advances stay locked inside labs or closed services. By contrast, here, the innovation connects with an ecosystem that researchers and developers already use to serve LLMs efficiently.
In addition, it also strengthens ties between universities, hardware platforms, and open-source software. As a result, this kind of collaboration often helps complex ideas reach real-world use faster.
What this advance means for the future of LLM inference
Taken together, this work leads to several clear conclusions.
The first is that LLM inference still has a lot of room for improvement at the systems level. In other words, progress does not depend only on training larger models. Instead, in many cases, the biggest gains come from changing how systems generate and verify text.
The second is that diffusion-style speculative decoding can, therefore, offer a strong alternative to traditional autoregressive methods, especially on hardware built for massive parallelism, such as Google TPUs.
Moreover, the third conclusion is clear: the bottleneck is shifting. As a result, if verifying large blocks is no longer the main problem, the focus moves to draft quality, token acceptance rates, and system tuning for each type of task.
Final Thoughts
In this sense, the integration of DFlash into the vLLM TPU ecosystem marks an important step toward making LLM inference more efficient. In addition, the UCSD team, led by Hao Zhang, showed that it can replace sequential drafting with parallel drafting and, in doing so, turn that idea into real performance gains on Google TPU v5p.
Overall, the results are clear: 3.13x average speedup, much bigger gains in math, and a strong lead over more traditional speculative decoding methods. More importantly, this work shows that a different architecture can use modern hardware more effectively and, in turn, create new paths for serving language models.
For anyone following the evolution of LLM inference, ultimately, this milestone sends a clear message: the future depends not only on more capable models, but also, more importantly, on smarter systems that run them efficiently.
