On-device AI is entering a new phase. In the past, it was mostly limited to text responses or simple tasks on a phone, tablet, or laptop. Now, however, it is moving toward richer multimodal capabilities, including image generation and audio generation directly on the device. As a result, developers can create personalized consumer experiences that are faster, more private, and better adapted to each user’s immediate context.
Takeaways 🚀
✅Faster on-device AI model performance
✅Lower memory use for inference
✅Multimodal image and audio capabilities
✅Streamlined optimization and deployment workflow
✅Personalized experiences on consumer devices
In this context, performance optimization becomes especially important. After all, running advanced AI models on consumer hardware requires a careful balance of three key factors at once: model quality, inference speed, and memory usage. At the same time, a recent approach presented by Google and Arm shows that this balance can be improved through a combination of software tools and CPU advances, particularly with Google AI Edge, LiteRT, XNNPACK, KleidiAI, quantization, and SME2.
The evolution of on-device AI toward multimodal experiences
On-device AI is changing in visible ways. For years, many experiences focused mainly on text, basic classification, or voice recognition. Today, however, models can work with multiple types of data at the same time and, as a result, generate more complex outputs directly on the device.
This shift toward multimodal AI includes several important use cases, for example:
On-device image generation
In particular, on-device image generation can help create more personalized visual content while also reducing reliance on the cloud and lowering latency.
On-device audio generation
Likewise, on-device audio generation makes it possible to create clips, sound effects, or music from user prompts. As a result, this capability can support a wide range of applications, including creative apps, games, educational tools, and entertainment experiences.
Personalized consumer experiences
When inference happens directly on the device, developers can, in turn, tailor features to the user’s immediate context. As a result, response times can improve, and, in some cases, data can remain closer to the user instead of being sent continuously to remote servers.
The challenge: quality, speed, and memory on consumer devices
In the case described by Google and Arm, the goal was to run a compact yet still advanced audio generation model within the real limits of mobile hardware. However, the central challenge was not simply getting the model to run. Rather, it was also necessary to maintain acceptable output quality while, at the same time, reducing the computational cost.
The main challenges, therefore, were clear:
Maintaining model quality
Applying quantization aggressively can, on the one hand, reduce model size and speed up inference. On the other hand, it can also lower the quality of the final output. In audio models, for example, this kind of loss can become noticeable very quickly.
Improving inference speed
For the experience to be truly useful, however, generation time must remain reasonable. Otherwise, if a task takes too long, the advantage of running AI locally begins to fade.
Reducing memory usage
Memory, meanwhile, remains a major constraint on phones and other consumer devices. For that reason, a lighter model can, in turn, improve compatibility and support more stable performance.
The role of Arm in accelerating on-device AI
In this approach, Arm plays a central role as a hardware platform that is already widely used in mobile devices and other everyday computing products. Rather than relying only on specialized accelerators, the strategy instead focuses on improving AI execution directly on the CPU. As a result, it presents a more practical path for bringing advanced on-device AI to a broader range of devices.
SME2 as the basis for acceleration
SME2 is an Arm extension designed to improve matrix operations, which are essential for many generative AI models. More specifically, according to the shared data, this architecture allows the CPU to handle intensive AI workloads more efficiently. As a result, it helps make on-device AI faster and more practical on everyday consumer hardware.
For a general audience, the key idea is straightforward: it is not only about having more raw power. Rather, it is also about using instructions that are better suited to the kinds of calculations these models rely on.
A more useful CPU for advanced AI
This, in turn, can help reduce a long-standing limitation of on-device AI. In the past, running complex models on a CPU often meant accepting high latency or relying on more fragmented hardware. With improvements such as SME2, however, the CPU becomes a more practical and valuable option for advanced inference tasks.
What Google AI Edge brings to optimization
From a software perspective, Google AI Edge serves as an integrated stack that helps move a model from its original form to efficient on-device execution. More importantly, its value lies not only in final speed but also in making what is often a complex process much simpler and easier to manage.
Within this workflow, several key components stand out. Together, they help simplify the path from model development to efficient on-device deployment.
LiteRT
LiteRT is the runtime used to bring models onto the device. In this case, it also serves as a key integration point for other layers of optimization. As a result, developers can move more smoothly from model preparation to efficient on-device performance.
XNNPACK
XNNPACK acts as the engine for CPU inference. In practice, it selects compute-intensive mathematical kernels and, as a result, helps make better use of the available hardware. Consequently, developers can achieve more efficient on-device performance without adding unnecessary complexity.
KleidiAI
KleidiAI provides Arm-optimized microkernels. When integrated with XNNPACK, it helps enable automatic acceleration, so developers can improve performance more easily without having to write low-level code themselves.
Quantization and analysis tools
Quantization, in turn, reduces the numerical precision of certain parts of the model in order to save memory and speed up calculations. At the same time, Google highlights visual tools that help developers identify which layers are safer to quantize, thereby improving efficiency without significantly affecting overall quality.
The Convert, Optimize, Deploy workflow
One of the most useful aspects of this approach is its clear workflow. Rather than treating optimization as a series of separate adjustments, it instead organizes the process into three connected stages: Convert, Optimize, and Deploy.
Convert: from PyTorch model to runtime environment
The first step, therefore, is to convert a model built in PyTorch into a format that is compatible with the Google AI Edge ecosystem. To do this, developers use LiteRT-Torch, which helps make the transition smoother and more efficient.
This matters because many models begin in research or prototyping environments. As a result, if the path to production is difficult, deployment to the device becomes slower and more expensive. By contrast, a more direct workflow helps reduce that friction and makes the transition much smoother.
Optimize: identifying what can be quantized
The optimization phase, in particular, focuses on finding the right balance between efficiency and quality. To do this, developers use visual inspection of the model graph to identify the most compute-intensive parts, as well as the areas that can better tolerate quantization. As a result, this step makes it easier to improve performance while still preserving overall output quality.
In the example described, dynamic INT8 quantization was applied to the diffusion transformer submodule. As a result, according to the report, this decision helped speed up that part of the model while still maintaining the perceived quality of the generated audio.
This point is especially important: not every part of the model should be treated the same way. Instead, effective optimization depends on understanding which layers can be compressed and which ones, by contrast, should be preserved with greater precision.
Deploy: efficient CPU execution with LiteRT
Once the model has been quantized and refined, the deployment stage begins. At this point, LiteRT uses XNNPACK for CPU inference and, in addition, takes advantage of its integration with KleidiAI to deliver better performance on Arm hardware.
For developers, the practical advantage is clear: automation simplifies the entire process. Instead of building each processor-specific optimization manually, they can rely on the software stack to handle much of the work. As a result, the model can be translated from a high-level design into more efficient execution on the available silicon.
Reported results: faster speed and lower memory usage
In this case, the example focused on an audio generation model running entirely on the CPU. More specifically, it was tested both on an Arm-based Android device with SME2 and on a MacBook powered by an Arm chip.
The reported results, in turn, highlight several important gains in speed, memory efficiency, and overall model performance.
Speed improvement
Overall, audio generation time was reduced by more than 2x compared with the original FP32 version. For example, in the shared test, the time dropped from 10 seconds to 4.3 seconds on a MacBook M4. Similarly, on an SME2-based Android device using a single thread, it fell from 14 seconds to 6.6 seconds. As a result, the optimized model delivered significantly faster performance across both platforms.
Memory reduction
In addition, the DiT submodel reduced its memory footprint by around 4x after optimization. This matters because, in many cases, available memory determines which devices can run a model reliably and efficiently.
Partial improvement by submodule
Google and Arm also reported a 3x improvement in the DiT submodule after applyingquantization INT8 dinámica.
Quality maintained
According to the report, the generated audio maintained perceptual parity with the FP32 version. In other words, even though the model became smaller and faster, there was no clear loss in the quality of the final output.
Why this approach matters for the future of multimodal AI
Beyond the specific case ofaudio generation, el trabajo apunta a una tendencia más amplia. Si herramientas como Google AI Edge pueden simplificar la optimization y CPUs Arm con SME2 pueden acelerar cargas de IA complejas, entonces será más viable llevar funciones multimodal avanzadas a dispositivos de consumo masivo.
As a result, this progress could benefit several types of applications in meaningful ways:
Creative apps
For example, on-device image generation and audio capabilities can deliver results more quickly and, at the same time, make those results feel more personalized. In turn, this can help creative apps offer smoother and more engaging user experiences.
Productivity tools
Likewise, local inference can support editing, summarization, translation, and contextual assistance while also reducing dependence on the cloud.
Personalized experiences
As a result, personalized consumer experiences can improve when AI responds to the user’s context in real time, directly on the device. In turn, this can make interactions feel faster, more relevant, and better tailored to individual needs.
An objective view of the progress
The Arm and Google AI Edge approach does not remove every challenge of on-device AI. Even so, these models still place heavy demands on devices, and each use case must carefully balance quality, latency, energy use, and memory. At the same time, the work shows a clear step forward. By combining hardware improvements, CPU-based execution, and software tools, it becomes more practical to bring generative AI directly onto everyday devices.
In practical terms, the main message is clear. On-device AI is moving beyond simple tasks and, increasingly, toward richer multimodal capabilities, such as image generation and audio generation. As this shift continues, optimization becomes just as important as the model itself. In other words, for this transition to work at scale, developers need both strong models and efficient performance. In this context, technologies such as LiteRT, XNNPACK, KleidiAI, quantization, and SME2 are helping bridge the gap between research and real-world use on everyday devices.
Next steps
For development teams, this kind of workflow suggests three practical next steps. In particular, it highlights areas where careful planning can improve both performance and deployment. More broadly, these actions can help teams move from experimentation to real-world implementation more smoothly.
Evaluate which parts of the model tolerate quantization
However, not every layer should be compressed in the same way. Instead, teams should evaluate each part of the model carefully, since measuring the impact on output quality remains essential throughout the process.
Design for CPU as well as dedicated accelerators
In addition, device coverage improves when the CPU can also deliver strong performance. As a result, developers can support a wider range of devices without depending entirely on specialized accelerators.
Adopt an integrated conversion, optimization, and deployment workflow
In addition, a clear workflow, such asConvert, Optimize y Deploy, puede reducir tiempo de implementación y facilitar pruebas en hardware real.
