AI on phones, tablets, and laptops is changing fast. In fact, it is no longer limited to chatbots and simple text tasks. More models now handle images, audio, and other inputs and outputs. As a result, this shift toward multimodal AI is opening the door to richer apps and more personalized consumer experiences.
Takeaways 🚀
✅Faster local AI, lower latency
✅Better multimodal performance on devices
✅Lower memory use, smoother inference
✅Easier optimization from model deployment
✅More personalized consumer AI experiences
At the same time, running advanced AI locally is still hard. After all, large models need a lot of compute power and memory. Therefore, that creates real limits for mobile and edge devices.
Still, a recent example from Arm and Google shows one way to address this problem. Their work focuses on on-device AI and how to make local inference faster and more efficient. More specifically, it combines new Arm CPU features with the Google AI Edge software stack. Together, these tools aim to simplify optimization and deployment for developers.
In this post, we look at what that means in simple terms. First, we will cover the rise of multimodal AI. Next, we will examine the challenge of local inference, the role of Arm SME2, and the convert-optimize-deploy workflow built around LiteRT, XNNPACK, KleidiAI, Model Explorer, and AI Edge Quantizer.
Why on-device AI matters more now
For years, many AI features relied on the cloud. That model still works well in many cases. However, it also has trade-offs. For example, cloud-based AI can add latency, depend on network quality, and raise privacy concerns.
That is why on-device AI matters. When a model runs on the device itself, it can respond faster and work more reliably in real time. In addition, it can reduce the need to send data elsewhere.
This shift becomes even more important as AI grows beyond text. In other words, multimodal AI can generate or process sound, images, and other media. For example, developers can now explore on-device image and audio generation for apps that feel more personal and responsive.
That matters for users. At the same time, it also matters for developers who want to build tools that adapt to context, preferences, and usage patterns more closely.
AI is moving beyond text to multimodal experiences
Text remains a major part of AI. However, it is no longer the whole story.
Modern models now support tasks such as:
- image generation
- audio generation
- speech understanding
- mixed media input and output
These capabilities can help developers build smarter products. For instance, a creative app might generate sound effects on a phone. Likewise, a media tool might create short audio clips from prompts. Similarly, a camera app might combine image understanding with local personalization.
This is where multimodal AI becomes practical, not just experimental. However, richer outputs also create heavier workloads. In particular, audio and image generation often need more matrix math, more memory, and better runtime efficiency than basic text tasks.
As a result, that puts pressure on the device.
The challenge of local inference
Running AI models locally sounds simple. In practice, it is not.
Most consumer devices rely on CPUs for broad compatibility. On the one hand, CPUs are everywhere. On the other hand, large generative models can push them hard. Because of this, developers often face a trade-off:
- run on the CPU and accept slower performance
- depend on specialized accelerators that vary across devices
As a result, this makes deployment harder at scale. For example, an app may work well on one device and struggle on another. In addition, memory limits add another problem. Even smaller models can consume a lot of RAM during inference.
There is also the question of quantization. On one side, compressing models can improve speed and reduce memory use. On the other side, if done badly, it can hurt output quality. This is especially risky for tasks like audio generation, where users can notice quality loss quickly.
So the goal is not just to run the model. Rather, the goal is to run it well.
How Arm SME2 changes the CPU’s role
One important part of this effort is Arm SME2, short for Scalable Matrix Extension 2.
According to the source material, SME2 adds dedicated matrix compute support directly into the CPU cluster. This matters because generative AI relies heavily on matrix operations. Therefore, if the CPU can handle those operations more efficiently, it becomes much more useful for AI workloads.
In simple terms, SME2 helps the CPU act more like an AI accelerator while still keeping the broad compatibility that CPUs already offer.
The source says SME2 can deliver up to 5x faster inference for some matrix-heavy workloads. However, that figure comes from source-reported results tied to specific scenarios. As such, it should not be read as a universal outcome for every model or device. Even so, it shows why Arm sees SME2 as a key step for faster on-device AI.
What Google AI Edge adds
Of course, hardware is only part of the story. Developers also need a workflow that helps them move from model creation to deployment without too much friction.
That is where Google AI Edge comes in. Specifically, it provides a software stack for edge and mobile AI deployment. In this case, it helps connect models, runtimes, and hardware-aware optimizations.
Several parts of the stack work together:
- LiteRT
- XNNPACK
- KleidiAI
- Model Explorer
- AI Edge Quantizer
Taken together, they support a practical path from research to production.
The convert-optimize-deploy workflow
The source describes a three-step approach: Convert, Optimize, Deploy. Overall, this structure makes the process easier to understand.
Convert: move from PyTorch to LiteRT
Many developers build models in PyTorch. That is common in research and experimentation.
With LiteRT-Torch, developers can convert a PyTorch model into a format that works within the Google AI Edge ecosystem. In doing so, this helps bridge the gap between training and device deployment.
That matters because moving a model from a lab setting to a mobile app often creates extra work. Therefore, a smoother conversion path saves time and reduces complexity.
Optimize: improve speed and size carefully
Once the model is converted, the next step is optimization.
This is where Model Explorer and AI Edge Quantizer help. For example, Model Explorer lets developers inspect the model graph visually. As a result, they can see which parts of the model use the most compute and which layers may be safer to quantize.
That is useful because not every layer responds the same way to quantization. For instance, some layers can move to INT8 with little visible or audible quality loss. However, others may not.
The source highlights this point with the DiT, or Diffusion Transformer, submodule used in an audio generation workflow. In this case, visual inspection helped identify layers where dynamic INT8 quantization produced low error. After that, AI Edge Quantizer compressed selected parts of the model from FP32 to INT8.
This selective method matters. In other words, it shows that good optimization is not about shrinking everything at once. Instead, it is about choosing the right places to reduce precision while protecting quality.
Deploy: run efficiently on the CPU
After conversion and optimization comes deployment.
In the source example, LiteRT runs the model on an Android device. For CPU inference, it uses XNNPACK. In turn, XNNPACK then benefits from built-in integration with KleidiAI, which provides optimized microkernels for key matrix operations.
As a result, this setup helps developers get better CPU performance without writing low-level hardware code themselves.
That is a practical advantage. Consequently, developers can focus more on the app and model logic while the stack handles much of the runtime tuning behind the scenes.
A real example: on-device audio generation
To demonstrate the workflow, the source uses Stable Audio Open Small from Stability AI.
This use case is a strong example of multimodal AI at the edge. After all, audio generation is more demanding than a basic text response. It needs efficient compute, careful memory handling, and quality-aware optimization.
The project aimed to run the model fully on Arm CPUs. This is important because CPUs remain the most widely available option across the mobile market. Therefore, if developers can run these workloads well on CPUs, they can reach more devices.
More broadly, this also connects back to the broader trend. As AI moves toward image and audio generation on-device, developers need workflows that make those experiences practical, not just possible.
Source-reported benchmark results
The source reports several gains when comparing the original FP32 model with an optimized mixed-precision FP16 and INT8 version.
Reported results include:
- more than 2x faster audio generation in tested scenarios
- about 4x lower memory use for the DiT submodel
- around 3x better performance for the DiT submodule after dynamic quantization
- output quality that remained close to the FP32 version in the evaluation described
The source also reports specific timing examples:
- audio generation dropped from 10 seconds to 4.3 seconds on an Apple MacBook M4
- audio generation dropped from 14 seconds to 6.6 seconds on an Arm SME2-based Android device using one thread
In addition, the source says SME2 showed more than 2x performance improvement over NEON in the scenario tested.
These numbers are useful, but they need context. Specifically, they are source-reported benchmark figures from a particular model and test setup. As such, they show the potential of this hardware and software combination. However, they do not guarantee the same results for every workload, app, or device.
What this means for developers
For developers, the main takeaway is clear: on-device AI is becoming more practical for richer applications.
This matters in several ways:
Better reach across devices
CPU-based deployment still offers the widest coverage. As a result, better CPU optimization helps developers support more phones, tablets, and laptops.
Faster local inference
Improved inference speed can make multimodal features feel usable in real products, not just demos.
Lower memory pressure
Likewise, memory savings can help models fit more comfortably on constrained devices. In turn, that can improve stability and user experience.
More room for personalized features
As local generation improves, developers can build more personalized consumer experiences with less reliance on the cloud.
Why this work stands out
What makes this effort interesting is not just one tool or one chip feature. Rather, it is the way the pieces fit together.
- Arm SME2 improves matrix-heavy CPU performance
- Google AI Edge provides the surrounding software stack
- LiteRT supports device execution
- XNNPACK and KleidiAI help accelerate runtime performance
- Model Explorer supports visibility into the model graph
- AI Edge Quantizer helps compress models carefully
Together, that combination creates a more complete path from model development to real deployment.
Conclusion
The push toward multimodal AI is changing what devices need to do. In other words, AI is moving beyond simple text interactions and toward on-device image and audio generation. As a result, that shift can help developers build richer and more personalized consumer experiences. However, it also raises the bar for performance and efficiency.
The Arm and Google AI Edge example shows one practical approach. Specifically, by combining Arm SME2 with tools such as LiteRT, XNNPACK, KleidiAI, Model Explorer, and AI Edge Quantizer, developers get a clearer path to better optimization for local inference.
The reported benchmarks suggest meaningful gains in speed and memory use while keeping quality close to the original model. Still, those results are specific to the source example, not a promise for every use case. Even so, they point to a broader trend: on-device AI is becoming more capable, more efficient, and more useful for real-world multimodal applications.
If you publish this in WordPress, you may also want to add a short editor’s note or a “Further reading” section linking to official product documentation for LiteRT, Google AI Edge, and Arm SME2.
