Modern smartphones can handle advanced AI applications directly on the device. As a result, users enjoy interactive, real-time experiences without relying on servers. However, these complex tasks can push phone processors to their limits and cause slowdowns. Fortunately, the Neural Processing Unit (NPU) helps solve this challenge. In this article, we share how developers can get the most out of the NPU using LiteRT Qualcomm AI Engine Direct (QNN) Accelerator.
Takeaways 🚀
✅Unified workflow simplifies AI deployment.
✅NPU enables faster mobile processing.
✅LiteRT boosts on-device AI efficiency.
✅Supports cutting-edge generative AI models.
✅Significant performance gains for developers.
The Power of the NPU for On-Device AI
Smartphones today use a System on a Chip (SoC) that includes a CPU, a GPU, and an NPU. While many AI apps use the GPU, relying only on the GPU can slow things down when the phone handles several demanding tasks at once—such as generating images and processing live camera input together.
Unlike other processors, the NPU focuses entirely on AI. It delivers more processing power, measured in Tera Operations Per Second (TOPS), than CPUs or GPUs. Additionally, it uses less energy, which helps save battery life. Since the NPU, CPU, and GPU work at the same time, each processor can do what it does best. The NPU takes care of heavy AI workloads, the CPU runs the main logic, and the GPU handles graphics. With this teamwork, apps remain smooth and responsive.
Making NPU Power Easier to Use with LiteRT
To help developers use the NPU more easily, Google and Qualcomm built the LiteRT Qualcomm AI Engine Direct (QNN) Accelerator. This tool replaces an older one called the TFLite QNN delegate. As a result, LiteRT QNN brings important benefits for creating fast, on-device AI.
Easier and Unified Deployment
With LiteRT, deploying AI models becomes much simpler. Previously, developers had to use complicated, vendor-specific software and target each chip version separately. Now, LiteRT takes that hassle away.
With LiteRT you can:
- Skip low-level SDKs. Because LiteRT handles compilers and runtimes behind a friendly API, you can focus on your model.
- Target many chips at once. It manages the differences between SoC versions, so you can now deploy your models across more devices with less effort.
Furthermore, you can also use pre-trained AI models with either ahead-of-time (AOT) or on-device compilation. Thus, the process is now smooth and straightforward.
Top-Level Performance
LiteRT aims for full use of the NPU. It supports a wide set of LiteRT operations, so the whole model can run on the NPU. You don’t need to split work between processors unless absolutely needed.
Moreover, this tool includes special features for running Generative AI and Large Language Models (LLMs). Because of this, even advanced models can run well on modern phones.
Impressive Performance in Real-World Tests
Benchmarks clearly show LiteRT QNN’s strengths. In tests covering 72 common machine learning models for vision, language, and audio tasks, the new accelerator delivered major gains.
- It boosts performance up to 100 times over CPU and 10 times over GPU for many tasks.
- It supports 90 LiteRT operations, letting 64 out of 72 models run fully on the NPU.
On the Snapdragon 8 Elite Gen 5 chip, more than 56 models finished under 5 milliseconds on the NPU. By comparison, only 13 models reached that speed on the CPU. Because of this, live, interactive AI is now available to mobile users—something not possible before.
Power for Advanced Vision Models
LiteRT also shines with large-scale language and vision models. For example, the FastVLM-0.5B vision model runs quickly when optimized for the NPU. By using quantization (int8 for weights, int16 for activations), the model taps into the NPU’s best features.
On Snapdragon 8 Elite Gen 5 NPU, the results include:
- Time-to-first-token (TTFT) of just 0.12 seconds for high-res images.
- Over 11,000 tokens per second during the prefill phase.
- More than 100 tokens per second for decoding.
Thanks to these speeds, applications like live scene understanding can work smoothly right on your phone.
Best Practices for Using LiteRT and the NPU
If you’re a developer, start with these tips to get the best results from LiteRT.
1. Use Ahead-of-Time (AOT) Compilation
AOT compilation isn’t required, but it is highly recommended for large AI models. By compiling the model before you ship it, you ensure faster startup and lower memory use for users. For instance, you can use Python to compile for all supported SoCs or specific chip versions. The final models go into a “Google Play AI Pack” for easy app updates.
2. Deploy via Google Play for On-device AI
When setting up your Android project, you can choose either method:
- On-device compilation: Add the
.tflitefile to your app’s assets folder. - AOT compilation: Add the AI Pack to your project and connect it with Gradle.
In addition, you’ll find scripts to fetch and add the needed NPU runtime and compiler libraries, making setup more efficient.
3. Run Models with the LiteRT Runtime API
LiteRT keeps your code simple and efficient. With just a few lines, you can run your model on the NPU. Built-in fallback options mean you can choose CPU or GPU if NPU isn’t available. Moreover, the fallback also supports AOT compilation, so unsupported pieces of a model can shift smoothly to the CPU or GPU.
What’s Ahead for On-Device AI
In summary, LiteRT Qualcomm AI Engine Direct (QNN) Accelerator bridges the gap between chip power and real-world app speed. It makes deployment easier and brings out the best in the NPU. Consequently, developers can now build the next generation of fast, responsive, and smart mobile apps.
