The New On-Device AI Runtime Stack: From ML Drift to TensorRT Edge-LLM

On-device AI is no longer only a silicon story. The shared runtime that loads, accelerates, and updates models is becoming the product surface developers actually ship.

For most of the 2020s, “on-device AI” conversations started with TOPS charts. That still matters—see our guide to on-device AI and NPUs—but late October 2026 made a quieter point: on-device AI runtime choices now decide whether a model is portable, updatable, and fast enough to feel local.

Three releases landed within three weeks. Google’s AI Edge team open-sourced ML Drift, its next-generation on-device GPU compute engine (8 October 2026). The same stack then landed as GPU acceleration inside LiteRT in Google Play services, with Google’s own claims of about 1.4× faster GPU inference versus the legacy GPU delegate and about 8 MB smaller apps when the runtime is shared rather than bundled (28 October 2026). NVIDIA followed with a developer path for TensorRT Edge-LLM, a lightweight C++ inference runtime that spans prototyping on DGX Spark and deployment toward Jetson Thor, DRIVE AGX Thor, and IGX Thor (30 October 2026).

This article stays on that runtime angle: what an on-device inference stack is, why fragmentation forced a rewrite of GPU delegates, how LiteRT’s Play-services model changes Android packaging, and how TensorRT Edge-LLM frames embedded LLM workflows. It is not a chip catalogue.

What an on-device AI runtime actually does

A runtime sits between a frozen model file and the silicon. In practice it owns five jobs:

1. Load and validate a model graph (for example .tflite, ONNX-derived engines, or vendor HEF/engine caches).
2. Partition work across CPU, GPU, and—when available—NPU or other accelerators, with sensible fallbacks.
3. Lower operators into kernels for OpenCL, OpenGL ES, Metal, WebGPU, CUDA/TensorRT, or vendor NPUs.
4. Manage memory: tensor buffers, KV caches for generative decode, and peak footprint under concurrent apps.
5. Serve the model to application code—native APIs, or increasingly OpenAI-compatible local HTTP for agent prototyping.

When any of those layers are wrong, users feel it as dropped camera frames, cold-start jank, or “local AI” that still phones home because the on-device path never became trustworthy.

ML Drift: an open GPU engine for fragmented devices

Google’s 8 October 2026 announcement frames ML Drift as a high-performance, cross-platform, on-device GPU compute engine built specifically for AI/ML inference, released under the Apache 2.0 license. The company positions it as both the core GPU acceleration engine inside LiteRT and a standalone library for custom graphics and inference runtimes.

Why the legacy GPU delegate was not enough

Unlike data-center inference on homogeneous clusters, device GPUs arrive as a zoo of architectures, drivers, and APIs. Google notes that the TensorFlow Lite GPU delegate established foundational acceleration, but today’s workloads stretch from real-time vision and audio to high-parameter generative models that create compute and memory bottlenecks legacy runtimes were not designed for.

ML Drift’s structural answers, per Google’s post, include:

• Tensor virtualization — decoupling a tensor’s logical layout from physical GPU objects so shaders can be unified across OpenGL ES, OpenCL, Metal, and WebGPU rather than maintained as separate backend-specific codebases.
• Extensible custom ops — registration APIs with low-level shading-language access, including tooling aimed at coding agents that author custom shaders.
• 5D tensor support — addressing a hard limit of the older 4D-centric TFLite GPU delegate for spatiotemporal and volumetric models.
• Stage-aware LLM kernels — switching layouts and kernels between compute-bound prefill and memory-bound decode, including a convolution-aligned KV-cache layout and in-kernel activation quantization during decode.

Google also reports that, in its Gemma benchmarks, ML Drift achieves up to 12% lower memory overhead than other frameworks—its claim, and one that matters because peak tokens-per-second is useless if the process cannot coexist with the camera pipeline.

Production proof points (still Google’s numbers)

Google states ML Drift already runs in production across millions of devices daily inside products such as Chrome, YouTube Shorts, Photos, and Meet. Product-team claims in the same post include:

• YouTube Shorts: up to a 40% reduction in average frame latency for segmentation-based effects on Android and iOS after migrating to ML Drift.
• Google Photos: up to 2 seconds speedup versus the legacy GPU delegate on computational photography / segmentation paths (Google’s phrasing).
• Adobe Lightroom / Photoshop mobile features: up to 30% faster on-device performance for named editing features after upgrading to ML Drift (as reported by Google).
• Snap: 30% model latency improvements for ML-powered face and style effects in Snapchat lenses via the LiteRT ML Drift GPU accelerator (as reported by Google).

Treat those as vendor- and partner-reported deltas, not independent lab results—and still recognize the pattern: the runtime, not a new SoC SKU, unlocked the win.

With ML Drift’s launch, Google says the legacy TFLite GPU delegate will no longer receive new feature updates, and encourages migration to the LiteRT ML Drift GPU accelerator with backwards compatibility for existing models.

LiteRT in Google Play services: ship the model, share the runtime

If ML Drift is the engine, LiteRT in Google Play services is the distribution strategy for Android custom models. Google’s 28 October 2026 post calls it Android’s on-device inference runtime for custom models, “handling more than a trillion inferences a day across billions of devices,” and describes the upgrade as its biggest yet: GPU inference powered by ML Drift through the CompiledModel API—LiteRT’s modern successor to the legacy Interpreter API.

Google’s headline claims (attribute carefully)

• GPU inference runs about 1.4× faster than the legacy TensorFlow Lite GPU delegate, powered by ML Drift.
• In Google’s testing, apps that switched to LiteRT in Google Play services are about 8 MB smaller, because the runtime—including the GPU accelerator—ships once with Play services rather than inside every APK.
• Google notes that more than 100K Android apps still bundle their own LiteRT or TensorFlow Lite copy, so shared-runtime savings compound as apps migrate.

Those figures are Google’s claims from its own testing and telemetry framing—not third-party measurements.

Why the packaging model matters

Bundling a GPU runtime in every app duplicates megabytes and freezes security and performance patches behind each developer’s release train. Play-services delivery gives Google’s stated benefits:

• smaller downloads and shared storage/memory;
• automatic updates via staged Play-services rollouts;
• broader surfaces: phones back to Android 7 (API 24), plus Wear OS 4+ and Android Automotive OS (Android 11+), per Google;
• reuse of the same infrastructure already powering ML Kit Vision and Text APIs at scale.

CompiledModel is positioned as a single workflow for CPU and GPU acceleration (OpenCL where available, OpenGL otherwise, with CPU fallback), keeping the familiar .tflite model format. Google also warns that legacy GPU delegate / custom delegate APIs are deprecated in favor of CompiledModel GPU acceleration and related plugin/dispatch APIs, and that NNAPI will be removed starting in 2027—a planning signal for any team still on Interpreter-era paths. NPU support in Play services, Google says, is “actively being worked on.”

TensorRT Edge-LLM: one framework from desk to device

Mobile GPU delegates are only half of the on-device runtime story. Physical AI and cabin/robot agents need LLM-class serving under tight memory, power, and latency budgets—often on NVIDIA’s embedded line rather than a phone SoC.

NVIDIA’s 30 October 2026 technical blog presents TensorRT Edge-LLM as a lightweight C++ inference runtime for embedded workloads, bringing the same inference framework to development on DGX Spark (including a 64GB configuration) and deployment on platforms such as Jetson Thor, DRIVE AGX Thor, and IGX Thor.

Developer surface, not a silicon brochure

The blog’s emphasis is operational:

• install tensorrt-edgellm and use the trtedge CLI to download a model and build optimized TensorRT engines;
• chat locally with models such as NVIDIA Nemotron-3.5 Lightning;
• serve via a local OpenAI-compatible API for application prototyping;
• optionally enable DSpark speculative decoding with a draft model.

That “few commands” framing is the product: shrink the gap between a Hugging Face checkpoint and an engine that can sit behind a cabin or factory agent.

NVIDIA’s published Spark numbers (NVIDIA’s measurements)

On DGX Spark 64GB, NVIDIA reports TensorRT Edge-LLM results for Nemotron-3.5 Lightning (NVFP4, thinking enabled, batch size one, one concurrent streaming request, 128-token output limit). Mean generation speed on HumanEval prompts rises from 73.4 tokens/s with DSpark off to 126.9 tokens/s with DSpark on (three draft tokens), while peak total system memory use during inference with DSpark was 27.9 GB including the OS—NVIDIA’s table and footnotes. Median TTFT figures in the same table (for example ~98 ms on HumanEval with or without DSpark) are likewise NVIDIA-reported under that setup.

The point for this article is architectural: Edge-LLM is an on-device AI runtime that tries to keep the same optimized path from a developer workstation to Thor-class deployment targets, rather than inventing a one-off mobile demo stack.

How the pieces fit (and where they do not)

They are complementary more than competitive. A phone camera effect wants LiteRT + ML Drift. A robot or vehicle agent prototype may want TensorRT Edge-LLM on Spark before locking an embedded engine. Teams that treat “on-device” as one binary miss that split.

Practical checklist for teams

1. Stop shipping duplicate runtimes by default. If you are on Android with custom .tflite models, evaluate LiteRT in Play services—Google’s ~8 MB claim is specifically about not bundling the runtime.
2. Plan the Interpreter → CompiledModel migration. Deprecation timelines (including NNAPI removal starting 2027, per Google) are calendar items, not blog fluff.
3. Profile prefill and decode separately for local LLMs. ML Drift’s stage-aware design and Edge-LLM’s speculative-decoding knobs both assume you know which phase dominates.
4. Treat partner speedups as directional. Google’s 30–40% product anecdotes and NVIDIA’s Spark tokens/s table are useful—and explicitly their measurements under their conditions.
5. Keep silicon literacy, but write contracts against runtimes. APIs, engine caches, and update channels outlive a single NPU generation—another reason the NPU explainer in our guide to on-device AI and NPUs pairs with this runtime piece rather than replacing it.

Bottom line

October 2026 did not invent on-device inference. It did make the on-device AI runtime impossible to ignore. ML Drift open-sourced a GPU engine built for generative and classical on-device models across OpenGL ES, OpenCL, Metal, and WebGPU. LiteRT pushed that engine into a shared Android delivery path, with Google claiming about 1.4× GPU speedups versus the legacy delegate and about 8 MB smaller apps when the runtime lives in Play services. TensorRT Edge-LLM showed the embedded half of the same idea: one C++/CLI path from Spark prototyping to Thor-class deployment, with NVIDIA publishing concrete Spark throughput numbers under DSpark.

Chips still set the ceiling. Runtimes decide whether you ever hit it—and whether you can update the floor without waiting for a new phone, car, or robot SKU.

Related reading

• On-Device AI in 2026: NPUs, Local Models, and What Still Needs the Cloud
• Always-On AI: When Inference Must Fit a Coin Cell
• Offline AI: When Robots and Vehicles Must Think Without the Cloud

Images: original illustrations generated for AndBeyond.Tech; no logos or real people depicted.