Google has introduced the LiteRT Tensor API, a lightweight C++ API for building, customizing and optimizing on-device machine-learning models. It lets developers define a computation graph operator by operator, then run that graph through LiteRT on CPU, GPU or NPU hardware without rewriting it for each target, according to Google.
The API also supports loading weights directly from checkpoints, including Hugging Face .safetensors files. Google says its zero-copy buffer interoperability can avoid copying accelerator buffers between preprocessing, inference and postprocessing. Coding agents can generate C++ to build, modify, validate and benchmark model graphs.
Google demonstrated the API with a multi-object segmentation and tracking app built using a coding agent. In its tests, the same SAM 2 C++ model code ran on a macOS laptop and an iOS phone using Metal, and an Android phone using OpenCL. Google reported that its MacBook Pro implementation was about 1.5 times faster than Apple MLX for the image encoder and mask decoder at fp16. That figure is a result from Google’s tests, not an independent benchmark.
Google said the coding agent moved SAM 2’s per-frame decisions into the computation graph and replaced operations that caused CPU fallback on WebGPU with GPU-compatible arithmetic. Commenting on the announcement, Arm AI director Laurence Moroney recommended starting with model conversion and considering the C++ API when graph-level control is needed or preprocessing and postprocessing must stay on the accelerator. He cited a WebGPU improvement from under 10 fps to over 30 fps at 512 × 512 resolution.
Google lists native ML Drift backend support, more NPU-friendly operations, automatic whole-graph quantization and broader dynamic-shape support as planned improvements.
Original article (InfoQ China) · English article linked by InfoQ China