On-device ML is no longer experimental. It's the default for anything privacy-sensitive, latency-sensitive, or required to work without connectivity. In Flutter, the two practical options are TensorFlow Lite (via `tflite_flutter`) and ONNX Runtime (via `onnxruntime`). We've shipped both in production.
Here are the numbers we use to pick between them — a custom 38MB voice model benchmarked on iPhone 11+ and a Samsung A52, with the platform-channel overhead nobody talks about included.
Setup overhead and model conversion
TFLite: install the `tflite_flutter` package and convert your model to `.tflite`. Native if you're training in TensorFlow or Keras. PyTorch-trained models need a TF intermediate, which sometimes loses precision on custom ops.
ONNX Runtime: install the `onnxruntime` package and export to `.onnx`. Direct path from PyTorch, Hugging Face, and most research codebases. Re-exporting a TF-trained model to ONNX is also straightforward via tf2onnx.
If your team trains in PyTorch, ONNX is the natural path and the conversion friction is significantly lower. If you're a TF/Keras shop, TFLite is the obvious answer.
Binary size impact
TFLite runtime adds approximately +6.2MB Android, +4.1MB iOS to your app bundle.
ONNX Runtime adds approximately +14.3MB Android, +8.7MB iOS — roughly double TFLite. For a 30MB app this is a noticeable bump; for a 100MB app it's noise. Your call depends on what category your app is in.
Load time and warm-up
Loading our 38MB VITS voice model on iPhone 11:
TFLite — 380ms cold, 80ms once warm.
ONNX — 290ms cold, 60ms once warm.
ONNX is faster on first load in our tests, marginally. Both are fast enough that you can do this in app initialization without user-visible delay.
Inference latency on a real workload
Generating a 5-word TTS phrase on iPhone 11:
TFLite (CPU): 320ms. TFLite (GPU delegate, where supported): 180ms.
ONNX (CPU): 280ms. ONNX (CoreML execution provider, iOS only): 145ms.
On Android (Samsung A52, no GPU delegate available for our model): TFLite 580ms, ONNX 510ms. Mid-range Android is the bottleneck — flagship iOS handles either runtime with margin to spare.
The platform-channel overhead nobody mentions
Every Flutter inference call crosses the Dart-to-platform boundary. For one-shot inference (image classification, single-shot TTS) this is negligible — 2–4ms per call. For streaming inference (live audio, frame-by-frame video processing) this overhead compounds quickly and the Flutter wrapper stops being the right architecture.
If you're streaming, drop down to native. Write a thin Kotlin/Swift class that holds the inference loop and exposes a single platform-channel method that returns batched results. The Flutter side calls it once per second instead of once per 30ms.
When to pick which
TFLite: smaller binary, simpler integration, mature on Android, native fit for TF-trained models.
ONNX Runtime: faster on iOS via CoreML, broader model compatibility (PyTorch, Hugging Face), bigger binary, the right call for cross-runtime portability.
Streaming inference at sub-50ms latency: avoid both Flutter wrappers — go native and call native via a single channel.
For static models — single-shot inference, image classification, on-demand TTS — the choice between TFLite and ONNX is mostly about your existing training stack and your binary-size tolerance.
For latency-sensitive streaming, both Flutter wrappers add overhead that the demos don't expose. Drop down to native for the inference loop and call it via a thin platform channel — and ship something that actually feels real-time, instead of one that's almost real-time on a flagship.
