Helio
Assess an edge AI project
Back to blog

Field notes

Android On-Device Animal Identification in Practice: From Training to LiteRT Inference

How Safari Go moves from PyTorch training and LiteRT export to local animal identification on Android, with observed GPU probing and a deterministic CPU fallback.

Safari GoOn-device AIModel training
A local image pipeline moving from an animal frame through detection and classification stages.

When an animal-identification app has to work in the field, “the model runs” is only the beginning. The image contract, export path, hardware delegate, memory lifecycle, and fallback behavior all affect the result users see.

Safari Go uses a two-stage cascade that runs locally on Android. This article walks through the engineering path from training data to an on-device result, including the hardware path we use today and the boundaries we do not claim to have solved yet.

1. The target: private, offline identification

The Android inference path has four practical requirements:

  • The photo should be decoded and classified on the device. Network calls are reserved for product services such as trial state, entitlements, purchases, and guide downloads.
  • PyTorch and Android must agree on the same input, output, and coordinate contracts.
  • GPU acceleration should be used only after a real compatibility and delegate probe succeeds.
  • A device that cannot run the delegate must still have a deterministic FP32 CPU path instead of a crash or a false “GPU accelerated” claim.

The result is a conventional edge-AI pipeline: detect the animal first, crop a region of interest, and classify that crop locally.

2. Overall architecture

The production stack contains two FP32 models:

StageAndroid contract
DetectorRT-DETR-R18, 512×512 RGB, batch 1
Detector outputNMS-free raw [100, 9] rows for animal, person, vehicle
Detection thresholdFixed 0.10
ROI selectionHighest-score animal, with stable query-order tie breaking
ROI padding18% in source-image coordinates
ClassifierMobileNetV4-Conv-Medium, 320×320 RGB
Classifier output49 species logits, exposed as top-1 and top-3
Low-confidence stateClassifier top-1 below 0.55 is shown as low confidence
Super-resolutionDisabled in the first-launch stack
flowchart TD
    A[Camera or photo] --> B[Canonical RGB image]
    B --> C[512x512 RGB letterbox]
    C --> D[RT-DETR-R18 FP32]
    D --> E[Decode raw 100x9 rows]
    E --> F[Choose highest-score animal]
    F --> G[Inverse-map box]
    G --> H[Add 18% padded ROI]
    H --> I[320x320 RGB letterbox]
    I --> J[MobileNetV4 FP32]
    J --> K[Top-1 / top-3 result]

Both models share the same image rules: EXIF-normalized orientation, sRGB 8-bit RGB pixels, a bounded source long edge, aspect-fit letterboxing, and gray 128 padding. This is more important than it looks. A different rotation, color conversion, or padding value changes the tensor before the model has a chance to make a prediction.

3. Training and export pipeline

3.1 Data and training

Training starts with an auditable manifest. Included images must satisfy the project’s location, date, license, attribution, and normalized-taxon rules. The classifier label set is fixed at 49 species.

Training runs on a CUDA-enabled NVIDIA host. The production classifier is MobileNetV4-Conv-Medium trained with a PyTorch/timm wrapper. The interim detector is RT-DETR-R18. The Android resolver accepts the FP32 production classifier only; benchmark and research paths do not silently become release inputs.

3.2 Export and validation

The export path uses LiteRT-Torch to produce .tflite files. The export wrapper keeps the forward definition and image normalization explicit, then runs the following checks before an artifact can enter the Android production asset directory:

  1. Validate input/output shape and dtype.
  2. Load, allocate, and execute the LiteRT interpreter.
  3. Reject custom and Flex operators.
  4. Compare the exported output with the PyTorch reference.
  5. Write a manifest containing artifact SHA-256, checkpoint, converter/runtime versions, preprocessing, and output schema.
flowchart TD
    A[Licensed data manifests] --> B[PyTorch training]
    B --> C[Checkpoint]
    C --> D[LiteRT-Torch export]
    D --> E[Interpreter smoke test]
    E --> F[PyTorch/LiteRT parity]
    F --> G[SHA + schema manifest]
    G --> H[Production model assets]
    H --> I[Android model resolver]

The current release identities are:

ArtifactRoleSHA-256
knp_detector_rtdetr_r18_v1.tfliteRT-DETR-R18 FP32 detector3a43fdf973c03844f30ac088542da29ced0c565b828e0b203d74d1c096729d4a
knp_m1_coreml_v1_0_fp32.tfliteMobileNetV4 FP32 production baseline6fa1f47fbb2b6e9f133daae03ef4e644114160dbddfc45916181ffbf7a2f3251

Parity and model quality are separate checks. The classifier’s recorded maximum PyTorch/LiteRT error is 6.85e-06, within the 1e-4 conversion gate. The detector export also has a passing conversion/parity result on the fixed fixtures. A2 detector quality is still tracked separately and remains a blocked research status; a passing export does not turn into a claim that field recall is solved.

4. Android inference: the cascade in code

At runtime, the app performs the same sequence for a camera frame or a photo:

  1. Canonical RGB image.
  2. 512×512 letterbox.
  3. RT-DETR raw [100, 9] output.
  4. Decode boxes, scores, and labels.
  5. Choose the highest-score animal.
  6. Inverse-map the box and add an 18% padded crop.
  7. 320×320 letterbox.
  8. MobileNetV4 49-class logits.
  9. Top-1, top-3, and confidence state.

The detector stays NMS-free in the model. Android computes the application score as score = objectness * max(animal_score, person_score, vehicle_score).

The selected animal box is mapped back to the original image before padding. That keeps the classifier ROI independent of the detector’s 512×512 padded coordinate space.

If no animal candidate passes the fixed detector threshold, the pipeline returns a clear no-animal result. If the classifier score is weak, the UI keeps the top candidates and labels the result as low confidence instead of treating an uncertain species as confirmed.

5. Hardware acceleration: GPU first, CPU fallback

The Android app uses the standalone LiteRT runtime and GPU delegate libraries at the same version (1.4.2). Production requests GPU_FIRST, but the app does not assume that a device GPU can execute this particular graph.

The runtime performs a capability check, creates a real delegate as a probe, and then invokes the model. Only a successful invocation is recorded as GPU execution. If the device is unsupported, delegate creation fails, or inference throws, the same model runs through the FP32 CPU interpreter and records CPU_FALLBACK.

flowchart TD
    A[Input tensor] --> B{GPU compatibility check}
    B -- No --> C[FP32 CPU interpreter]
    B -- Yes --> D{Delegate probe}
    D -- Failed --> C
    D -- Passed --> E[GPU delegate + interpreter]
    E --> F{Invocation succeeds?}
    F -- No --> C
    F -- Yes --> G[Record GPU_FP16]
    C --> H[Record CPU_FALLBACK]

The model file remains FP32 on both paths. GPU_FP16 describes delegate execution; it is not a separately trained FP16 or INT8 artifact. The GPU delegate and interpreter are created and used on the same single-thread executor, which is required by the delegate’s thread-affinity contract.

There is no universal Android NPU switch. NPU execution requires a vendor-specific LiteRT delegate and a model/operator combination that the delegate supports. We therefore do not label a CPU run as NPU acceleration, and we do not use NNAPI as a generic MediaTek solution. The current hardware path follows the LiteRT GPU delegate pattern; vendor-specific NPU options are described in the LiteRT NPU documentation.

6. Memory and stability on real phones

The current detector artifact is about 81.8 MB and the classifier artifact is about 34.1 MB on disk. Native interpreter graphs, delegate allocations, image buffers, and tensor storage add to that footprint. This is why APK size and runtime memory are separate measurements.

The production runner avoids keeping both native model graphs alive during one identification:

sequenceDiagram
    participant P as Cascade pipeline
    participant D as Detector runtime
    participant C as Classifier runtime
    P->>D: map model and run
    D-->>P: raw detector rows
    P->>D: close interpreter, delegate, and mapping
    P->>C: map model and run ROI
    C-->>P: 49 logits
    P->>C: close interpreter, delegate, and mapping
    P-->>P: release image and ROI buffers

The implementation also:

  • memory-maps uncompressed model assets read-only;
  • reuses direct input/output buffers on the inference executor;
  • bounds decoded source images to a 2048-pixel long edge;
  • starts CPU fallback at two interpreter threads;
  • explicitly closes the interpreter, delegate, mapping, and executor;
  • keeps SR out of the required model stack.

The benchmark harness runs warmups and measured repetitions for both CPU and GPU delegates, recording P50/P95 latency, thermal status, native RSS, and finite-output status. A Pixel 4 XL is useful as a stress case; it should not be treated as proof that every Android device has the same memory or acceleration profile.

7. Trace the full path when a result looks wrong

Debug builds can trace six stages:

INPUT -> DETECTOR_RAW -> BBOX -> ROI -> CLASSIFIER_RAW -> RESULT.

The trace contract records hashes and structured metadata rather than relying on a screenshot alone:

  • source and canonical RGB hashes;
  • detector input hash and raw shape;
  • detector and classifier artifact hashes;
  • selected detection and source-space ROI;
  • classifier top-3;
  • backend and latency;
  • final result state.

Release builds do not write the original photo or complete logits to logs. They retain only aggregate backend and latency information. This lets us compare a Pixel, a MediaTek device, and a Samsung device without turning production logs into a copy of the user’s image.

8. What is supported today

The Android path is a real FP32 cascade with:

  • a fixed detector/classifier contract;
  • artifact SHA and schema validation;
  • local inference from Camera or Photos;
  • optional GPU execution based on an observed delegate probe;
  • deterministic FP32 CPU fallback;
  • explicit model lifecycle management for memory-constrained phones;
  • debug-only stage tracing and release-safe aggregate telemetry.

The MobileNetV4 FP32 classifier is the production baseline. RT-DETR-R18 is the current detector artifact used by the first-launch stack, while its independent quality gate remains a separate improvement track. The next gains must come from better detector quality or a verified hardware delegate—not from silently changing the preprocessing contract or claiming unsupported NPU execution.

Conclusion

The practical lesson from this deployment is simple: mobile inference is a system, not a model file. Training, export, preprocessing, delegate selection, memory lifecycle, and observability must agree on the same contract.

For Safari Go, that contract currently means RT-DETR-R18 FP32 followed by MobileNetV4 FP32, GPU when it is proven usable, CPU when it is not, and a local photo pipeline that can be inspected stage by stage.

Further reading