On-device embedding models and runtimes

You need a small embedding model that runs locally.

Candidates: all-MiniLM-L6-v2 (~22M params, 384-dim) bge-small gte-small a quantized biomedical model Runtimes: ONNX Runtime — cross-platform ExecuTorch — PyTorch's on-device runtime Core ML — Apple Neural Engine TensorFlow Lite / LiteRT — Android NNAPI/GPU The model + tokenizer are bundled in the app so embedding needs no network.

Measure cold-start load time and per-query embed latency on a mid-tier device — that's the number an interviewer will ask for.