Quantization stores weights at lower precision (fp16, int8, int4) to shrink the model and speed inference, at a small accuracy cost. An fp32 MiniLM is ~90MB; int8 can be ~25MB. This is what makes on-device feasible.
For embeddings, int8 usually preserves retrieval quality well.
Also consider dimension: a 384-dim model produces a smaller index than a 768-dim one — meaningful when you store thousands of vectors on a phone.