Three options, each with a privacy story.
(a) Fully on-device LLM (e.g., a small Llama/Phi/Gemma via llama.cpp/MLC) —
maximum privacy, weakest quality, heaviest battery.
(b) Hosted LLM API, sending only the query + retrieved snippets — strong
quality, but those snippets leave the device, so they must be
non-identifying or contractually protected (BAA).
(c) Your own backend / a local bridge that proxies to an LLM — lets you enforce
the privacy boundary and audit logging in one place.
The honest answer in an interview: retrieval/embeddings on-device for privacy; generation wherever quality+compliance allow, sending the minimum necessary context.