You need to know if the system works in the field without exfiltrating sensitive data.
Log metadata only: Retrieval scores Which chunk ids were used Latency Model/version Abstain rate User thumbs-up/down Never the query text, snippet text, or any PHI.
Keep logs on-device by default; if you aggregate, aggregate non-identifying counters. This is both a privacy requirement and a genuinely useful eval signal (rising abstain rate or falling thumbs-up flags a retrieval regression).