Private RAG is a strange product category. Almost every team can stand one up in a week. Almost no team can keep one healthy for a year. The teams that succeed usually pick the same boring defaults early, before anyone has had a chance to be clever.
One index, many sources
Keep a single index. Tag chunks with a source identifier. Do not let “we will have a separate index for Slack” turn into a separate retrieval pipeline. The cost of a parallel pipeline is paid every time anyone tries to debug retrieval, which is more often than you think.
Boring chunk strategy
Token-window chunking with overlap, capped at the model context budget, with one small twist: keep a heading trail and the source path in metadata. When retrieval surfaces a chunk, the heading trail is what the model needs to feel anchored. The fancier strategies tend to make retrieval look smarter on a benchmark and worse in the wild.
Visible recall, not just answers
Every answer should be reproducible. That means the model never speaks without surfacing the chunks it leaned on. If a chunk cannot be displayed back to a user, it should not be in the index. This is what kills the “we cannot tell why it said that” failure mode that quietly poisons trust.
Reindex like you mean it
Reindexing should be a single command. Not a manual playbook. Not a notebook a person has to walk through. If a reindex is painful, it will not happen, and within a quarter the index quietly disagrees with reality.
Local-first when you can get away with it
On laptops, local embeddings and an on-device retriever are usually good enough and remove a whole class of privacy conversations. Only escalate to a remote model when the task earns it, and log the escalation so you can see whether it was actually worth it.
None of this is impressive in a demo. All of it is what separates a maintained system from a project that quietly stops being used.