How to Build a Private AI Knowledge Base for Your Company Using RAG

Company knowledge is scattered across documents, tickets, meeting notes, and internal tools. A language model can make that material easier to search, but it cannot repair missing permissions or stale source data. Retrieval-augmented generation (RAG) retrieves relevant source material and supplies it to a model so the answer can be grounded in that evidence. It is a system design, not a security feature.

The first design decision is often whether to configure a managed search product, build a custom RAG pipeline, or combine both. That choice depends on data locations, access rules, freshness needs, and how much control the product requires.

Choose the level of control you need

Start by testing a managed option against representative questions and access scenarios. A custom stack is justified when the test reveals a product requirement the managed system cannot meet, not merely because embeddings are familiar.

When comparing managed products, test the connector against the source systems your organization actually uses. Verify whether it honors user-level permissions, how it handles shared links and inherited access, what happens after an account is disabled, and whether administrators can see indexing and deletion status. Read the provider's data-processing and retention terms for both source data and prompts. A connector demo that retrieves the right paragraph is not a complete access review.

For a hybrid design, assign clear ownership. One system may manage connectors and document synchronization while a product service applies domain-specific authorization, retrieval rules, and response behavior. Decide where canonical metadata lives, how failures are surfaced, and which system owns deletion. If both systems independently index the same content, add a reconciliation strategy before operators have to debug mismatched versions.

Design the ingestion path as a data product

Ingestion should preserve the source identity, workspace or tenant, owner, access labels, last-modified time, and content hash. Parse each source into useful sections; chunking should preserve enough structure to understand headings, tables, and document boundaries. Very small chunks lose context, while very large chunks can crowd out the relevant passage and increase inference cost.

Store the original document reference separately from search representations. A practical pipeline is:

Approved source connectors
        │ content + identity + ACL metadata
        ▼
parse / normalize / chunk / deduplicate
        │ document versions + source references
        ▼
lexical index + embedding index
        │
Authenticated question ──► resolve workspace + user grants
        │                              │
        └──────────────► filter candidates by ACL first
                                       │
                              hybrid retrieval / rerank
                                       │ evidence IDs + excerpts
                                       ▼
                         generate cited answer or abstain
                                       │
                              feedback + eval trace

An embedding index helps find semantic similarity; BM25 or full-text search helps with exact phrases, names, and identifiers. Hybrid retrieval combines these candidate sets before reranking. The best mix depends on the corpus and questions, so evaluate it rather than assuming vector search wins.

The chunker should preserve context that changes meaning: heading hierarchy, table headers, document version, and nearby definitions. Attach metadata such as source ID, workspace, ACL groups, content hash, and modification time to every chunk. Avoid indexing boilerplate repeatedly; deduplication can reduce noise and ingestion cost. Keep an index version so a parser or embedding-model change can be rolled out and compared deliberately.

For ranking, begin with a simple baseline: exact and lexical search for terms and identifiers, semantic candidates for paraphrase, then merge and optionally rerank a bounded set. A reranker can improve relevance but adds latency and another model dependency. Measure retrieval recall and source diversity against labeled questions first. Add complexity when it addresses a visible miss pattern.

Authorization belongs before answer generation

Resolve the caller's identity and allowed sources in trusted server code. Apply workspace, document, and, when needed, section-level permissions to candidates before they reach the model. Avoid asking the model to filter unauthorized material after retrieval. A cross-tenant filter bug can expose data even if a database is described as private.

For a database-backed implementation, enforce authorization in the query layer and consider database policies as defense in depth. For example, this illustrative query only searches evidence rows for a server-resolved workspace; a production system also needs document-level grants and tested membership checks:

SELECT id, source_id, excerpt, modified_at
FROM document_chunks
WHERE workspace_id = :authenticated_workspace_id
  AND deleted_at IS NULL
  AND search_vector @@ websearch_to_tsquery('english', :question)
ORDER BY ts_rank(search_vector, websearch_to_tsquery('english', :question)) DESC
LIMIT :bounded_limit;

The authenticated_workspace_id must come from verified session state, never from a model argument. Supabase's row-level security guidance also makes clear that policies work alongside grants, and privileged keys bypass RLS. A vector database does not itself provide tenancy or privacy.

Permission filtering should be applied before ranking results are passed to generation. Depending on the search engine, metadata filters may be part of the query, or the application may need to retrieve candidate IDs and validate them against an authorization service. The latter can leak data through timing, logs, or accidental prompt construction if not carefully implemented. Tests should assert not only that a user cannot read a forbidden final answer, but that restricted text never enters model context, traces, caches, or user-visible citations.

Make citations and abstention part of the interface

Return source title, location, and last-updated time with each retrieved passage. Ask the model to cite the evidence IDs it used, then validate that every returned citation refers to an allowed retrieved item. Provide a useful “I could not find this” response when evidence is weak or conflicting. A plausible answer without traceable support should not be presented as company policy.

Keep generated answers separate from canonical records. If a user wants to create a task or update a policy, create a proposed change and send it through the appropriate review process. The context-management article discusses why retrieval is a hint system while canonical business state remains the source of truth.

Freshness and deletion are first-class requirements

Use incremental indexing where sources provide change events or reliable modification timestamps. Store document versions and content hashes to avoid unnecessary re-embedding. Plan a reconciliation job for missed events. On deletion or access revocation, remove or invalidate indexed chunks and ensure caches and derived summaries follow the same policy. Define how quickly those changes must take effect and test that requirement.

Data lifecycle also includes backups, logs, provider retention, support access, and export/deletion requests. Review contracts and deployment regions. Encrypt transport and storage where applicable, limit credentials, and prevent document text from leaking into unrestricted traces. “Private” should describe a set of verified data and access controls, not the name of the database.

Deletion needs an end-to-end contract. When a source is removed, mark it unavailable to retrieval immediately, then remove chunks and embeddings asynchronously with a retryable deletion job. Ensure caches, summaries, replicas, and evaluation fixtures do not reintroduce content. Keep a tombstone or deletion ledger where needed to prevent a delayed connector event from indexing the deleted version again. Retention and backup behavior depends on policy and vendor terms, so communicate it to administrators clearly.

For permissions inherited from folders or groups, capture enough source metadata to reproduce the effective access decision. If permissions change frequently, plan how to synchronize grants and how stale a grant is allowed to be. Query-time authorization against the source system can offer freshness but adds latency and availability dependencies; precomputed ACL metadata can be faster but must stay current. Test both approaches with revocation cases.

Evaluate retrieval separately from answers

Create questions with expected source passages and access context. Measure whether the correct document appears in top-k results, whether unauthorized documents are excluded, whether citations point to supporting passages, and whether answers abstain when evidence is absent. Slice results by source type, language, document age, and permission class.

The ProfMe repository describes a local synthetic retrieval report: hybrid Recall@5 and Recall@10 were 1.0 over nine cases; top-1 citation correctness was 7/9; answer faithfulness was not measured. That is evidence about a small fixture and configuration, not production accuracy. The ProfMe case study records those limits alongside the implementation details.

In operation, monitor indexing lag, failed connectors, deleted-document propagation, retrieval latency, model errors, and user corrections. A good answer may depend on an upstream document that changed; preserve enough version information to investigate it.

Include adversarial and operational cases in the evaluation set: a user asks for a restricted policy, two documents disagree, a relevant file is deleted, a document is updated between indexing and query time, or the query uses an internal abbreviation. For each case, define expected allowed source IDs, not only a preferred answer sentence. This distinguishes a retrieval miss from an answer-generation problem.

Use user feedback carefully. A thumbs-up is a signal, not proof that the cited policy was correct. Collect structured correction reasons such as “wrong source,” “stale source,” “missing access,” or “answer overstates evidence.” Use those reports to update labeled cases and route fixes to ingestion, permissions, ranking, or answer generation.

What should an initial implementation include?

For a controlled pilot, choose one or two source types, a defined user group, explicit access rules, source citations, and a small evaluation set. Add the minimum connector health and document lifecycle controls needed for those sources. Defer broad connector coverage and complex reranking until real questions show where the basic path fails.

The main effort often sits outside the model call: source API behavior, parsing edge cases, permissions, indexing freshness, deletion, evaluation data, and support. Costs vary with source volume, ingestion frequency, embedding and generation usage, storage, connector licensing, and the review burden. Measure those drivers with a representative pilot rather than quoting a generic “RAG price.”

An implementation estimate should separate connector setup, parsing and normalization, access mapping, index design, retrieval evaluation, answer UI, admin tools, deployment, and ongoing source maintenance. Include any required security or compliance review and the time for subject-matter experts to label questions and inspect results. These contributors often determine feasibility more than the choice between two embedding models.

Treat the knowledge base as an evolving product. Assign an owner for connector credentials, source quality, access changes, user feedback, and incidents. Plan a response when a source becomes unavailable or its format changes. A useful first launch can be narrow and still require thoughtful maintenance; broad ingestion is not a substitute for clear ownership.

Plan a knowledge-system discussion

For a pilot, include an administrator view of the source inventory and last successful sync. Give operators a way to retry a failed document, inspect why a source is unavailable, and confirm that a deletion request was processed. Keep user-facing “as of” information close to an answer when freshness matters. These controls make the system supportable without exposing internal ingestion details to every user.

Review how model-provider requests are handled. Check whether prompts and retrieved passages are retained, which region processes them, whether provider training or abuse-monitoring terms apply, and which team can access traces. This requires reading the terms for the actual provider and plan. A self-hosted vector index does not answer these questions because generation may still send text to an external service.

Document these decisions for administrators; data location and retention are part of the system's operating contract.

List the data sources, who may read each source, how often the content changes, what questions users ask, what an answer must cite, and any retention or residency constraints. Existing identity and document systems matter too. If these details are incomplete, a short inventory is still useful.

Further reading

• Context management for AI products
• How AI evaluations can separate retrieval and answer quality
• ProfMe retrieval and memory implementation
• PostgreSQL row security policies
• Supabase row-level security