RAG over SharePoint, Confluence, and Google Drive: Permissions-First Architecture

The hard problem in enterprise RAG is permissions: the assistant must never surface a document to someone who is not allowed to see it. An embedding does not carry or enforce the source authorization by itself, so a naive "embed everything, retrieve top-k" pipeline matches on similarity, not access. Enforce authorization before generation: attach source-native access context to every chunk, or maintain an authoritative lookup, and apply a mandatory filter or live authorization decision before any result reaches the model. Preserve each source's rules for direct users, groups, domains, links, inheritance, and protection instead of reducing them to a universal group-only or deny-wins model.

This is the implementation companion to our RAG production-readiness checklist, which names per-user permissions as the hard security problem. This post is how you actually solve it. Vendor-specific and preview features move fast; re-check the linked docs before you build.

Why permissions are the hard part

The failure mode is simple: the model composes a fluent answer from a document the asking user was never allowed to see. Chunking and embedding do not automatically copy the source system's effective authorization into a shared index. A vector store only knows the caller and their entitlements if the retrieval system supplies and enforces them.

Microsoft says Microsoft 365 Copilot only accesses data the signed-in user is authorized to access and respects SharePoint controls and sensitivity-label encryption. It can nevertheless make existing oversharing easier to discover. Microsoft therefore offers governance controls such as Restricted Content Discovery, a temporary SharePoint control with documented scope limits, and Data Access Governance reports. These controls reduce discovery or help remediate permissions; they do not replace correct source permissions. (1, 2)

Telling the model not to reveal a document is not access control. The current OWASP LLM Top 10 lists prompt injection as LLM01, and NIST documents indirect prompt injection arriving through retrieved content. Authorization must be enforced by trusted application or data-layer logic before untrusted content reaches the model. (3)

The principle: enforce at the retrieval layer

The model must never process an unauthorized chunk. Prefer authorization-aware pre-filtering inside a trusted retrieval service. A two-stage design can over-fetch candidates and authorize them before they leave that service, but it must continue searching until it has enough authorized results and must never place rejected content in model context, logs, or client responses. Filtering in the prompt is not enforcement.

Per-source permission models

Each source has different principals, inheritance, sharing links, and protection controls. Use a dedicated connector identity with the least privileges and repository scope needed for the approved corpus. Normalize fields for retrieval efficiency, but retain enough source-native facts to reproduce the source's effective authorization.

For Microsoft Graph, code against grantedToV2 and grantedToIdentitiesV2; the older fields are deprecated. Choose the least-privileged delegated or application permissions that fit the connector, and prefer resource-scoped approaches such as selected-site access over tenant-wide read access when feasible. (4)

Implementation patterns that hold up

• Store stable source-native principals. Groups reduce cardinality, but direct user grants, domain and anyone links, inherited restrictions, and protected-content state can also decide access. Resolve nested memberships where the source supports them.
• Bind every query to the signed-in subject. Use immutable subject and tenant identifiers, then resolve effective entitlements or ask the source. Microsoft's OAuth on-behalf-of flow carries a user's delegated permissions, not application roles; Google and Atlassian require their own delegation or authorization designs. (8)
• Make authorization mandatory in retrieval. Use set-membership filters where the source semantics are additive, and explicit checks for ancestor restrictions, limited access, links, and encryption. There is no safe universal "deny overrides grant" rule across these products.
• Handle Entra group overage. Microsoft documents a 200-group limit for JWT group claims and 150 for SAML. Above the limit, the token omits the group list and supplies an overage indicator, so the application must obtain the full membership set before authorizing. (7)
• PostgreSQL row-level security can enforce the predicate near pgvector data, but it is not impossible to bypass. Superusers and roles with BYPASSRLS always bypass it; table owners normally do too unless FORCE ROW LEVEL SECURITY applies. Keep application roles unprivileged, test policy composition, and set request identity transaction-locally in pooled connections. (9)

There is a real architecture trade-off. A synchronized authorization index gives predictable retrieval latency but creates a revocation window. A live source check offers fresher decisions but adds dependency latency and availability risk. Many systems combine both: pre-filter with synchronized metadata, re-check sensitive results live, and fail closed whenever authorization state is stale or unavailable.

The freshness problem nobody budgets for

When permissions are materialized at ingest, a source-side revocation is not automatically reflected in the index. Change feeds are useful signals, not proof that every descendant's effective access was recomputed. Google documents that ACL changes appear in a user's change log only for owners, service accounts on the ACL, or directly affected users; shared-drive connectors should use the shared-drive change log where applicable. Combine incremental events with periodic reconciliation of ancestor-dependent authorization and a fail-closed freshness threshold. (10)

Deletion is the other half. Where a GDPR erasure obligation applies, and derived chunks, embeddings, caches, or logs remain personal data, delete or anonymize those artifacts subject to the regulation's grounds and exceptions. Stable document-to-chunk lineage makes this practical. Embeddings are not categorically personal data, but they can be when they relate to an identifiable person. One published attack recovered 92% of 32-token inputs exactly in a specific white-box setup against two embedding models; that result shows a real confidentiality risk, not a universal recovery rate. (11, 12)

GDPR and residency

GDPR data minimization and storage limitation support limiting the indexed corpus and retaining derived artifacts only as long as needed. Retrieval audit logs can support accountability and incident investigation, but the logs themselves may contain personal data, so minimize, protect, and expire them. If a model provider acts as a processor, Article 28 requires appropriate contractual terms; Chapter V governs transfers to third countries. An EU inference endpoint alone does not prove that all processing, storage, support access, and subprocessors stay in the EU, so verify the full data flow and contract. See EU data residency for AI apps in 2026. (11)

The reference architecture, and the anti-patterns

End to end: scope a least-privileged connector to the approved repositories; capture each item's source-native principals, inheritance, links, and protection state; attach a versioned authorization reference or filterable projection to every chunk; bind retrieval to the signed-in subject; authorize before model context; log a minimized decision record; reconcile change feeds with periodic effective-access sweeps; and cascade applicable deletions through derived artifacts.

Anti-patterns include a shared index with no mandatory authorization; prompt-only filtering; querying as a broad service account; storing groups while dropping direct users or links; flattening all products into one deny/grant rule; treating a direct Confluence restriction response as effective access; assuming every Drive child only gains access; ignoring SharePoint label encryption; accepting stale authorization indefinitely; and deleting a source while retaining derived personal data without a lawful reason.

"The model is not your access control, and the prompt is not your security boundary. If a user is not allowed to read a document, the vector store must never return it. Everything else, the labels, the freshness, the audit log, is in service of that one rule."