Propagate Access Controls into Vector Indexes

Tier: Advanced · Workflow area: RAG data pipelines

Access-control propagation is a data problem, and the data engineer owns it — source ACLs must travel with content into the index as filterable metadata, retrieval must enforce the caller's entitlements in the engine, and revocations must propagate on a measured SLA. This playbook builds and adversarially tests that path.

When to reach for this

• •
  Indexing any source with per-document or per-group permissions: SharePoint/Drive, Confluence spaces, HR or finance folders, ticket systems with private tickets.
• •
  A multi-tenant RAG platform (pb-31 tenancy) that now also needs within-tenant entitlements.
• •
  Security review asked "can user X retrieve document Y they can't open in the source system?" and the answer must be a tested no.
• •
  An entitlement revocation (offboarding, deal-team wall) must demonstrably remove index access, not just source access.

The workflow

1. 1.

   Extract ACLs at ingestion time, normalized to groups. Alongside content (pb-27), pull each document's permission principals from the source API and resolve users to durable group IDs where possible — group-based tags survive personnel churn; user-ID tags rot. Land them in the pb-31 schema's acl_tags:

   Copy

   -- pb-31 schema already has: acl_tags TEXT[] NOT NULL DEFAULT '{}'
   UPDATE rag_chunks SET acl_tags = ARRAY['grp:finance', 'grp:exec']
   WHERE doc_id = 'finance/comp-memo-2026.pdf';
   

   Default-deny: a document whose ACL extraction fails gets acl_tags = ['quarantine'] and retrieves for no one — never an empty-means-public convention.

2. 2.

   Enforce in the engine, not the application. Every retrieval carries the caller's entitlements and the filter is applied inside the store, on both hybrid legs (pb-33). With pgvector, row-level security makes the database itself the enforcement point:

   Copy

   ALTER TABLE rag_chunks ENABLE ROW LEVEL SECURITY;
   CREATE POLICY chunk_acl ON rag_chunks USING (
     tenant_id = current_setting('app.tenant_id')
     AND acl_tags && string_to_array(current_setting('app.caller_groups'), ',')
   );
   

   Copy

   def search(q, caller, k=10):
       with pool.connection() as con:
           con.execute("SET app.tenant_id = %s", (caller.tenant_id,))
           con.execute("SET app.caller_groups = %s", (",".join(caller.groups),))
           return hybrid_search(con, q, k)   # RLS applies to both legs automatically
   

   Dedicated-engine equivalents: Qdrant payload filters composed server-side into every query, Pinecone metadata filters per namespace, OpenSearch document-level security. The invariant is the same everywhere: no code path can reach vectors without the filter attached.

3. 3.

   Propagate ACL changes through the sync pipeline. Permission changes arrive without content changes, so pb-36's content-hash diffing will skip them — add an ACL hash to change detection: classify() returns acl_changed when permission sets differ, triggering a metadata-only update (no re-embed, no cost). Revocations are deletes' sibling: dead-letter loudly on failure and track propagation lag as its own SLA metric.

4. 4.

   Build the leakage test suite — the metric layer. Assertions, run as CI for the retrieval service and on a schedule in production:

   • •
     Negative retrieval: for N (user, forbidden-document) pairs, paraphrase-heavy queries targeting each document's distinctive content return zero chunks of it in the top 50. The proving metric: leak rate = 0 over ≥100 adversarial query-user pairs; a single hit is a release blocker.
   • •
     Positive control: the same queries issued by an entitled user do retrieve the document (recall ≥ launch baseline) — otherwise "secure" may just mean "broken".
   • •
     Revocation SLA: remove a test user's group at the source, measure minutes until their canary query for previously-visible content returns empty. Expected output: a number (e.g. "p95 revocation propagation: 11 min"), stated in the runbook.
5. 5.

   Cover the side channels. Retrieval is not the only exit: cached answers generated for a privileged user must not serve to others (scope any answer/semantic cache by entitlement hash); logs and traces (pb-37) that store retrieved chunks inherit document sensitivity — restrict the observability store accordingly; and "no results" versus "you lack access" responses should be indistinguishable to avoid confirming a document's existence.

Verify before you trust it

• Leakage suite green0 leaks across ≥100 adversarial pairs including the nastiest cases — documents whose content is similar to broadly-readable ones, and users with partial (one-of-two-groups) entitlements.
• Positive controls greenentitled-user recall within a point of the unfiltered baseline, per pb-32's filtered-recall measurement.
• Revocation drilltimed and documented; quarantine default verified by ingesting a document with a deliberately broken ACL fetch and confirming it retrieves for no one.
• Bypass auditgrep the codebase for any query path touching the vector store outside the shared entitlement-enforcing function — including admin scripts, evals, and the pb-37 canary runner (which must run with a defined entitlement context).

Pitfalls

• Snapshot ACLs. Permissions copied at first index and never refreshed drift from the source within weeks; the index becomes a permanent record of who used to have access. ACL sync is continuous (step 3), not one-time.
• Filtering in application code per caller. Twelve call sites each remembering to add the filter is eleven too many opportunities; one forgotten path in a batch eval script is a breach. Engine-level enforcement or a single choke-point function.
• User-ID-based tags. Team changes then require re-tagging millions of chunks. Tag with groups; resolve membership at query time from the identity provider.
• Testing only the happy path. "Alice can see her docs" proves nothing. The suite's value is adversarial negatives and revocation timing.
• Forgetting derived artifacts. Contextual enrichment text (pb-35), sparse indexes (pb-33), caches, and traces are all copies of controlled content; each inherits the ACL obligations of its source.

Tools

• pgvector + row-level securityenforcement in the database; the strongest default for the lab and for Postgres-based production.
• Pinecone namespaces / Qdrant payload filtersdedicated-engine patterns: hard namespace isolation for tenancy plus server-side composed filters for entitlements.
• OpenSearch document-level securitymature DLS model when retrieval runs on an OpenSearch/Elastic estate.

Client-safe variant

For client work, ACL propagation is usually the difference between a demo and a deployable system — so scope it explicitly in the SOW, and insist on integrating with the client's real identity provider (their groups, their revocation events) rather than a parallel permission table that will drift. Run the leakage suite on a synthetic corpus with synthetic users first and demo it to the client's security team; then re-run inside their environment against real ACLs before go-live, delivering the leak-rate result, the revocation SLA number, and the quarantine-default behavior as written evidence. Never accept "index only public documents" as a permanent posture without a tested enforcement mechanism — corpora drift, and today's public folder acquires tomorrow's confidential file.