This module installs the operator's view of a vector index: a downstream consumer of change data capture — synced incrementally, watched by canaries, filtered inside the store from verified identity, and isolated per tenant, because a stale or leaky index fails fluently.
The Friday the index lied. Meena owns the production RAG platform behind an insurance client's policy assistant — the M5–M6 pipeline, now live.
A support lead escalates. The assistant just quoted a benefits clause the company retired three weeks ago, confidently and with a citation.
The document was corrected in the warehouse 21 days ago. THE INDEX NEVER GOT THE MEMO | The nightly full-rebuild job had been silently failing since a parser bump, and nobody noticed because retrieval kept returning plausible results the whole time.
Worse: a former employee's record, subject to an erasure request honored in the warehouse last month, is still retrievable and quotable.
Both silent, both fluent — a stale index and an un-propagated delete, each returning yesterday's reality stated as today's fact.
The Monday it couldn't happen again. Meena and Divya, the platform lead, rebuild the platform as an operated system instead of a pile of vectors. Incremental CDC sync with a sync ledger replaces the all-or-nothing rebuild; a sentinel freshness probe pages if end-to-end index lag exceeds a 15-minute SLA; delete propagation is wired to erasure and retention events; ACL filtering is enforced inside the store via row-level security from verified identity; and per-tenant freshness SLAs go in ahead of the Q3 enterprise onboarding Divya has already sold. When the next document is retired, its chunks leave retrieval within the SLA; when an erasure lands, a leakage test proves the chunk is gone. Same corpus, same model — the difference is that the index became a downstream consumer with SLAs, canaries, and a kill switch.
M5 taught you to turn messy documents into measured chunks and embeddings; M6 taught you to retrieve from them well. Both modules quietly assumed a corpus that holds still and a reader who is allowed to see everything. Production revokes both assumptions on day one. Source systems change continuously — edits, new documents, hard deletes, permission changes — and the index either tracks them or drifts into confident staleness. Documents carry entitlements, and a retrieval layer that ignores them is a privilege-escalation engine with a chat interface. And the moment your platform serves more than one team or client, tenancy stops being someone else's problem.
The good news is that almost nothing here is exotic. The vector store is a downstream consumer of your CDC stream — the same discipline you apply to any warehouse sink applies here: idempotent upserts, delete propagation, tombstones, blue-green cutover, freshness SLAs, row-level security. What's new is the failure modes. A stale warehouse table returns yesterday's numbers; a stale index returns yesterday's policy stated as today's fact, fluently. A permissions bug in a dashboard shows a wrong chart; a permissions bug in retrieval reads the restricted document out loud to whoever asked a nearby question. This module is the operations layer that keeps a RAG data platform fresh, observed, sealed, and isolated.
The vector store is just another downstream consumer of change data capture — but its failure modes are worse than a stale table, because a stale or leaky index fails fluently. Sync it incrementally, observe it with canaries, filter it at the store, and isolate it per tenant.
The naive production pattern is the nightly full rebuild: drop the collection, re-parse, re-chunk, re-embed everything. It survives demos and dies at scale for three separate reasons:
Incremental sync treats the index the way you already treat every other sink: detect what changed, transform only that, write idempotently, propagate deletes. The one piece of state you need is a sync ledger:
CREATE TABLE rag_sync_state (
doc_id TEXT PRIMARY KEY,
source_hash TEXT NOT NULL, -- hash of normalized content
acl_hash TEXT NOT NULL, -- hash of the permission set, tracked separately
chunk_count INT NOT NULL,
pipeline_version TEXT NOT NULL, -- embedding model + chunking config that produced the vectors
indexed_at TIMESTAMPTZ NOT NULL,
deleted_at TIMESTAMPTZ -- tombstone; NULL while live
);
Change detection is a comparison against that ledger. With a CDC stream (Debezium, warehouse streams) you consume insert/update/delete events directly; without one, a watermark scan plus hash comparison gets you the same classification:
WITH current AS (
SELECT doc_id,
md5(coalesce(title, '') || chr(1) || body) AS source_hash,
md5(array_to_string(allowed_groups, ',')) AS acl_hash
FROM source_documents
WHERE updated_at > :last_watermark
)
SELECT c.doc_id,
CASE WHEN s.doc_id IS NULL THEN 'insert'
WHEN c.source_hash <> s.source_hash THEN 'content_change'
WHEN c.acl_hash <> s.acl_hash THEN 'acl_change'
END AS change_type
FROM current c
LEFT JOIN rag_sync_state s USING (doc_id)
WHERE s.doc_id IS NULL
OR c.source_hash <> s.source_hash
OR c.acl_hash <> s.acl_hash;
Splitting acl_hash from source_hash is deliberate: a permission change must update chunk metadata immediately but requires no re-embedding — collapsing both into one hash makes every ACL edit pay the embedding bill and, worse, makes teams batch ACL updates to save cost. Never put a revocation behind a cost decision.
The sync task itself diffs at chunk granularity, so a one-paragraph edit re-embeds one chunk, not a document:
def sync_document(doc, vector_store, embed_batch):
chunks = chunk(parse(doc)) # the M5 pipeline, pinned to pipeline_version
new = {chunk_id(doc, c): c for c in chunks} # chunk_id = f"{doc.id}:{sha1(c.text)[:16]}"
old = set(vector_store.list_ids(prefix=f"{doc.id}:"))
to_embed = [c for cid, c in new.items() if cid not in old]
to_delete = [cid for cid in old if cid not in new]
if to_embed:
vectors = embed_batch([c.text for c in to_embed]) # M5 batch pipeline: retries, DLQ, cost log
vector_store.upsert([
(chunk_id(doc, c), v,
{**c.metadata, "doc_id": doc.id,
"allowed_groups": doc.allowed_groups,
"tenant_id": doc.tenant_id,
"indexed_at": utcnow_iso()})
for c, v in zip(to_embed, vectors)
])
if to_delete:
vector_store.delete(to_delete)
update_sync_state(doc, chunks)
Content-addressed chunk IDs (doc_id plus a hash of the chunk text) make the upsert idempotent and the diff trivial. One caveat worth knowing before it bites: recursive splitters can cascade — an edit near a chunk boundary shifts every boundary after it, turning a small edit into a large re-embed. Structure-aware and parent-child chunking (M5) bound the blast radius, which is one more argument for them beyond retrieval quality.
Hash normalized content only — strip volatile fields like updated_at, view counts, and rendering artifacts before hashing, or every no-op touch in the source system triggers a re-embed and your "incremental" pipeline quietly becomes the nightly rebuild you were avoiding.
Deletes deserve their own paragraph because they are the part nobody's demo handles. Deletions arrive from three directions — hard deletes in the source (CDC delete events, or an anti-join against a snapshot if you're batch-scanning), retention policy expiry, and erasure requests under DPDP or GDPR. All three must propagate to the vector index, the sync ledger, any retrieval caches, and your eval fixtures. An index that only ever receives upserts never forgets: the deleted document remains retrievable — and quotable by the LLM — indefinitely.
Delete propagation is a compliance control, not an optimization — an erasure request that is honored in the warehouse but not in the vector index has not been honored, and the failure is silent until someone's chatbot quotes the deleted record back to them.
Incremental sync assumes the pipeline configuration is constant. The moment you change the embedding model, the chunking strategy, or the parser, that assumption breaks — vectors from different embedding models live in different spaces and cannot be mixed in one collection, and mixed chunking regimes make retrieval behavior unexplainable. These changes require a full rebuild, and the discipline for a full rebuild is blue-green cutover, the same shape as a warehouse migration:
kb_v8__bge_m3__parent_child_512), and record pipeline_version in the sync ledger so a half-migrated state is detectable, not mysterious.Full re-embeds are the expensive operation in this module, which is why they're scheduled and batched rather than accidental: run them off-peak through your provider's batch tier — batch endpoints typically price at a substantial discount to synchronous calls (Anthropic's Message Batches API, for example, processes at 50% of standard prices) — and track cost per rebuild in the same cost log the M5 embedding pipeline already writes.
When serving reads go through an alias, promotion and rollback are the same operation — a pointer switch. If rolling back a bad index requires a rebuild, you don't have a rollback, you have an outage with homework.
An unobserved sync pipeline fails in the worst possible way: retrieval keeps returning results, answers keep sounding plausible, and nobody knows the index stopped tracking reality three days ago. The observability layer has two halves — pipeline metrics you compute from the sync ledger, and canary queries that prove the end-to-end path.
The metrics, all derivable from rag_sync_state plus your DLQ:
| Metric | Definition | What it catches |
|---|---|---|
| Freshness lag | p95 of indexed_at − source updated_at over recent changes | Sync falling behind, silently or otherwise |
| Sync backlog | Documents classified as changed but not yet indexed | Throughput mismatch, stuck workers |
| Embedding failure rate / DLQ depth | Failed embedding calls awaiting retry | Provider issues, rate limits, poison documents |
| Chunk-count drift | Total chunks vs. trailing baseline | Parser regressions (a sudden drop means a parser started returning empty extractions) |
| Delete debt | Tombstoned docs whose chunks still exist in the index | The compliance failure from the previous section |
These are ordinary data-observability signals — put them on the same stack you already run (Airflow SLAs and sensors on the sync DAG, Elementary or Great Expectations monitors on the sync-state table) rather than inventing a parallel one.
Canary retrieval queries are the piece with no warehouse analogue. A canary is a fixed query with a known expected result, run against the production index on a schedule:
CANARIES = [
{"query": "What is the parental leave policy?", "must_hit": {"hr-policy-042"}},
{"query": "Standard indemnification clause for MSAs", "must_hit": {"legal-msa-007"}},
# 20-50 of these, stratified across document types and tenants
]
def run_canaries(retriever, top_k=10):
failures = [c for c in CANARIES
if not c["must_hit"] <= {r.doc_id for r in retriever.search(c["query"], k=top_k)}]
return failures
def freshness_probe(retriever, sla_minutes=15):
token = f"probe-{utcnow_iso()}"
update_sentinel_document(token) # write a timestamped change to a dedicated doc
wait(minutes=sla_minutes)
hits = retriever.search(token, k=5) # the probe token must be retrievable within SLA
assert any(token in r.text for r in hits), "freshness SLA breached end-to-end"
The sentinel probe is the highest-value fifteen lines in this module: it exercises the entire path — change detection, chunking, embedding, upsert, index visibility — and fails if any stage silently stalls. Canary failures page the pipeline owner the way a freshness SLA breach on a gold table would.
Canaries and the evaluation harness answer different questions — canaries are production smoke tests that detect regressions cheaply and continuously; the RAGAS harness (M9) measures quality and gates changes in CI. Teams that run only the harness find out about production staleness from users; teams that run only canaries ship quality regressions that canaries were never designed to see.
Query-time observability — traces of retrievals, rerank scores, and generation — belongs to Langfuse or Phoenix and is largely the application team's half; your data-side metrics join to it through document and chunk IDs, which is one more reason chunk IDs are deterministic.
This is the security-critical section of the track. Retrieval collapses N permissioned source systems into one index, and unless entitlements travel with the content, the collapse erases them. The failure is not hypothetical or subtle: a chatbot with an unfiltered index will summarize the executive compensation file for any employee whose question happens to embed near it. Security postings now name "securing RAG against leakage" explicitly, and the data side of that job — this section — is yours.
Three rules, in order of importance:
Carry entitlements on every chunk. At ingestion, stamp each chunk's metadata with the access-control facts: tenant_id, allowed_groups (or roles), and a sensitivity classification. Resolve entitlements to groups, not individual users, wherever the source system allows — group membership changes constantly, and resolving group→user at query time means membership changes take effect instantly without touching the index. This is also why the sync ledger tracks acl_hash separately: a permission change is a metadata-only update with its own propagation SLA, and revocation latency — time from revoke in the source system to the filter taking effect — is a metric you measure and report like freshness.
Filter inside the store, before ranking. The entitlement filter must be applied server-side as part of the similarity search — a metadata pre-filter in pgvector/Qdrant/Pinecone terms, document-level security in OpenSearch terms — constructed by a trusted service from a verified identity (the JWT's claims, never anything the user or the model supplied). In Postgres, row-level security makes the database itself the enforcement point, which means no retrieval code path can forget it:
CREATE TABLE kb_chunks (
chunk_id TEXT PRIMARY KEY,
doc_id TEXT NOT NULL,
tenant_id TEXT NOT NULL,
allowed_groups TEXT[] NOT NULL,
body TEXT NOT NULL,
embedding vector(1024) NOT NULL
);
ALTER TABLE kb_chunks ENABLE ROW LEVEL SECURITY;
ALTER TABLE kb_chunks FORCE ROW LEVEL SECURITY; -- applies to the table owner too
CREATE POLICY chunk_access ON kb_chunks
USING (
tenant_id = current_setting('app.tenant_id')
AND allowed_groups && string_to_array(current_setting('app.user_groups'), ',')
);
The retrieval service sets app.tenant_id and app.user_groups from the verified token, and every ANN query — however it's written, whoever wrote it — runs under the policy. Dedicated vector stores give you the same shape with metadata filters plus namespaces; the invariant to preserve is that the filter derives from verified identity and is not constructable or influenceable from user input.
Deny by default. A chunk with missing or malformed ACL metadata must be unreachable, not public. Enforce it structurally (NOT NULL columns, a policy that matches on membership rather than on absence-of-restriction) and test it explicitly.
What is not a control: the prompt. "Only answer from documents the user is authorized to see" is a request to a text generator, not an enforcement mechanism, and it fails under the mildest prompt injection. What is also not a control: filtering candidates in application code after they come back from the store.
If restricted chunks are fetched and then dropped in application code, they have already leaked — into traces, logs, caches, and every code path one bug away from the response. Post-filtering is not defense in depth; it is exposure with extra steps. Filter inside the store, pre-ranking, from verified identity.
Then prove it, continuously, with a leakage test suite — the negative tests almost nobody writes. For each protected fixture document: query as an unauthorized principal using the most adversarial query you can construct (a verbatim sentence from the restricted document — the query most likely to embed right next to it), and assert zero restricted chunks in the results at any k. Add a missing-metadata fixture to prove deny-by-default, cross-tenant probes with real tenant identities, and a revoked-user probe that runs right after a revocation to measure the propagation SLA. Run the suite in CI against fixtures and on a schedule against production canary tenants.
Leakage-scenario items have one right shape — entitlement metadata stamped at ingestion, filtered inside the store at query time from verified identity, deny by default, proven by negative tests. Any option where the prompt, the LLM, or post-retrieval application code enforces access is the planted distractor.
Multi-tenancy is ACL propagation's structural sibling: instead of filtering rows within a shared audience, you're choosing how much physical separation each tenant's data gets. The spectrum, in ascending isolation and cost:
| Model | Isolation mechanism | Operational cost | Fits |
|---|---|---|---|
| Shared collection + tenant filter | A metadata filter you must never omit | Lowest | Many small tenants, internal multi-team platforms |
| Namespace / partition per tenant | Store-enforced partition (Pinecone namespaces, Qdrant payload-partitioned multitenancy) | Low | The SaaS default — partition safety without index sprawl |
| Collection or index per tenant | Separate index structures; per-tenant tuning and cutover | Medium — index proliferation, per-collection memory overhead | Few large tenants; per-tenant SLAs, models, or index parameters |
| Cluster / instance per tenant | Physical separation | Highest | Regulated clients, data-residency and contractual isolation |
Three operational consequences follow from whichever point you pick:
tenant_id, because "the index is fresh on average" is worthless to the one tenant whose connector has been dead for a week; premium tiers often buy tighter freshness, which you can only sell if you can measure it.Version cutovers compound with tenancy: an embedding-model upgrade in a per-tenant-collection design is a per-tenant blue-green migration, which is exactly why large multi-tenant platforms stage cutovers tenant-by-tenant, canary tier first.
For services companies, isolation is a commercial decision as much as a technical one — client MSAs and BFSI-sector rules frequently mandate dedicated infrastructure regardless of what the engineering trade-off table says. Get the isolation model into the SOW before you get it into the architecture diagram.
Your first 15 minutes Monday — if your production index has no end-to-end freshness signal, add a sentinel probe (write a timestamped change to a dedicated doc, assert it's retrievable within your SLA); if it has one, write a single leakage negative test that queries a restricted document's verbatim sentence as an unauthorized principal and asserts zero restricted chunks returned.
Check yourself: A teammate proposes fetching candidate chunks from the vector store and then dropping the ones the user isn't allowed to see in the retrieval service's Python code. Why is that not a valid access control, and where must the entitlement filter live?
This module anchors LAB-11 — CDC → vector sync: change detection via content hashing from a source table, selective re-embed/upsert/delete in the vector store, orchestrated in Airflow, with an index-freshness monitor and a canary retrieval query — artifact: the sync DAG plus a freshness dashboard demonstrating sub-15-minute index lag after source updates. The exercises extend the lab into the deletion audit, the blue-green cutover, and the leakage suite, which together make the strongest portfolio pieces in this track: almost every candidate can build a RAG demo; very few can show a revocation SLA and a passing leakage test suite. The playbooks provide the sync-ledger schema template, the versioned-cutover runbook, the canary-set and sentinel-probe templates, the ACL design checklist (entitlement resolution, filter placement, deny-by-default verification), the tenancy decision matrix, and the deletion-certificate checklist. Everything runs on the free path: local Postgres/pgvector or Qdrant OSS, Airflow OSS, and lab corpora sized to keep embedding spend within the ~$5-per-lab budget. Do this at work this week → pb-36 Sync Vector Indexes Incrementally from Change Data Capture, pb-37 Monitor RAG Pipeline Freshness and Retrieval Quality, and pb-38 Propagate Access Controls into Vector Indexes.