When you ask a question on Ask for the Moon, the answer is assembled from three very different databases at once. MongoDB holds the documents. A Neo4j knowledge graph knows who wrote what, which answer validated which question, and how people, tags and documents relate. A Milvus vector database lets us retrieve the semantically closest content even when nobody used the same words.
The obvious question — the one we get asked every time we describe this — is: how do you keep three databases telling the same story? If MongoDB says a question was edited or deleted, Neo4j and Milvus have to agree, quickly, without a human in the loop and without slowly drifting apart over months of production
traffic.
This article is about that machinery. It is not glamorous, but it is the part that decides whether a RAG system stays trustworthy. We will start with why we need three databases in the first place, then walk through the pipeline we built, and finish with the honest trade-offs — including the ones we are still living with.
Why three databases?
The instinct of any engineer is to resist this. Three databases is three times the operational surface, three ways to be inconsistent, three things to back up.
So the bar for adding one has to be high. Each of ours earns its place by answering a question the others cannot.
| Database | What it is good at | The question it answers |
| MongoDB | Flexible documents, transactions,
the app's read/write path | “What is the current state of this question, answer, person or document?” |
| Neo4j | Traversing relationships cheaply
at any depth | “Who are the experts on this topic? Which answers were validated, and by whom?” |
| Milvus | Nearest-neighbour search over
high-dimensional vectors | “What content means the same thing as this query, regardless of wording?” |
MongoDB is our source of truth. It is where the application writes, where atransaction is authoritative, and where "the current state of the world" lives.Everything else is a projection of that truth, shaped for a job MongoDB is badat.
Neo4j exists because relationships are a first-class concern for us.
"Find the people who authored validated answers to questions tagged with X"
is a graph traversal — trivial in Cypher, painful and slow as a pile of MongoDB
$lookup
stages. So we project the documents into a graph of
Profile,
QuestionForum,
AnswerForum,
Document,
Tag
and
Space
nodes connected by typed edges like
AUTHORED,
ANSWERED,
VALIDATED
and
TAGGED_WITH.
Milvus exists because keyword search is not enough for an industrial knowledge
base. A technician asking "how do I torque this flange" should retrieve the
procedure that never once uses the word "torque". So we embed the embeddable
content — questions, answers, tags — into vectors and let Milvus find the
nearest neighbours.
The moment you accept that these projections are worth having, you inherit the
problem this article is about: a projection that drifts from its source is
worse than no projection at all, because people trust it. So how do you keep
them honest?
The naïve approach, and why we dropped it
The first version everyone builds is dual writes: when the app saves a document,
it also writes to Neo4j and Milvus in the same request. It is simple, and it is
a trap. If the graph write fails after the Mongo write succeeds, you are now
inconsistent with no record of it. If you wrap all three in a distributed
transaction, you have coupled the availability of your write path to the
availability of every downstream store — a bad day for Milvus becomes a bad day
for saving a question. And embeddings are slow: you do not want a user's save
button waiting on a GPU.
We wanted the opposite properties: the app should only ever talk to MongoDB,
projections should update asynchronously, a downstream outage should cause
lag rather than data loss, and the whole thing should be able to catch up on
its own after a failure. That points to one pattern.
Change Data Capture: let MongoDB tell us what changed
Instead of asking the application to notify everyone, we listen to MongoDB
itself. MongoDB exposes change streams — a live, ordered feed of every
insert, update, replace and delete, built on the same replication oplog the
database already maintains. (This is also why change streams require a replica
set or a sharded cluster; happily, our MongoDB Atlas cluster is a replica set by
definition, so we get it for free.)
We wrote a small, single-purpose Go service —
aftm-mongo-cdc-dispatcher
— whose entire job is to tail that change stream and turn it into a clean stream of events.
The full pipeline looks like this:
MongoDB Atlas ───────▶ ┌────────────────────────────────┐
(source of truth, │ aftm-mongo-cdc-dispatcher │
replica set) │ (Go, single replica) │
│ • tails the change stream │
│ • persists a resume token │
│ • publishes a stable event │
└───────────────┬────────────────┘
│
│ topic exchange "mongoevents"
│ key: <base>.<collection>.changed
▼
┌──────────────────┐
│ RabbitMQ │
│ (3-node cluster) │
└───────┬────┬─────┘
│ │
mongoevents.elm4j │ │ mongoevents.elmbed
(binds "#") │ │ (binds
│ │ *.questions/answers/records.changed)
│ │
▼ ▼
┌──────────┐ ┌──────────┐
│ elm4j │ │ elmbed │
│ (Python) │ │ (Python) │
└────┬─────┘ └────┬─────┘
│ │
▼ ▼
┌──────────┐ ┌──────────┐
│ Neo4j │ │ Milvus │
│ (graph) │ │(vectors) │
└──────────┘ └──────────┘
Three design decisions in the dispatcher do most of the work.
1. It publishes a stable, database-agnostic event. A raw MongoDB change
document is a messy, deeply-nested BSON structure that leaks Mongo internals. We
transform it once, at the source, into a flat JSON contract that every consumer
can rely on:
{
"event_id": "1721815890123456789-66a1f...",
"source": "mongo.cdc",
"collection": "questions",
"entity_type": "Question",
"operation": "update",
"document_id": "66a1f...",
"occurred_at": "2026-07-24T09:31:30.12Z",
"ordering_key": "questions:66a1f...",
"payload": {
"full": { "...": "the full document after the change" },
"changed_fields": { "message": "the edited text" }
},
"meta": { "schema_version": 1, "mongo_operation": "update" }
}
The event carries the full document (we enable
updateLookup
on the change stream) so consumers rarely have to call back into MongoDB,
plus the set of
changed_fields
so a consumer can decide whether a change is even relevant to it.
The
schema_version
field is our escape hatch: the day we need to change the contract,
consumers can tell old from new.
2. It checkpoints after publishing, never before.
MongoDB hands out a resume token with every change event — a cursor position you can restart from.
The dispatcher saves that token back into a MongoDB collection
(cdc_checkpoints),
but only after RabbitMQ has confirmed the message is safely persisted:
if err := d.publisher.Publish(d.ctx, event, routes); err != nil {
return fmt.Errorf("publish failed, will retry: %w", err)
}
if err := d.saveCheckpoint(resumeTokenRaw); err != nil {
return fmt.Errorf("checkpoint save failed: %w", err)
}
The ordering matters more than it looks. If the service crashes between publishing and checkpointing, it restarts from the old token and re-publishes the last event. That gives us at-least-once delivery: we might send a duplicate, but we never silently drop a change. We chose "occasionally duplicated" over "occasionally lost" on purpose — and, as you will see, the consumers are built to make duplicates harmless.
3. It runs as exactly one replica. A change stream is an ordered log, and the resume token is a single position in it. Running two dispatchers would mean two
cursors, duplicated events and no ownership of the checkpoint. So it is a single-replica Kubernetes Deployment with autoscaling deliberately turned off. It is a lightweight tail — one small pod comfortably keeps up — and the constraint buys us clean ordering. On any failure it reconnects to both MongoDB
and RabbitMQ with backoff and resumes from its last checkpoint.
RabbitMQ: the buffer that absorbs bad days
The dispatcher publishes to a RabbitMQ topic exchange called
mongoevents,
with a routing key per collection of the form
<base>.<collection>.changed.
That routing key is the seam that lets each projection subscribe only to what it
cares about.
RabbitMQ is not just a pipe here; it is the shock absorber. Because events sit
durably in queues, a consumer can be down for a deploy — or an hour — and simply
catch up when it comes back. Publisher confirms make sure the dispatcher only
advances its checkpoint once a message is truly persisted. In production it runs
as a 3-node cluster with
pause_minority
partition handling, so a network split fails safe rather than forking history.
Each projection gets its own durable queue with its own
dead-letter queue.
A message the consumer cannot process is rejected without requeue and lands in a
...Failure
queue rather than spinning in a poison-message loop. That separation — one exchange,
independent queues — is what lets the graph and the vector projections fail, deploy
and scale completely independently of each other.
Two consumers, two shapes of the same truth
Downstream, two small Python services subscribe to the stream. They are
deliberately not one service: the graph and the vector store have almost nothing
in common except their source.
-
elm4j
binds the catch-all routing key
#
and filters by collection in code, because the graph touches nearly every
entity type. It maps each document to nodes and relationships and writes
them to Neo4j.
-
elmbed
binds only
.questions.changed,
.answers.changed
and
.records.changed,
because only those carry embeddable content. It turns text into vectors
and writes them to Milvus.
Making duplicates harmless: idempotency by construction
Remember that the dispatcher is at-least-once — the same event can arrive twice.
The consumers survive this not by tracking what they have seen, but by making
every write idempotent, keyed on the MongoDB
_id.
On the Milvus side, every write is an
upsert
on a primary key equal to the Mongo document id. Re-processing an event just
re-writes the same row:
collection.upsert({
"id": question.id,
"organisation": question.organisation,
"text": question_message,
"sparse_embedding": docs_embeddings.sparse[0],
"dense_embedding": [float(x) for x in docs_embeddings.dense.flatten()],
})
On the Neo4j side, the same principle expressed in Cypher: every node and every
relationship is a
MERGE
keyed on
{id, organisation_id},
never a blind
CREATE.
Run it once or five times, the graph ends up identical.
MERGE (q:QuestionForum {id: $id, organisation_id: $org})
SET q.text = $text, q.synced_at = $now
This is the quiet payoff of the earlier decision. Because every write is
idempotent, "at-least-once" delivery stops being a liability. We never had to
build exactly-once delivery — a famously hard problem — because we made
duplicates a no-op instead.
Handling deletes without hard deletes
Deletes are the classic way projections rot: a document vanishes from the source
and lingers forever downstream. Our application uses soft deletes,
which turns a delete into an ordinary update carrying
changed_fields.deleted = true.
Each consumer recognises that and propagates it: Milvus issues a
collection.delete(expr="id == '<id>'"),
Neo4j issues a
DETACH DELETE,
which removes the node and every edge attached to it in one step.
Only re-embedding when it matters
Embeddings cost real money and GPU time — they are served by a separate inference
service running BGE-M3, which produces both a dense (1024-dim)
and a sparse vector for hybrid search. So
elmbed
looks at
changed_fields
before doing any work: if a question was edited but its
message
field did not change (say only a tag was added), it reuses the existing vectors
instead of re-embedding. The
changed_fields
we put into the event at the very start of the pipeline is what makes that
optimisation possible.
Multi-tenancy, two ways
One detail worth calling out, because the two stores solved the same problem
differently. Milvus gives each organisation its own physical database
(org_<id>), a hard isolation boundary.
Neo4j uses a single database with an
organisation_id
property baked into every node and every
MERGE/MATCH
key — softer, but far cheaper to operate at our number of tenants. Same requirement,
two legitimate answers; the right one depends on the store.
Staying in sync — because "eventually" isn't a guarantee
Everything so far gives us eventual consistency on the happy path. But overmonths, entropy wins: a consumer bug, a message that dead-lettered and was neverreplayed, a manual database fix, a deploy at exactly the wrong moment. Any ofthese leaves a projection subtly wrong. A pipeline you cannot audit is a pipelineyou cannot trust, so we treat drift as inevitable and build for it directly.
A rebuild-from-scratch path.
Because MongoDB is the single source of truth, any projection can be thrown away
and rebuilt from it. Both consumers expose this:
elmbed
can drop an organisation's Milvus collections and re-embed everything from Mongo;
elm4j
can clear an organisation's subgraph and repopulate it in dependency order
(people, then questions, then answers, then validations, and so on).
This is our ultimate backstop — worst case, we regenerate the entire projection —
and it is only possible because we were disciplined about what is authoritative.
A scheduled audit.
A cron job runs every six hours and compares the two sides: not just counts,
but the actual set of ids present in MongoDB versus present in Milvus.
It streams ids from each side and computes two differences —
missing_in_milvus
(in Mongo, absent downstream) and
orphan_in_milvus
(downstream, gone from Mongo). That is the difference between "we think it's fine"
and "we checked."
A guarded reconciliation.
A second daily job acts on what the audit finds. It re-embeds the missing documents
(via the same idempotent
upsert)
and deletes the orphans. Deletion carries a deliberate safety catch:
POST /synchronisation/reconcile/milvus-mongo?apply=true&max_delete=500
If reconciliation ever wants to delete more than
max_delete
vectors from a collection, it refuses and logs instead of deleting.
The scenario we are defending against is a transient audit — a flaky read,
a half-connected org — concluding that thousands of valid vectors are "orphans"
and wiping them. The re-embed half still runs; only the destructive half is capped.
We would rather carry a few stale vectors for a day than mass-delete good data
in a millisecond.
The trade-offs we chose
A pipeline like this is a series of deliberate trade-offs, not a search for
perfection. A few of the calls we made:
- We favour simplicity over strict global ordering. Events are processed serially, which keeps them in order in practice, and idempotent writes mean the occasional redelivery is harmless. For our workload — people editing documents, not thousands of writes per second to a single key — that has been exactly the right place to draw the line, and it avoids a whole class of distributed-systems complexity we simply don't need.
- We invest more where the cost of drift is higher. The vector store, which is expensive to rebuild and central to retrieval quality, gets the strictest delivery guarantees and the automated audit-and-reconcile loop. The graph, which is cheap to regenerate from the source, leans more on periodic rebuilds.
Matching the effort to the stakes keeps the system lean instead of uniformly gold-plated.
We keep an internal audit of these choices so they stay visible and revisited
deliberately, rather than quietly hardening into assumptions.
What we would tell another team
If you take one thing from this, let it be the shape rather than the specific
tools. RabbitMQ could be Kafka; Milvus could be another vector store; the Go
dispatcher could be Debezium. The principles are what travelled well for us:
-
Pick one source of truth and mean it.
Everything else is a disposable projection. That single decision is what makes
rebuilds, audits and reconciliation even possible.
-
Capture change at the source, not in the application.
CDC decouples your write path from your projections. A downstream outage becomes
lag, not loss.
-
Prefer at-least-once plus idempotent writes over exactly-once.
Idempotency is local and testable; exactly-once delivery is a distributed-systems
tar pit.
upsert
and
MERGE
on the source id did more for our reliability than any clever broker configuration
could.
-
Assume you will drift, and build the audit before you need it.
Eventual consistency is a promise you can only keep if you continuously check it —
and put a guard rail on anything that deletes.
Three databases, one truth. The trick was never a magic sync engine. It wasbeing ruthless about which database is authoritative, capturing its changes atthe source, making every downstream write safe to repeat, and never trusting"eventually" without a job that verifies it.
.png)
.png)
