Scaling, migration & upgrades (mechanism)

One mechanism behind everything#

Migration, scaling, mode-switching, and software upgrades reuse a single primitive: single-leader replication plus the engine's change-stream. Each operation adds a replica, syncs it live while the current node keeps serving every request, then cuts over. The enabler is that the storage/WAL format is IDENTICAL in standalone and cluster — so every switch is non-destructive (add/remove replicas, never a data conversion).

   ┌─ add a fresh replica (new node / new version) ─┐
   │  stream WAL + change-stream until CAUGHT UP     │  ← ops keep running on the
   │  (the live node serves the whole time)          │     live node, no break
   └───────────────────────┬─────────────────────────┘
                           ▼
        atomically PROMOTE + REDIRECT (epoch-fence the old)  ← writes pause ~sub-second
                           ▼                                    (degraded, not broken)
                    drain + drop the old node
   Enabler: the storage/WAL format is IDENTICAL in standalone & cluster, so every
   switch is non-destructive — add/remove replicas, never a data conversion.

How replication actually works today#

The replication seam every cutover reuses is built: from-scratch TCP WAL replication, master -> follower, over the engine's change-stream hook, with zero third-party dependencies (plain std::net + length-prefixed framing). It is single-direction primary/replica.

Master

Registers a change observer (Qdb::subscribe_changes) and ships every committed (seq, op_frames) batch to all connected followers over TCP. A bounded in-memory ring buffer of recent batches lets a reconnecting follower catch up from its last-applied seq.

Follower

Connects to the master, announces its current applied seq, then receives the catch-up backlog + live stream, applying each batch via Qdb::apply_replicated (which durably logs on the follower so it can itself fail over). Reconnects with capped backoff on disconnect.

Op frame

encode_op(OP_PUT|OP_DELETE, table, record) — the exact bytes the master's WAL committed, which apply_replicated consumes. Batches are sent whole so partial batches never apply.

Catch-up ring

A bounded ring (DEFAULT_RING_CAPACITY = 4096 batches) of recent commits. A follower too far behind the retained tail receives a RESYNC marker and resets to pull the retained tail.

Promotion / failover is cheap because a follower already holds the data durably: Follower::promote() stops the follow loop, after which serve_master can run on the same Qdb. Promotion also flips the node's WriterGate to writer, so writer-gated background work (the job scheduler, including the TTL sweep) resumes on the newly-promoted leader and never double-runs across the cluster.

The silent cutover#

The cutover is silent because the storage model is the same for standalone and cluster, and the write path is already FIFO-ordered. At the cutover, reads keep flowing from data present on both nodes; new writes are held briefly in the FIFO lane, the location repoints, and the held writes flush to the new node in order. Nothing is dropped, nothing restarts.

 SILENT cutover  (same storage model · FIFO write-hold · reads never pause)

 t0  OLD node (v1 / node-A) serving reads + writes normally
      │  spin the NEW node (v2 / node-B), stream WAL ──▶ NEW catches up LIVE
      ▼
 t1  CUTOVER (milliseconds):
      reads ───────────── never pause (data is on both nodes) ─────────────▶
      writes → held in the FIFO write lane ──┐  ← the ONLY effect: mutations
      flush last WAL delta  OLD ──▶ NEW       │    queue for the cutover window,
      epoch-fence OLD  +  REPOINT location ───┤    then apply on NEW in order
        (connection / domain / tunnel → B)    │
      release queued writes ──────────────────▶ NEW applies them (FIFO, none lost)
      ▼
 t2  NEW node serving; OLD drained + dropped.   No restart. No break.

Every scenario, and what it costs#

The rule is uniform — every operation is silent, with no restart, standalone included.

Managed upgrades: CI/CD with proper drain#

Qirava builds and signs each release (transparency-logged); the node agent verifies the signature before running it, then rolls it out per the tenant's update policy — automatic, approval-required, or pinned to a version.

 Qirava CI → build → SIGN release → transparency log → artifact store
      │  control plane schedules a rollout per TENANT POLICY: auto | approve | pinned
      ▼
 node agent: pull + VERIFY signature  →  for each node, rolling:
   ① spin new-version replica → ② sync (ciphertext, live) → ③ DRAIN old:
        router stops new conns → in-flight finish (grace) → FIFO-hold writes → flush WAL
   → ④ epoch-fence + promote new → ⑤ REPOINT route → ⑥ release held writes
   → ⑦ drop old → ⑧ HEALTH/LAG gate: fail → abort + rollback; pass → next node
   Cluster = zero downtime (peers mask each node); standalone = the same silent cutover.

Proper drain means the router repoints FIRST (so no new request lands on the dying node), in-flight requests get a grace window, stragglers FIFO-hold and forward to the new leader, the WAL flushes, and only then does the old process exit. A failed health or replication-lag check aborts and rolls back.

The primitives this needs#

Single-leader replication and the change-stream are built. Three pieces turn them into seamless cutovers and are the focused remaining work:

How the control plane drives a cutover (example)#

The real replication test demonstrates the mechanism end to end on two in-process engines: a master ships inserts/updates/deletes over a real TCP socket, the follower mirrors them exactly, tracks the applied seq, and can be promoted. This is the seam every scale/migrate/upgrade reuses.

// master side — ordinary writes through the engine
master_db.execute(r#"INSERT INTO t { name: "ada" }"#);
master_db.execute(r#"INSERT INTO t { name: "bob" }"#);
master_db.execute(r#"UPDATE t SET name = "ada2" WHERE _id = 1"#);
master_db.execute("DELETE FROM t WHERE _id = 2");

// follower side — mirrors the master exactly via replicated op-frames
follower_db.execute("SELECT count(*) FROM t")           // -> { "count_all": 1 }
follower_db.execute("SELECT name FROM t WHERE _id = 1") // -> name = "ada2"
follower.last_applied()                                 // -> 4 (four commits applied)

The follower received the master's exact committed op-frames (the same bytes the WAL committed), applied them in seq order via apply_replicated, and durably logged them — so it could be promoted to leader and serve as master on the same store. A reconnecting follower resumes from its last-applied seq via the catch-up ring; one too far behind gets a RESYNC and re-pulls the retained tail.