Mode switching: standalone <-> cluster

The two modes#

standalone

Exactly one instance. replicas = 1. No followers, no failover peers.

cluster

A leader (master) plus one or more followers on distinct nodes. replicas >= 2.

A tenant is created in one mode (provision defaults to standalone; choosing cluster starts it with 2 replicas). Mode switching changes between them later without recreating the tenant. The same call handles both directions — there is no separate "convert to cluster" vs "collapse to standalone" function.

Why it is non-destructive in both directions#

The engine's storage and WAL format is IDENTICAL whether a node runs standalone or as part of a cluster. A follower is just another node applying the leader's committed op-frames to the same on-disk format via apply_replicated, which durably logs them so the follower can itself fail over. That means:

• standalone -> cluster: bring up follower(s) on distinct nodes and stream the leader's WAL + change-stream to them until caught up. Nothing on the original instance is converted or moved.
• cluster -> standalone: drain the follower(s); the leader already holds all the data, so it simply keeps serving as a single instance.
• Neither direction performs a data conversion, dump/restore, or rebuild — it is purely add/remove replicas around an unchanged store.

Replica-count effects#

Switching mode sets the replica count deterministically from the target:

When switching TO cluster, if the tenant somehow already carried a higher replica count it is preserved (max with 2); a fresh standalone goes to exactly 2. When switching TO standalone, replicas collapses to 1. The node allocation always moves by the per-instance cap times the replica delta, and node counters are clamped at 0.

Inputs and validation#

Validation rejects, before any write: a missing id or mode (InvalidInput: missing required field), an unrecognized mode value (mode must be standalone|cluster), an unknown tenant (no tenant '<id>'), and a no-op where the tenant is already in the requested mode (tenant '<id>' is already <mode>). The role gate requires admin or higher (deny-by-default).

What it does, step by step#

1. Re-check ctx.role >= admin.
2. Require id and mode fields; normalize mode to exactly standalone or cluster (else InvalidInput).
3. Load the tenant row; reject if absent.
4. Reject if the tenant is already in the target mode (no-op guard).
5. Compute new replicas: max(old, 2) for cluster, or 1 for standalone.
6. UPDATE _cp_tenants SET mode, replicas via the write funnel.
7. Apply node allocation delta: cap x (new_rep - old_rep) on the placed node.
8. Write a _cp_audit row (action=switch_mode, result=the target mode).
9. Return the new state with a PLANNED note naming the direction and the non-destructive guarantee.

standalone (replicas 1)                 cluster (replicas >= 2)
        │   switch_mode mode=cluster            │   switch_mode mode=standalone
        ▼                                       ▼
  add follower(s) on distinct nodes      drain follower(s), keep ONE instance
  via replication path (catch up live)   leader already holds all data
        │                                       │
  REAL: mode=cluster, replicas=2 (or max)  REAL: mode=standalone, replicas=1
  REAL: node alloc += cap x (new-old)     REAL: node alloc += cap x (1-old)  (releases)
        └──────────── on-disk layout NEVER changes ───────────┘

Worked example: both directions#

Tenant acme starts standalone (replicas 1). We switch it to cluster, then back to standalone. This is the exact non-destructive round-trip the test suite asserts.

{
  ctx: { user: "op", role: "admin" },
  id: "acme",
  mode: "cluster"
}

// code = Ok
{
  tenant_id: "acme",
  simulated: "PLANNED (standalone->cluster: add follower(s) on distinct nodes via replication) — NON-DESTRUCTIVE: on-disk layout is identical across modes",
  status: "active",
  mode: "cluster",
  threads: 2,
  mem_gb: 4,
  storage_gb: 20,
  replicas: 2,
  node_id: "node-eu-1"
}

Persisted: _cp_tenants(acme) mode=cluster, replicas=2.
Node alloc doubles for the tenant (cap charged once per replica): e.g. alloc_threads 2 -> 4.

{
  ctx: { user: "op", role: "admin" },
  id: "acme",
  mode: "standalone"
}

// code = Ok
{
  tenant_id: "acme",
  simulated: "PLANNED (cluster->standalone: drain follower(s), keep one instance) — NON-DESTRUCTIVE: on-disk layout is identical across modes",
  status: "active",
  mode: "standalone",
  threads: 2,
  mem_gb: 4,
  storage_gb: 20,
  replicas: 1,
  node_id: "node-eu-1"
}

Persisted: _cp_tenants(acme) mode=standalone, replicas=1.
Node alloc returns to one instance's worth (the follower capacity is released).

{
  ctx: { user: "op", role: "admin" },
  id: "acme",   // already standalone
  mode: "standalone"
}

// code = InvalidInput
{
  message: "tenant 'acme' is already standalone"
}

Console usage and relationship to scaling#

On the Tenants screen (/cloud/tenants, admin+), the mode form POSTs id + mode to cloud.action.switch_mode, which after session + CSRF + rank checks calls cloud.switch_mode. The success flash carries the direction + non-destructive note. Mode switching is the gateway to horizontal scaling: a standalone tenant must be switched to cluster (which sets it to 2 replicas) before cloud.scale_horizontal will accept a replica change. Vertical scaling, by contrast, works in either mode since it resizes the per-instance cap rather than the count.