Horizontal scaling: cluster replica count

What horizontal scaling changes#

Where vertical scaling makes one instance bigger, horizontal scaling makes MORE (or fewer) instances. A cluster tenant runs a leader plus followers; the replica count is the total number of copies. Adding replicas spreads read load and improves failover; removing them frees node capacity. The per-instance cap (threads/mem/storage) is unchanged — only the count changes — so the node allocation moves by exactly cap x (new_replicas - old_replicas).

replicas

Total copies of the tenant: the leader/master plus its followers. A cluster always has at least 2 (master + 1 follower on distinct nodes).

per-instance cap

threads / mem_gb / storage_gb for ONE copy. The node allocation is charged once per replica.

cluster-only

Horizontal scaling is rejected unless mode is cluster; switch the tenant to cluster first (see Mode switching).

Cluster-only precondition#

A standalone tenant runs exactly one instance and has no notion of replica count. Calling cloud.scale_horizontal on a standalone tenant is rejected before any write.

{
  ctx: { user: "op", role: "admin" },
  id: "solo",        // a standalone tenant
  replicas: 3
}

// FunctionResponse: code = InvalidInput
{
  message: "horizontal scaling needs cluster mode — switch mode first"
}

No rows change. To proceed: call cloud.switch_mode with mode=cluster (which sets replicas to at least 2), then call cloud.scale_horizontal.

Inputs and bounds#

The new replica count is clamped to the range 1..16. Because the function rejects non-cluster tenants up front, the effective floor in practice is whatever the cluster already has (a cluster is created with 2). Setting replicas down toward 1 drains followers; setting it up adds them. Both directions are supported by the same call — there is no separate scale-down function.

min 1

The hard lower clamp on the replica field is 1; the value cannot go below a single copy.

max 16

The hard upper clamp is 16 replicas per tenant.

What it does, step by step#

1. Re-check ctx.role >= admin (deny-by-default; AccessDenied otherwise).
2. Load the tenant row; reject if absent (no tenant '<id>').
3. Reject if mode is not cluster (horizontal scaling needs cluster mode — switch mode first).
4. Read the OLD replica count (at least 1) and compute the NEW count from the replicas field, clamped 1..16.
5. UPDATE _cp_tenants SET replicas = new via the write funnel.
6. Apply the node allocation delta: cap x (new - old) for each of threads, mem, storage, on the tenant's placed node (clamped >= 0).
7. Write a _cp_audit row (action=scale_horizontal, result=ok).
8. Return the new tenant state and the PLANNED rebalance note.

old_rep = 2          new_rep = 3        cap = (threads, mem_gb, storage_gb)

  _cp_tenants: replicas 2 ──▶ 3   (REAL update)
  node alloc += cap x (3 - 2) = +1 instance worth   (REAL, clamped >= 0)
  _cp_audit  += { action: scale_horizontal }        (REAL)
        │
        ▼
  SIMULATED (PLANNED): rebalance 2 -> 3 replicas across distinct nodes
                       via the replication path (promote / drain),
                       preserving isolation

Worked example#

Tenant solo is switched to cluster (giving it 2 replicas) and then scaled out to 3 replicas. This is the exact flow exercised by the test suite.

{
  ctx: { user: "op", role: "admin" },
  id: "solo",
  replicas: 3
}

// FunctionResponse: code = Ok, data = encoded record
{
  tenant_id: "solo",
  simulated: "PLANNED: rebalance cluster 2->3 replicas across distinct nodes via the replication path (promote/drain), preserving isolation",
  status: "active",
  mode: "cluster",
  threads: 1,
  mem_gb: 2,
  storage_gb: 10,
  replicas: 3,
  node_id: "node-eu-1"
}

Persisted result:
  _cp_tenants(solo): replicas = 3
  _cp_nodes: alloc bumped by one instance's cap (threads x1, mem x1, storage x1)
Scaling back down (e.g. replicas: 2) subtracts one instance's cap; node alloc never goes below 0.

Console usage#

On the Tenants screen (/cloud/tenants, admin+), the horizontal-scale form POSTs id + replicas to cloud.action.scale_horizontal. The handler verifies the session and CSRF token, does the client-side admin rank check, then calls cloud.scale_horizontal with the operator's role in ctx. Success redirects with a flash containing the PLANNED rebalance note; a standalone tenant produces the cluster-mode error flash.