Lifecycle: suspend, resume, terminate

The tenant status model#

A tenant row lives in the control DMS's own _cp_tenants catalog and carries a status field. The lifecycle functions are state machines over that one field. They never reach into the tenant's data — they only flip control-plane state and record the infrastructure effect they *would* perform.

Roles and authorization#

Cloud governance is a separate authority domain from any tenant's own governance, with the ranking custodian (3) > admin (2) > user (1). Every cloud.* function re-checks the caller's ctx.role server-side, deny-by-default: a missing or unknown role ranks 0 and is rejected. The Cloud Console performs a cheap client-side rank pre-check so the UI can 403 fast, but that is a convenience only — the real boundary is the role check inside each function plus the planner gate on the _cp_* write.

cloud.suspend — stop serving (non-payment)#

Suspend is the typical response to non-payment. It sets status = suspended and returns a PLANNED note describing the simulated effect: stop the tenant DMS from serving while preserving its data and seed. It refuses to act on an already-terminated tenant (InvalidInput: "tenant is terminated"). The node allocation is intentionally NOT released on suspend — the capacity is held so resume can restart in place.

Required field

id — the tenant id.

Required role

admin or higher (ctx.role).

Real effect

UPDATE _cp_tenants SET status = "suspended" WHERE id = <id>, plus a _cp_audit row (action: suspend, result: suspended).

Simulated effect

PLANNED: stop the tenant DMS from serving (data + seed preserved; resume re-provisions in place).

cloud.resume — restart in place (inverse of suspend)#

Resume is the exact inverse of suspend. It only acts on a tenant whose status is currently suspended — calling it on any other tenant returns InvalidInput ("tenant is not suspended"). On success it restores the tenant to active if it still holds a node (node_id non-empty); if the tenant has no node, it goes to pending_capacity so the placer can retry. Because suspend preserved everything, resume is non-destructive: the simulated effect is simply restarting the tenant DMS in place.

Required field

id — the tenant id.

Required role

admin or higher.

Precondition

status must equal "suspended"; otherwise InvalidInput.

Real effect

UPDATE _cp_tenants SET status = "active" (or "pending_capacity" when node_id is empty) WHERE id = <id>, plus a _cp_audit row (action: resume).

Simulated effect

PLANNED: restart the tenant DMS in place (data + seed preserved).

cloud.terminate — wipe the tenant (custodian only, destructive)#

Terminate is the destructive top-tier action and the only one restricted to a cloud custodian. It sets status = terminated, releases the node allocation the tenant held (the per-instance cap multiplied by the replica count, applied as a negative delta to the node's alloc_* counters), and cancels the tenant's subscription (_cp_subscriptions.status = "cancelled"). It refuses to re-terminate an already-terminated tenant (InvalidInput: "tenant already terminated"). The simulated effect — wiping the tenant's data and seed — is irreversible.

Node release is the key difference from suspend. Where suspend holds capacity for a later resume, terminate frees it back to the fleet. For a tenant with 8 threads at 1 replica, the placed node's alloc_threads drops by 8 (clamped at 0); the same applies to memory and storage, each scaled by replicas.

Required field

id — the tenant id.

Required role

custodian (rank 3). admin is rejected.

Real effect

UPDATE _cp_tenants SET status = "terminated"; node_alloc_delta releases threads*replicas, mem_gb*replicas, storage_gb*replicas from the placed node; UPDATE _cp_subscriptions SET status = "cancelled" WHERE tenant_id = <id>; plus a _cp_audit row (action: terminate, result: terminated).

Simulated effect

PLANNED: wipe the tenant's data + seed (irreversible); node capacity released.

Flow: suspend, resume, terminate#

  provision
     |
     v
  [active] <-----------------+
     |   ^                    |
 suspend  | resume            |
     |   | (status must be    |
     v   |  suspended)        |
 [suspended]                  |
     |                        |
 (node capacity STILL held    |
  through suspend)            |
     |                        |
     +--- resume -------------+   (-> active if node_id set,
     |                            else pending_capacity)
     |
 terminate (custodian only)
     |
     v
 [terminated]  -- releases node alloc, cancels subscription
               -- terminal: cannot suspend/resume/scale

Worked example: suspend then resume#

The Cloud Console is a client: the operator clicks Suspend on a tenant row, which POSTs a form to /cloud/actions/suspend; that handler injects the session's role into ctx and calls cloud.suspend through the runtime. Below is the underlying function call (the form values become the request record) and the real response envelope, transcribed from the source. Assume tenant acme is currently active, placed on node-eu-1, with 8 threads / 16 GB mem / 50 GB storage, 1 replica.

cloud.suspend
ctx  = { user: "op", role: "admin" }
input = { id: "acme" }

{
  "tenant_id": "acme",
  "simulated": "PLANNED: stop the tenant DMS from serving (data + seed preserved; resume re-provisions in place)",
  "status": "suspended",
  "mode": "standalone",
  "threads": 8,
  "mem_gb": 16,
  "storage_gb": 50,
  "replicas": 1,
  "node_id": "node-eu-1"
}

Console flash (PRG redirect to /cloud/tenants?flash=...):
  Suspended 'acme' — PLANNED: stop the tenant DMS from serving (data + seed preserved; resume re-provisions in place)

cloud.resume
ctx  = { user: "op", role: "admin" }
input = { id: "acme" }

{
  "tenant_id": "acme",
  "simulated": "PLANNED: restart the tenant DMS in place (data + seed preserved)",
  "status": "active",
  "mode": "standalone",
  "threads": 8,
  "mem_gb": 16,
  "storage_gb": 50,
  "replicas": 1,
  "node_id": "node-eu-1"
}

Calling cloud.resume a second time now returns an error envelope:
  code = InvalidInput, message = "tenant is not suspended"

Worked example: terminate#

Terminate must be run as a custodian. The same tenant acme (8 threads / 16 GB / 50 GB, 1 replica, on node-eu-1) is terminated below. Note the response still reports status terminated and the node it was on; the side effect is that node-eu-1's alloc_threads drops by 8 and acme's subscription flips to cancelled.

cloud.terminate
ctx  = { user: "op", role: "custodian" }
input = { id: "acme" }

{
  "tenant_id": "acme",
  "simulated": "PLANNED: wipe the tenant's data + seed (irreversible); node capacity released",
  "status": "terminated",
  "mode": "standalone",
  "threads": 8,
  "mem_gb": 16,
  "storage_gb": 50,
  "replicas": 1,
  "node_id": "node-eu-1"
}

If an admin (not custodian) attempts the same call:
  code = AccessDenied, message = "cloud role 'custodian' or higher required"

After this call the placed node's allocation has been released. Verifying via a planner read shows the freed capacity:

SELECT id, alloc_threads, alloc_mem_gb, alloc_storage_gb FROM _cp_nodes WHERE id = "node-eu-1"

[ { "id": "node-eu-1", "alloc_threads": 0, "alloc_mem_gb": 0, "alloc_storage_gb": 0 } ]

(values shown assume acme was the only tenant on the node; deltas are clamped at 0)

Error conditions#

Audit trail#

Every lifecycle action appends a row to _cp_audit (best-effort — an audit write never blocks the action, which already committed through the funnel). Each row records the operator (from ctx.user), the action, the tenant id, the node id, and the result. This is how the control plane keeps a tamper-evident history of who suspended, resumed, or terminated which tenant.

suspend

action: "suspend", result: "suspended"

resume

action: "resume", result: "active" or "pending_capacity"

terminate

action: "terminate", result: "terminated"