System catalogs: config-as-data (_sys_*)

The idea: config as data#

The whole DMS is one executor plus a function registry. A worker is just an orchestrator that maps an HTTP/WebSocket request to functions and returns a uniform envelope. None of that orchestration is hard-coded per deployment — it is described by rows in the _sys_* system tables and loaded at boot. To deploy a new worker, route, function, or scheduled job you simply INSERT a row; to change one you UPDATE it; to remove one you DELETE it. There is no separate config file format and no redeploy step for most changes.

There is exactly ONE thing that is NOT data: the DMS's own app — the trusted system worker that serves /api/qql (and the bundled UI/docs routes). It is hard-coded in the binary and can never be defined or overridden by a _sys_* row. If a _sys_workers row uses the reserved id system.worker, the loader ignores it and logs a warning. Everything else is configuration stored as rows.

The catalog at a glance#

The first five tables (workers, routes, groups, functions, jobs) are the deploy surface: writing them hot-reloads the running server. The auth catalogs (users, sessions, roles, api_keys) are consulted by the security functions at request time. The Quill catalogs (pages, assets, theme) feed the page-serving path. The catalog tables (tables, indexes) make your own table/index definitions durable so they replay from the WAL on restart.

How the catalogs are created and loaded on boot#

On startup the launcher calls ensure_system_tables(&db), which idempotently creates every _sys_* table if it is absent. It then calls db.restore_catalog() to re-create your user tables and indexes from _sys_tables / _sys_indexes (replaying the WAL on a durable start), registers the built-in Quill handlers, seeds the demo + docs pages, wires Qirava Studio, and finally loads the data-defined config: user functions via db.load_functions(), user workers via host.load_from_system(&db), and scheduled jobs via the JobScheduler.

// src/bin/qdms.rs (boot order, abbreviated)
ensure_system_tables(&db);     // create _sys_* if absent (idempotent)
db.restore_catalog();          // re-create user tables + indexes from the catalog
// ... register Quill handlers, seed docs/studio routes ...
let user_fns = db.load_functions();              // _sys_functions -> registry
let user_workers = host.load_from_system(&db)?;  // _sys_workers/routes/groups -> live workers
host.watch_config(Arc::clone(&db));              // hot-reload on a _sys_* write

The exact set of tables ensured on boot is the list passed to ensure_system_tables: _sys_workers, _sys_routes, _sys_groups, _sys_api_keys, _sys_functions, _sys_jobs, _sys_tables, _sys_indexes, _sys_users, _sys_sessions, _sys_roles, _sys_pages, _sys_assets, and _sys_theme.

The deploy loop: write config, hot-reload, no restart#

Every write to a _sys_* table bumps an internal config version counter. Two background pollers (started at boot) watch that counter once per second: WorkerHost::watch_config rebuilds the data-defined workers and reloads user functions, and the JobScheduler reloads job rows. So the deploy loop is: write the row via QQL, and within about a second the running server picks up the change with no restart.

   QQL write to a _sys_* table
            |
            v
   config_version bumped  (engine increments on every _sys_* mutation)
            |
   +--------+-----------------------------+
   |                                       |
   v                                       v
 WorkerHost::watch_config (1s poll)   JobScheduler poller (1s poll)
   - reload_from_system  (workers,       - reload (re-read _sys_jobs,
     routes, groups, scope-checked)        preserve in-memory timing)
   - reload_functions    (_sys_functions)
   - serve_pending       (bind new workers)

_sys_workers — the network apps#

One row per worker: a network-bound app that fronts a set of routes. The loader (build_system_workers) reads these columns:

id

Unique worker id (the foreign key routes/groups/jobs reference). The reserved value system.worker is ignored — you cannot redefine the DMS's own app.

address

host:port the worker binds. Fixed at bind time; an address change needs a re-bind.

app

The app/namespace id used for scope checks (each route's handler must be in this app's scope).

dbs

CSV of database names the app may touch.

domains

Optional CSV of Host values the worker accepts (virtual hosting). Empty = any host.

# INSERT a worker definition via QQL
curl -s http://127.0.0.1:7191/api/qql \
  --data 'INSERT INTO _sys_workers { id: "api", address: "127.0.0.1:7192", app: "shop", dbs: "shop", domains: "*" }'

{"error":null,"data":[{"_id":1,"id":"api","address":"127.0.0.1:7192","app":"shop","dbs":"shop","domains":"*"}],"root":{"count":1}}

Within ~1s the launcher logs:
[worker] hot-reloaded 1 worker(s), 0 fn(s) (config v44)
  serving: api on http://127.0.0.1:7192

_sys_routes — method + path → handler#

One row per route, bound to a worker by the worker column. Built by route_from_record:

id

Route id (referenced by group routes CSV).

worker

FK to _sys_workers.id.

method

HTTP method, or * for any.

path

Path pattern (supports {param} segments).

handler

The function KEY the route dispatches to (e.g. qql, or a user function key, or a Quill handler like qq.serve_static).

query

Optional QQL template for the qql handler (with {param} interpolation from path/query).

role

Optional required role for the route.

output

Optional output kind: Html | Record | RawJson | Text | Passthrough. Omitted = default JSON envelope. Html/etc. bind native SSR output instead.

curl -s http://127.0.0.1:7191/api/qql \
  --data 'SELECT id, worker, method, path, handler, output FROM _sys_routes LIMIT 3'

{"error":null,"data":[{"id":"index","worker":"system.worker","method":"GET","path":"/","handler":"qq.render.index","output":"Html"},{"id":"islands","worker":"system.worker","method":"GET","path":"/islands","handler":"qq.render.islands","output":"Html"},{"id":"ssg","worker":"system.worker","method":"GET","path":"/ssg","handler":"qq.serve_static","output":"Html"}],"root":{"count":3}}

curl -s http://127.0.0.1:7191/api/qql \
  --data 'INSERT INTO _sys_routes { id: "adults", worker: "api", method: "GET", path: "/adults", handler: "qql", query: "SELECT name FROM users WHERE age >= {min}" }'

{"error":null,"data":[{"_id":1,"id":"adults","worker":"api","method":"GET","path":"/adults","handler":"qql","query":"SELECT name FROM users WHERE age >= {min}"}],"root":{"count":1}}

_sys_groups — before/after function chains#

A group attaches a chain of functions to a set of routes, running them before or after the handler. This is how auth, validation, logging, and similar cross-cutting logic is wired as data. Built in build_system_workers:

id

Group id.

worker

FK to _sys_workers.id.

kind

before (default) or after — when the chain runs relative to the handler.

mode

chain (default, ChainMode::Chain) or all (ChainMode::All) — chain passes context through the functions in order; all runs them as a set.

routes

CSV of _sys_routes ids the group covers.

fns

CSV of function KEYs to run (each becomes a waited Bound in the chain).

curl -s http://127.0.0.1:7191/api/qql \
  --data 'INSERT INTO _sys_groups { id: "apikey", worker: "api", kind: "before", mode: "chain", routes: "adults", fns: "auth.api_key,auth.check_table_grant" }'

{"error":null,"data":[{"_id":1,"id":"apikey","worker":"api","kind":"before","mode":"chain","routes":"adults","fns":"auth.api_key,auth.check_table_grant"}],"root":{"count":1}}

_sys_functions — user functions in QQL#

A user function is a named QQL statement registered as a public handler. Qdb::load_functions (and reload_functions on hot-reload) reads every row and registers each enabled one. Columns read by apply_functions:

key

The function key (how routes/jobs reference it). Required.

definition

The QQL body. Required. Supports legacy {param} text interpolation (deprecated) and injection-safe :name bound parameters resolved as typed values from the call record.

scope

Optional app scope (used for the worker scope check).

enabled

Set to false to skip registration (the row stays as data but the function is unregistered).

curl -s http://127.0.0.1:7191/api/qql \
  --data 'INSERT INTO _sys_functions { key: "shop.cleanup", definition: "DELETE FROM carts WHERE updated < 0", scope: "shop", enabled: true }'

{"error":null,"data":[{"_id":1,"key":"shop.cleanup","definition":"DELETE FROM carts WHERE updated < 0","scope":"shop","enabled":true}],"root":{"count":1}}

On hot-reload the registry is made to match the catalog exactly: each present row is overwritten in place (no transient missing window) and only keys that vanished from the catalog are unregistered. Disabled rows (or rows with an empty key or definition) are skipped.

_sys_jobs — scheduled jobs (data)#

One row binds a function key to a schedule, run by the JobScheduler. The full mechanics — interval semantics, the deferred cron kind, the overlap policy, writer-gating across a cluster, and the privilege gate — are documented on the Scheduler & jobs page. The columns are: id, worker (app scope FK), fn (the key to run), schedule_kind (interval | cron), schedule (seconds for interval), args (optional encoded input), enabled, last_run_us, next_run_us (microsecond run timestamps mirrored back onto the row).

curl -s http://127.0.0.1:7191/api/qql --data 'SELECT * FROM _sys_jobs'

{"error":null,"data":[{"_id":1,"id":"_builtin.ttl_sweep","worker":"","fn":"db.ttl_sweep","schedule_kind":"interval","schedule":"60","enabled":true,"last_run_us":1782569091193872,"next_run_us":1782569151193872}],"root":{"count":1}}

_sys_api_keys — machine credentials#

API keys for the qql surface. Only the SHA-256 HASH of the key is stored, never the plaintext. Each key carries a scope and per-table grants. The auth.api_key before-function hashes the presented key (from the X-API-Key header or an Authorization: Bearer token), looks it up here, and injects its scope + grants into the shared context; auth.check_table_grant then enforces table-level RBAC deny-by-default.

You never INSERT into this table directly. Mint a key with the admin function auth.create_api_key, which generates a random secret, stores only its hash + scope, and returns the plaintext exactly once. Grant/revoke a table on an existing key with auth.grant_table (grant = "none" revokes). Admin keys bypass table grants; non-admin keys need an explicit table:read | write | read_write grant per table they touch.

curl -s -X POST http://127.0.0.1:7191/api/qql \
  -H 'X-API-Key: <your-key>' \
  --data 'SELECT * FROM users'

# or with a bearer token:
curl -s -X POST http://127.0.0.1:7191/api/qql \
  -H 'Authorization: Bearer <your-key>' \
  --data 'SELECT * FROM users'

_sys_users / _sys_sessions / _sys_roles — human governance#

These three catalogs back the human management surface (Qirava Studio) and the custodian/admin governance model. They store only hashes, never plaintext secrets.

_sys_users

{ id, username, pwd_hash (PBKDF2), role, db_grants, enabled }. role ∈ {custodian | admin | user | guest}; db_grants is a CSV of db:perm pairs (perm ∈ read | write | read_write | none).

_sys_sessions

{ token_hash, user, role, created_us, expires_us }. Only the SHA-256 hash of the bearer token is stored. expires_us is the idle deadline; auth.session_extend bumps it by the 30-minute idle window on each successful request.

_sys_roles

{ name, perms } — custodian-defined RBAC role definitions.

curl -s http://127.0.0.1:7191/api/qql \
  --data 'SELECT id, username, role, enabled FROM _sys_users'

{"error":null,"data":[{"id":"021de6499638e95e","username":"custodian","role":"custodian","enabled":true}],"root":{"count":1}}

On a first install with no custodian, the launcher mints ONE bootstrap custodian and prints its random password to the console exactly once — capture it. Custodian-only governance functions then manage the rest: auth.create_user, auth.set_db_grant, auth.promote_admin, auth.create_custodian, auth.define_role. Login (auth.login) verifies credentials against _sys_users (PBKDF2), mints a random session token, stores only its hash in _sys_sessions, and returns the plaintext token once.

_sys_pages / _sys_assets / _sys_theme — the Quill cache#

The Quill (qquill) page pipeline stores its prerendered output here. They are empty until the framework's build/SSG steps populate them, after which the static/ISR serve handlers read them with zero per-request render cost.

_sys_pages

{ id, route, html, etag, css_ids, cache_policy } — prerendered HTML served by qq.serve_static / qq.serve_isr with a strong content-hash ETag and conditional 304s.

_sys_assets

{ id, path, bytes, content_hash, mime, immutable } — static assets served by the qq.asset handler.

_sys_theme

{ id, tokens, css } — design tokens and generated CSS.

curl -s http://127.0.0.1:7191/api/qql \
  --data 'SELECT id, route, etag, cache_policy FROM _sys_pages LIMIT 2'

{"error":null,"data":[{"id":null,"route":"/ssg","etag":"\"92baeba7fdb9e20a66592bfe3274b259b3b53649633911bd385b59d9e6f8fa7a\"","cache_policy":"public, max-age=31536000, immutable"},{"id":null,"route":"/docs","etag":"\"5343efdc8c94e6dd045b44c90010d406a9a096ed68b8304b3a4ee4ba1d58108d\"","cache_policy":"public, max-age=31536000, immutable"}],"root":{"count":2}}

The response envelope you'll see#

The qql routes use Passthrough output, so a /api/qql request returns the engine's own QQL JSON envelope directly (not the worker meta-envelope). Its shape is always {"error":..., "data":..., "root":{"count":N}} where exactly one of error/data is meaningful, data is a list of row maps (or a single aggregate map), and count is the number of rows. On error, error is {"code":..., "message":...} and data is null.

curl -s http://127.0.0.1:7191/api/qql \
  --data 'SELECT count(*) FROM _sys_pages'

{"error":null,"data":[{"count_all":25}],"root":{"count":1}}

Use count(*) (aliased count_all). A bare count() without an argument is a parse error:
{"error":{"code":"bad_request","message":"expected identifier, found Some(RParen)"},"data":null,"root":{"count":0}}

Who may write the catalogs#

1. Write config via QQL against the qql endpoint with an admin credential.
2. The engine bumps config_version on the write.
3. The background watchers (worker host + job scheduler) reload within ~1s.
4. New workers bind automatically; route/handler/function/job changes apply live; only an address change needs a re-bind.