Per-function scheduler: interval & daily cron

Two schedulers, one idea#

Both schedulers are an OUTWARD layer over the executor: when a job is due they call runtime.execute(...) (or execute_detached) on the function key, exactly the same path an HTTP handler uses, so the resource budget governs scheduled work identically to request-driven work.

JobScheduler (data-driven)

Reads jobs as rows from _sys_jobs. This is the production path the launcher wires up. Schedule kinds: interval (seconds) and cron (reserved/deferred). Adds overlap-skip, writer-gating, a privilege gate, and hot-reload. Ticks once per second in microseconds.

Scheduler (in-code)

A lower-level scheduler built from Schedule values in Rust. Supports EverySecs intervals AND Daily-at-time runs, each with day-of-week whitelist/blacklist and date blackout. Ticks once per second in whole seconds. Use it when you need daily/cron-style filtering today.

In a default DMS deployment you interact with the JobScheduler by writing _sys_jobs rows. The in-code Scheduler is the building block for richer schedules and is fully documented below because it is the only path that implements daily/day-filter semantics.

The data-driven path: _sys_jobs#

A job is one row in _sys_jobs. Each row binds a function KEY to a schedule, scoped to a worker's app (an FK to _sys_workers, like a route). On every tick the JobScheduler reads the enabled jobs whose next_run_us is due, executes the bound key through the executor, and advances last_run_us / next_run_us both in memory (authoritative) and back onto the row (for observability).

interval schedules#

An interval job fires every N seconds, where N is the schedule value parsed as an integer. The minimum interval is MIN_INTERVAL_SECS = 1 second. An interval that parses to 0 would fire on every tick (a busy-loop / accidental DoS), so it is REJECTED and the job is treated as disabled, logged once. A nonzero value below the floor is clamped UP to 1 second. A schedule that fails to parse is treated as disabled.

When a job becomes due, its next_run_us is advanced to now + interval (in microseconds) before the function is dispatched, and the new timestamps are persisted to the row. A newly inserted job with next_run_us = 0 is due immediately on the first tick after it loads.

# Create a job that runs shop.cleanup every hour (3600s). next_run_us:0 = due now.
curl -s http://127.0.0.1:7191/api/qql \
  --data 'INSERT INTO _sys_jobs { id: "shop.cleanup.hourly", worker: "api", fn: "shop.cleanup", schedule_kind: "interval", schedule: "3600", enabled: true, last_run_us: 0, next_run_us: 0 }'

{"error":null,"data":[{"_id":2,"id":"shop.cleanup.hourly","worker":"api","fn":"shop.cleanup","schedule_kind":"interval","schedule":"3600","enabled":true,"last_run_us":0,"next_run_us":0}],"root":{"count":1}}

After the next tick fires it (and the function shop.cleanup is a registered _sys_functions key), the row's timing has advanced — next_run_us = last_run_us + 3600s:

  SELECT id, last_run_us, next_run_us FROM _sys_jobs WHERE id = "shop.cleanup.hourly"
  {"error":null,"data":[{"id":"shop.cleanup.hourly","last_run_us":1782569119198493,"next_run_us":1782572719198493}],"root":{"count":1}}
  # 1782572719198493 - 1782569119198493 = 3,600,000,000 us = 3600 s

The built-in TTL-sweep job#

The engine's TTL cleanup is itself a job, seeded as data so TTL and custom maintenance run through ONE mechanism. On boot the launcher calls JobScheduler::seed_builtin_ttl(&db, 60), which idempotently inserts a row with id _builtin.ttl_sweep binding the privileged key db.ttl_sweep to a 60-second interval. It is a no-op if the row already exists, and it bumps config_version so a running watcher hot-reloads it.

curl -s http://127.0.0.1:7191/api/qql --data 'SELECT * FROM _sys_jobs'

{"error":null,"data":[{"_id":1,"id":"_builtin.ttl_sweep","worker":"","fn":"db.ttl_sweep","schedule_kind":"interval","schedule":"60","enabled":true,"last_run_us":1782569091193872,"next_run_us":1782569151193872}],"root":{"count":1}}

cron schedules (reserved / deferred)#

The schedule_kind value cron is RESERVED: the column and dispatch arm exist so a row can declare it today, but parsing is deferred. A cron job is currently treated as DISABLED (never due) and logged once on load. The schema will not change when a minimal 5-field (m h dom mon dow) parser lands.

curl -s http://127.0.0.1:7191/api/qql \
  --data 'INSERT INTO _sys_jobs { id: "demo.cron", worker: "api", fn: "shop.cleanup", schedule_kind: "cron", schedule: "0 3 * * *", enabled: true, last_run_us: 0, next_run_us: 0 }'

{"error":null,"data":[{"_id":3,"id":"demo.cron","worker":"api","fn":"shop.cleanup","schedule_kind":"cron","schedule":"0 3 * * *","enabled":true,"last_run_us":0,"next_run_us":0}],"root":{"count":1}}

The launcher logs once and never fires the job:
[jobs] job `demo.cron`: schedule_kind=cron is not yet supported (interval-only MVP); skipping

For daily/cron-style scheduling today, use the in-code Scheduler's Daily mode (below).

How a tick decides what is due (JobScheduler)#

A background thread ticks the scheduler once per second with the current wall-clock time in microseconds, and reloads job definitions whenever config_version changes. The tick is deterministic and unit-testable (the function body runs asynchronously on the executor).

1. Writer gate first: if this node is not the writer (a Follower), the tick fires nothing and returns 0.
2. Under the jobs lock, collect every job that is enabled, has a supported interval, is due (now_us >= next_run_us), and is NOT already in flight. Mark each in-flight, set last_run_us = now and next_run_us = now + interval, and record the new timestamps to persist.
3. Release the lock, then persist the new timestamps to each row (never holding the lock across a db write).
4. For each due job apply the privilege gate, then dispatch via runtime.execute_detached with a stable per-job execution key. The completion callback (fires exactly once) clears the in-flight flag.

// run_jobs_background: the 1-second loop
let mut last_cfg = db.config_version();
loop {
    let cfg = db.config_version();
    if cfg != last_cfg { last_cfg = cfg; scheduler.reload(&db); } // hot-reload
    let now_us = SystemTime::now().duration_since(UNIX_EPOCH)
        .map(|d| d.as_micros() as i64).unwrap_or(0);
    scheduler.tick(now_us);
    std::thread::sleep(Duration::from_secs(1));
}

Overlap policy: SKIP#

A job NEVER starts a new run while its previous run is still in flight. Each job carries an in_flight flag set before execute_detached and cleared in the completion callback; a due job whose flag is set is skipped this tick (and becomes eligible again once the prior run completes). This is chosen over queueing so a slow job can't build an unbounded backlog of pending runs. The dispatch is fire-and-forget: the tick does not wait for the function to finish — it returns the number of runs STARTED.

Cluster safety: writer-gated#

Job execution is gated on is_writer(). Only a Standalone or Master (the leader/writer) runs jobs; a Follower's gate returns false, so it ticks but fires nothing — jobs never double-run across a replicated cluster. The gate is a closure so the replication Role is threaded in without the scheduler depending on the repl module. The launcher reports the state on startup:

  sched:   _sys_jobs (data, hot-reload); built-in db.ttl_sweep every 60s
  # on a follower, the line ends with:  [follower: jobs paused]

For a standalone node you can build an always-writer scheduler with JobScheduler::standalone(db, runtime), which passes a gate of || true.

The privilege gate (deny-by-default)#

A DATA-defined (non-trusted) job may only run a key that is BOTH (a) not privileged and (b) a registered app function present in _sys_functions. Trusted jobs (members of TRUSTED_JOB_IDS, e.g. the TTL sweep) bypass this. If a data job names a privileged or unknown key, the scheduler clears the in-flight flag it optimistically set, SKIPS the run, and logs once. This blocks the privilege-escalation CVE class where a tenant row drives a privileged handler.

Privileged keys are the auth.* and db.* and system.* namespaces plus an explicit deny list: auth.create_api_key, auth.grant_table, auth.api_key, auth.check_table_grant, auth.require_role, db.ttl_sweep, the qql handler, db.read, and db.mutate.

curl -s http://127.0.0.1:7191/api/qql \
  --data 'INSERT INTO _sys_jobs { id: "evil", worker: "api", fn: "auth.create_api_key", schedule_kind: "interval", schedule: "1", enabled: true, last_run_us: 0, next_run_us: 0 }'

{"error":null,"data":[{"_id":4,"id":"evil","worker":"api","fn":"auth.create_api_key","schedule_kind":"interval","schedule":"1","enabled":true,"last_run_us":0,"next_run_us":0}],"root":{"count":1}}

The row inserts (it is data), but every tick refuses to run it — and no API key is ever minted:

[jobs] job `evil`: refusing to run key `auth.create_api_key` — a data-defined job may not run a privileged/unknown function (deny-by-default)

  SELECT count(*) FROM _sys_api_keys
  {"error":null,"data":[{"count_all":0}],"root":{"count":1}}

Hot-reload of job definitions#

Jobs are DATA: reload(db) re-reads _sys_jobs and reconciles by id. Jobs that still exist keep their in-memory runtime timing (last_run_us / next_run_us / in_flight) so a reload doesn't reset cadence or break the overlap policy mid-run; only the definition fields (worker, key, args, enabled, interval, trusted) are refreshed. New rows are added (seeding timing from the row, or 0 = due now), and rows that were deleted are dropped. The reload is triggered by a config_version change, exactly like the worker reload.

curl -s http://127.0.0.1:7191/api/qql \
  --data 'UPDATE _sys_jobs SET enabled = false WHERE id = "shop.cleanup.hourly"'

{"error":null,"data":[{ ... "id":"shop.cleanup.hourly", "enabled":false ... }],"root":{"count":1}}

Within ~1s the watcher reloads and the job stops firing while its row (and its history in last_run_us) is preserved. Set enabled = true to resume, or DELETE the row to drop it entirely.

The in-code Scheduler: interval + daily with day/date filters#

The lower-level Scheduler is the only path that implements daily-at-time runs and day/date filtering. A job is a (key, input, Schedule, seeded-now) tuple added with .add(...); a background thread (run_background) ticks it once per second in whole seconds, and tick(now) is a pure, deterministic function returning how many jobs fired.

A Schedule has a repeat mode plus three filters that apply to BOTH modes:

repeat: EverySecs(n)

Fire on every interval of n seconds. Timing is seeded from the now passed to add(), so an EverySecs job first fires after its interval, not instantly.

repeat: Daily

Fire once per listed time-of-day. at_secs_of_day is a Vec of seconds-since-midnight UTC trigger points; each slot fires at most once per UTC day (tracked by fired_slots, cleared at midnight).

days_only

Run ONLY on these weekdays (0=Sun..6=Sat). Empty = any day.

days_skip

Never run on these weekdays.

dates_skip

Never run on these (year, month, day) UTC dates — holidays/blackouts.

use qdms::workers::{Scheduler, Schedule};
use std::sync::Arc;

let mut sched = Scheduler::new(runtime.clone());
let now = /* unix seconds */ 0;

// Every 5 minutes, but only on weekdays (Mon..Fri = 1..5), never on a holiday.
sched.add(
    "app.metrics.flush",
    Vec::new(),                 // encoded input
    Schedule::every_secs(300)
        .days_only(vec![1,2,3,4,5])
        .dates_skip(vec![(2026, 12, 25)]),
    now,
);

// Daily at 02:00 and 14:00 UTC (seconds-since-midnight), skipping Sundays.
sched.add(
    "app.report.daily",
    Vec::new(),
    Schedule::daily_at(vec![2*3600, 14*3600]).days_skip(vec![0]),
    now,
);

qdms::workers::scheduler::run_background(Arc::new(sched)); // 1s tick loop

Day and date filtering use a dependency-free civil-date calculation (Howard Hinnant's algorithm) over days-since-epoch, so there is no third-party date library. Builder methods days_only / days_skip / dates_skip chain onto every_secs(...) and daily_at(...).