Scheduling
A pipeline can run itself on a cron cadence. A single in-process goroutine ticks, finds what's due, and fires it through the exact same orchestration a manual run gets. An external heartbeat backstops the ticker so schedules still fire on a host that fell asleep.
Scheduling is deliberately boring where it counts: no in-memory queue, no bespoke run path, no second source of truth. A schedule is a row (cron, enabled, next_run_at); a tick is a query for due rows plus a fire; and a fire is the same orchestrateRun a human triggers. The only genuinely new machinery is the cron math and the two clocks that drive the tick — an in-process ticker for latency and an external heartbeat for liveness.
The cron engine
Setu uses standard 5-field cron (minute hour day-of-month month day-of-week), parsed by robfig/cron/v3. One parser instance, configured for exactly those five fields, backs every cron operation — validation, next-fire computation, and the multi-fire preview — so the console builder, the PUT endpoint, and the scheduler can never disagree about what an expression means.
// cronParser accepts standard 5-field expressions (minute hour dom month dow).
var cronParser = cron.NewParser(
cron.Minute | cron.Hour | cron.Dom | cron.Month | cron.Dow,
)
// NextRun returns the next fire time after "from". Shared with the API so schedule
// edits compute next_run_at the same way the scheduler does.
func NextRun(expr string, from time.Time) (time.Time, error) {
sched, err := cronParser.Parse(expr)
if err != nil { return time.Time{}, err }
return sched.Next(from), nil
}
func Valid(expr string) bool { _, err := cronParser.Parse(expr); return err == nil }
// NextRuns returns the next n fire times — powers the builder's live preview.
func NextRuns(expr string, from time.Time, n int) ([]time.Time, error) { /* loops sched.Next */ }The builder's preview below and the server's scheduling decision are the same function. As you type, the console calls the preview endpoint, which runs NextRuns(expr, now, 3) — the identical sched.Next the scheduler will call at tick time. The preview cannot lie because there is no second implementation to drift from. Try it:
/pipelines/{id}/schedule/preview?cron=<expr>memberValidates cron and returns {valid, next: [t1, t2, t3]}. An invalid expression returns {valid:false, next:[]} rather than an error — the builder just shows “invalid” as you type.
/pipelines/{id}/scheduleeditorCreates or replaces the schedule. It validates the expression (rejecting anything that isn't 5 fields) and computes next_run_at from NextRun(cron, now) — or leaves it null when the schedule is disabled, so a disabled schedule is simply never due.
if !scheduler.Valid(req.Cron) {
s.fail(w, http.StatusBadRequest, errors.New("invalid cron expression (expected 5 fields: minute hour day month weekday)"))
return
}
next := pgtype.Timestamptz{} // NULL when disabled → never selected by ListDueSchedules
if req.Enabled {
t, err := scheduler.NextRun(req.Cron, time.Now())
if err != nil { s.fail(w, http.StatusBadRequest, err); return }
next = pgtype.Timestamptz{Time: t, Valid: true}
}
s.q.UpsertSchedule(r.Context(), sqlc.UpsertScheduleParams{
PipelineID: id, Cron: req.Cron, Enabled: req.Enabled, DryRun: req.DryRun, NextRunAt: next, WorkspaceID: ws,
})The in-process ticker
The scheduler is one goroutine. It ticks every 30 seconds, fires an initial tick on startup (so schedules that came due while the server was down are caught immediately), and stops when its context is cancelled. All state lives in Postgres — schedules survive restarts, and there is no in-memory queue to lose.
func (s *Scheduler) Run(ctx context.Context) {
t := time.NewTicker(s.interval) // 30s
defer t.Stop()
s.tick(ctx) // prompt initial tick: catch schedules that came due while down
for {
select {
case <-ctx.Done():
return
case <-t.C:
s.tick(ctx)
}
}
}
// TickOnce runs a single pass on demand — the external heartbeat calls this so due
// schedules still fire when the periodic goroutine is paused. State is all in
// Postgres, so an on-demand tick and the periodic one are equivalent.
func (s *Scheduler) TickOnce(ctx context.Context) { s.tick(ctx) }
func (s *Scheduler) tick(ctx context.Context) {
due, _ := s.q.ListDueSchedules(ctx) // next_run_at <= now
now := time.Now()
for _, sc := range due {
next, err := NextRun(sc.Cron, now)
if err != nil {
// A bad expression should never fire forever — log and skip advancing.
s.log.Error("scheduler: bad cron", "pipeline", sc.PipelineID, "cron", sc.Cron, "err", err)
continue
}
fired, err := s.disp.FireSchedule(ctx, sc.PipelineID, sc.DryRun)
// Advance the cadence REGARDLESS: a skipped or failed fire must not stall the
// schedule; the next occurrence runs once the runtime is back.
s.q.AdvanceSchedule(ctx, sqlc.AdvanceScheduleParams{PipelineID: sc.PipelineID, NextRunAt: /* next */})
s.log.Info("scheduler: ticked", "pipeline", sc.PipelineID, "fired", fired, "next", next)
}
}Note the one place a schedule does not advance: an unparsable cron. If the expression is somehow bad (e.g. a hand-edited row), NextRun errors and the tick continues before advancing — leaving next_run_at in the past. That is intentional: a bad schedule that keeps showing up in the logs is a signal to fix it, and since it never fires it does no harm. Every other outcome — fired, skipped, or a dispatch error — still advances.
AdvanceSchedule. A schedule that couldn't fire must not tight-loop on the same due row every 30 seconds — it simply moves to its next occurrence and tries again then.Why scheduled runs go through orchestrateRun
A scheduled run is not pushed straight onto the worker queue. It goes through Dispatcher.FireSchedule, which applies the same placement-aware orchestration a manual run gets. This matters: enqueuing directly onto the worker would send an agent-placed run to the cloud, which can't reach the edge source or destination. Routing through FireSchedule → orchestrateRun means a scheduled agent run dispatches an agent job exactly like a manual one.
The seam is a one-method interface. The scheduler package knows nothing about placement, agents, or the worker queue — it only knows how to find due rows and call FireSchedule. The apiserver implements that interface, so all the routing logic lives in one place and is shared with manual runs:
// Dispatcher fires a due pipeline through the SAME placement-aware orchestration a
// manual run gets. fired=false (no run created) when the bound runtime is offline —
// the dispatcher notifies the owners instead of leaving a failed run for every tick.
type Dispatcher interface {
FireSchedule(ctx context.Context, pipelineID uuid.UUID, dryRun bool) (fired bool, err error)
}func (s *Server) FireSchedule(ctx context.Context, pipelineID uuid.UUID, dryRun bool) (bool, error) {
pipe, _ := s.q.GetPipelineForWorker(ctx, pipelineID)
src, _ := s.q.GetConnection(ctx, /* source */)
dst, _ := s.q.GetConnection(ctx, /* dest */)
placement := derivePlacement(src.Placement, /* … */, dst.Placement, /* … */)
// Skip-when-offline: if the stage that runs FIRST on a runtime is offline,
// don't create a run destined to fail — notify the owners and return fired=false.
if !placement.Extract.Cloud {
if reachable, name := s.agentReachable(ctx, ws, placement.Extract.AgentID); !reachable {
s.notifyScheduleSkipped(ctx, ws, pipe, name)
return false, nil
}
} else if !placement.Load.Cloud {
if reachable, name := s.agentReachable(ctx, ws, placement.Load.AgentID); !reachable {
s.notifyScheduleSkipped(ctx, ws, pipe, name)
return false, nil
}
}
run, _ := s.q.CreateRun(ctx, sqlc.CreateRunParams{PipelineID: pipelineID, DryRun: dryRun})
return true, s.orchestrateRun(ctx, ws, pipe, run) // identical routing to a manual run
}Skip when the runtime is offline
A manual run and a scheduled run diverge on exactly one thing: what to do when the bound agent is offline. This is the one place where “same orchestration” is deliberately not quite the same, and the reason is the human in the loop.
| Manual run | orchestrateRun calls failRunUnreachableAgent — it creates a failed run with a fixable message (“Start it with setu run on its host, then re-run”). A human is watching, so a visible failure is useful. |
|---|---|
| Scheduled run | FireSchedule creates no run at all. It notifies the workspace owners once and returns fired=false. A schedule firing every few minutes against an offline agent would otherwise pile up a wall of failed runs. |
Reachability itself is shared: agentReachable checks the agent is active and checked in within the last 90 seconds (agentOnlineWindow). Only the response to “not reachable” differs. The skip notification goes to owners only, and is best-effort — a failure to notify is logged, never fatal:
members, _ := s.q.ListWorkspaceMembers(ctx, ws)
body := fmt.Sprintf(
"%q didn't run — its runtime %q is offline. It runs on the next schedule once the runtime is back.",
pipe.Name, agentName,
)
for _, m := range members {
if m.Role != "owner" { continue } // owners only
s.notify(ctx, s.q, m.UserID, &ws, "schedule.skipped", "Scheduled run skipped", body,
map[string]any{"pipelineId": pipe.ID.String()})
}The external heartbeat
The in-process ticker has one blind spot: on a free-tier host that sleeps between requests, the goroutine is paused too — so nothing ticks until the next inbound request wakes the process. The fix is an external heartbeat that POSTs a tick endpoint on a schedule. The request itself wakes the sleeping host, and the endpoint runs one scheduling pass. Because scheduler state is entirely in Postgres, an on-demand tick and the periodic one are equivalent.
/internal/scheduler/tickBearer SETU_TICK_TOKENForces one TickOnce. Returns 204 No Content on success, 404 when the token env is unset, 401 on a bad token.
// POST /internal/scheduler/tick — bearer token from SETU_TICK_TOKEN.
func (s *Server) schedulerTick(w http.ResponseWriter, r *http.Request) {
want := strings.TrimSpace(os.Getenv("SETU_TICK_TOKEN"))
if want == "" {
http.NotFound(w, r) // no token configured → endpoint doesn't exist
return
}
got := strings.TrimPrefix(r.Header.Get("Authorization"), "Bearer ")
if subtle.ConstantTimeCompare([]byte(got), []byte(want)) != 1 {
s.unauthenticated(w)
return
}
scheduler.New(s.q, s, s.log).TickOnce(r.Context()) // one pass, same as the periodic tick
w.WriteHeader(http.StatusNoContent)
}Three properties make this endpoint safe to expose on the public control plane:
- 1Closed by default. With
SETU_TICK_TOKENunset the handler returns404 Not Found— the endpoint simply doesn't exist. It is never silently left open with a blank token. - 2Constant-time comparison. The presented bearer is checked with
subtle.ConstantTimeCompare, so the check leaks no timing signal about how many leading bytes matched. - 3No privileged surface. A valid tick does exactly what the internal ticker does — nothing more. The worst a leaked token buys an attacker is firing already-due schedules slightly early.
404 Not Found unless SETU_TICK_TOKEN is set — it is never silently left open. With the token set, callers must present it as Authorization: Bearer <token>, compared in constant time.The GitHub Actions caller
A GitHub Actions workflow is the caller. It runs on a cron of every 5 minutes (GitHub's minimum granularity) and simply curls the tick endpoint with the bearer token, retrying generously so a cold start doesn't fail the ping. GitHub cron is best-effort and can be delayed under load — which is fine, because a tick just fires whatever is currently due.
on:
schedule:
- cron: "*/5 * * * *" # every 5 min (GitHub's minimum granularity)
workflow_dispatch: {}
concurrency: # don't pile up ticks if a cold-start run is still going
group: scheduler-heartbeat
cancel-in-progress: false
jobs:
tick:
runs-on: ubuntu-latest
steps:
- name: Ping scheduler tick
env:
TICK_TOKEN: ${{ secrets.SETU_TICK_TOKEN }}
API_URL: ${{ vars.SETU_API_URL }}
run: |
url="${API_URL:-https://setu-api-f776.onrender.com}"
curl -fsS -X POST "$url/internal/scheduler/tick" \
-H "Authorization: Bearer ${TICK_TOKEN}" \
--max-time 120 --retry 4 --retry-all-errors --retry-delay 10 \
-o /dev/null -w "tick -> HTTP %{http_code}\n"The concurrency group with cancel-in-progress: false keeps a slow cold-start ping from being cancelled by the next scheduled one, and --retry-all-errors plus a 120-second timeout absorbs the free tier waking up. Two clocks can both fire a tick against the same due row — that is fine, because the tick is idempotent (see below).
The full heartbeat path
End to end: GitHub cron wakes the host and hits the endpoint, which forces oneTickOnce; that finds due schedules and fires each through the placement-aware path — creating and orchestrating a run, or skipping and notifying when the bound runtime is offline.
tick over the same Postgres state, so they can never double-fire or disagree — ListDueSchedules plus AdvanceSchedule make a tick idempotent with respect to what's already been advanced. Whichever clock ticks first advances next_run_at past now, so the other finds the row no longer due.