Design decisions · Setu Docs

Every architecture is a pile of tradeoffs. What follows is the list of the ones that shaped Setu most: what was decided, why, and what it costs. Read each as decision → rationale → tradeoff. Most of them bend toward one goal, spelled out at the end.

The load-bearing split — control plane vs data plane

Before the individual choices, the separation everything else rests on. The control plane is always Setu’s cloud — pipeline definitions, schedules, cursors/watermarks, chunk manifests, run status, and the UI. It is metadata only: for edge-executed runs it never receives a single row of customer data. The data plane — the actual extract, transform, load — runs wherever the data can be reached.

Decision	Two committed topologies — A (SaaS direct: the cloud worker reaches both ends) and B (hybrid outbound-only agent inside the customer boundary). Placement lives on connections; a run's extract/load placement is derived from its bindings.
Why	Sources and destinations often sit in a private VPC, on-prem, or a compliance-restricted network the cloud cannot reach. Separating planes lets the data plane relocate into the customer's boundary while the control plane still orchestrates — and, when required, sees no row data at all.
Tradeoff	Topologies C (client push, no agent) and D (fully self-hosted / air-gapped controller) are explicit non-goals. There is no self-hosted controller in scope: the control plane is always Setu's cloud.

Topology B — the outbound-only agent

The edge runtime opens no inbound ports. It reaches the control plane by making outbound HTTPS requests and long-polling for work· the cloud never dials in.

Decision	The agent is a pure client. Work flows via an outbound long-poll (GET /agent/v1/jobs/next), an atomic lease, a heartbeat, and a posted result. A one-time enrollment token is exchanged for a persistent, hashed, revocable agent API key; all job-pull is outbound HTTPS with that key.
Why	A plain outbound HTTPS request traverses NAT, proxies, and corporate firewalls with zero config. No inbound firewall rules, no VPN, no exposed surface inside the customer network — the single biggest adoption blocker for on-prem data tools. We don't fork the data plane, we relocate it: the agent embeds the same engine/ the cloud worker uses.
Tradeoff	The server can't push. Pickup latency is bounded by the poll window, and the control plane needs a leasing + reaper protocol to hand each job to exactly one agent and reclaim work a crashed agent left behind.

Where this is spelled out

The mechanics — bounded wait, FOR UPDATE SKIP LOCKED leasing, heartbeat, reaper, exponential backoff — live in Long-polling & leasing.

Blob as the seam between cloud and edge

Data moves through an S3-compatible blob store staged as parts + a manifest. Extract writes to a run’s blob prefix· load reads it back — whether either side is the cloud worker or an edge agent.

Decision	A run is staged: extract → blob → load. Both stages address the same run prefix (ws/…/pl/…/run/…/), and the extract-spec / load-spec hand the agent the same blob backend the cloud uses. The blob interface is provider-agnostic — one S3 adapter (AWS, Supabase, R2, B2, MinIO, GCS-interop) plus a local-filesystem adapter.
Why	It makes cloud and edge symmetric and composable. Any placement — cloud→cloud, agent→cloud, cloud→agent, agent→agent — is just 'who writes the prefix' and 'who reads it'. The two halves never need to talk directly or be online at the same instant. Edge blob is configurable per agent: default a Setu-managed bucket (easy onboarding), optionally the customer's own bucket so row data never touches Setu (residency).
Tradeoff	An extra hop and a copy of the data at rest, cleaned up by a retention TTL (SETU_BLOB_RETENTION_DAYS, default 7). For multi-host deployments the blob must be real S3, not a local filesystem, so both sides can reach it. And the serialized blob.Spec carries the S3 access/secret keys, so it may only be handed to trusted, authenticated agents.

Staged execution over direct streaming

Rather than one process streaming source → sink end-to-end, a run is split into independently dispatched extract and load stages bridged by the blob.

Decision	Extract and load are separate agent_jobs / worker jobs, sequenced by the orchestrator as each result arrives (onAgentExtractResult flips the run to the load phase and dispatches load per the destination's placement).
Why	It's what makes the hybrid model work at all: the two ends can run in different places, at different times, on different hosts. It also gives natural checkpoints — the staged artifact is inspectable, and a failed load can retry from the blob without re-reading the source.
Tradeoff	More moving parts and more state to reconcile: run phases, artifact records, per-stage placement resolution, and 'don't resurrect a cancelled run with a late result' guards. Direct streaming would be simpler for the pure cloud→cloud case — which is exactly why there's an escape hatch (next).

…but small deltas stream direct — an automatic escape hatch

Staging pays for itself on large, resumable, cross-plane runs. On a tiny incremental delta it’s pure overhead, so the worker skips it automatically.

Decision	A run whose incremental delta is at or below SETU_DIRECT_MAX_ROWS (default 50,000 — roughly one blob part) streams DIRECTLY source → operators → destination, no blob. Above it, the run stages. SETU_STAGED_EXECUTION=off is a global kill switch back to the legacy single-pass copy.
Why	Small deltas don't benefit from staging's resume/replay, so the blob round-trip is dead weight. The threshold makes the common recurring-sync-of-a-few-rows case as cheap as legacy while keeping staging's guarantees where they matter.
Tradeoff	Two code paths to keep behaviorally identical, and a magic number to tune. Direct runs give up mid-run resume — acceptable precisely because they're small enough to just re-run. Setting the threshold to 0 forces every run to stage.

Cursor incremental, not CDC

Recurring syncs move only changed rows using a per-stream cursor watermark — a column value, not the source’s change log.

Decision	Incremental extraction reads only rows past a durable per-stream cursor (SQL/Mongo push the predicate down; CSV/JSON/REST filter client-side). The watermark commits only once data reaches the destination, and resume restarts from the last landed chunk — including the first big backfill.
Why	A cursor works across every connector with no source-side setup, replication slot, or elevated privilege. It gives correct incremental (inclusive boundary + idempotent upsert = no missed rows, no dupes) with a fraction of CDC's operational weight.
Tradeoff	A cursor sync sees inserts and (with an updated_at cursor) updates, but never hard deletes. NULL-cursor rows are excluded from the delta. True change-data-capture (logical replication, binlog) is a deliberate non-goal for v1.

Dataset-centric, not connector-centric

The user-facing unit is a dataset with a stable identity, not a connection plus a stream-name string. The single pluggable seam is the provider (a dataset-format driver)· Postgres is one provider among equals.

Decision	A connection is just a place (credentials/scope). Datasets are first-class: uniquely identified, typed, browsable like a filesystem (folders + tags), and referenced by pipelines by ID. One pipeline can fan over many source datasets. This was a ground-up, greenfield rewrite of the abstraction.
Why	The old model conflated the driver code with the user-facing unit — datasets were ephemeral strings with no identity or operations. Making datasets first-class turns Setu into 'a data catalog you act from': manage your estate, then move it. Pipelines binding to IDs (not name strings) survive renames and make where-used real.
Tradeoff	Greenfield meant no migration path — the schema was reshaped freely and dev data reseeded. The engine internals (record, operator, transform/expr, the runner copy loop) were carried over verbatim; only the connector layer was reimplemented as providers.

Postgres + River, no Kafka in v1

The job queue is River on the same Postgres that holds metadata — no separate message broker.

Decision	Cloud jobs run through River (Postgres-backed). Agent jobs are a plain table leased with FOR UPDATE SKIP LOCKED. Scheduler state is entirely in Postgres too.
Why	One durable store to run, back up, and reason about. Transactions span 'create the run' and 'enqueue the job' atomically. At Setu's scale (coarse, batch-shaped migration jobs) Postgres is comfortably enough — and it removes an entire operational component.
Tradeoff	Postgres isn't a firehose. Very high-throughput, fan-out streaming would eventually want a real log (Kafka/Redpanda). That's a deliberate 'not yet', not a 'never' — the seam is the job abstraction, which could be re-backed later.

Secrets resolved where the connection lives

Connection config is stored with ${secret.NAME} references, never plaintext. Cloud connections resolve against the encrypted cloud vault· agent-placed connections resolve on the edge, from the runtime’s own local store.

Decision	Every spec handed to an agent (extract-spec, load-spec, job payloads) carries config with secret refs INTACT. The cloud never resolves an edge secret; the agent runs `setu secret set` locally (encrypted file, pgpass, token, or Vault).
Why	Edge credentials — a database password inside the customer's network — should never leave that network. Keeping refs unresolved means the control plane can orchestrate a connection it structurally cannot read.
Tradeoff	Secrets are managed in two places (cloud vault + each edge host), and a misconfigured edge secret only surfaces at run time on the runtime, not when you save the connection in the console.

Cross-site cookies + authenticated downloads

The console and API are served from different sites, so the session cookie is SameSite=None; Secure with an explicit CORS allowlist, and file downloads go through credentialed fetches instead of naked links.

cookiePolicy — cross-site flips SameSite=None (excerpt)go

if strings.EqualFold(os.Getenv("SETU_COOKIE_CROSS_SITE"), "true") {
    return http.SameSiteNoneMode, true // None is only honored with Secure
}
return http.SameSiteLaxMode, isSecureRequest(r)

Decision	httpOnly session cookie; SETU_COOKIE_CROSS_SITE flips it to SameSite=None; Secure; CORS echoes only allowlisted origins with Allow-Credentials. Artifact downloads use fetch(), not <a href>.
Why	It let the console ship on Vercel and the API on Render (different origins) without a reverse-proxy tier — the cheapest possible split for a v0.1. httpOnly keeps the token out of JS.
Tradeoff	SameSite=None is strictly more permissive; it must be paired with a tight origin allowlist and HTTPS everywhere. And because the cookie can't ride a plain link, every download needs the fetch-then-blob dance in the client.

An external heartbeat for the free tier

A GitHub Actions cron pings /internal/scheduler/tick every 5 minutes rather than relying solely on the in-process ticker.

Decision	The apiserver has an in-process scheduler ticker AND exposes a token-guarded tick endpoint an external cron calls. Both do the same thing — fire whatever is due — because scheduler state is entirely in Postgres.
Why	Render's free tier sleeps the host after ~15 min idle, pausing the in-process ticker. The external ping both fires due schedules and wakes the host. It's the pragmatic cost of running the demo on free infrastructure.
Tradeoff	GitHub cron is best-effort (delayed under load, default-branch only) and adds an external dependency + a shared token (SETU_TICK_TOKEN, which 404s the route when unset). On a paid always-on host the in-process ticker alone suffices.

The thread through all of them

Almost every decision here bends toward one goal: let Setu move data inside a network it doesn’t control, without asking that network to open anything. The two-plane split, outbound-only agents, the blob seam, staged execution (with a direct-streaming hatch for small deltas), cursor incremental, and edge-resolved secrets are faces of the same constraint — and Postgres-only queueing, the dataset-centric model, cross-site cookies, and an external heartbeat are the pragmatic choices that let a v0.1 ship on free infrastructure while keeping that goal intact.