Agent overview

Setu is a two-plane system: a control plane (the apiserver + console, where pipelines are defined and runs are orchestrated) and a data plane (where bytes actually move). For a cloud-reachable source and destination, the cloud worker is the data plane. But a great deal of real data sits somewhere the cloud can't open a socket to. That's what the agent is for.

Topology B: outbound-only, full data plane at the edge

There are two ways to bridge a private network. Topology A would punch a hole inward — open a port, run a tunnel, allow-list the cloud's IPs. Setu deliberately chose Topology B: the agent opens no inbound ports and makes only outbound HTTPS calls to the control plane. It asks for work rather than being pushed work.

Why this shape:

• NAT / firewall traversal for free. A plain outbound HTTPS request already works from inside virtually every network — home routers, corporate proxies, locked-down VPCs — with zero firewall changes. There is no inbound rule to negotiate with a security team.
• Tiny attack surface. Nothing listens. There is no inbound endpoint to scan, exploit, or DDoS. The only credential on the host is a single bearer key the agent minted at enrollment, kept in the OS keychain (or a 0600 file where none exists).
• Edge secrets. Database passwords and API keys are stored on the agent host and resolved locally at run time. The control plane only ever holds ${secret.NAME} references — never the plaintext. See Edge secrets.

What the agent actually runs

The agent isn't a thin proxy — it embeds the same engine/ packages the cloud worker uses, so an edge run is byte-for-byte the same code path as a cloud run. Only the trigger (a leased agent_job) and the secret source (a local store) differ. It handles five job kinds:

Extract and load are the heavy ones and are tied to a run; test, discover, and preview are connection-scoped, on-demand probes. The full state machine — how a job is leased, heartbeated, and completed — is covered in Job lifecycle.

The job envelope: IDs only, never secrets

A leased job is deliberately tiny. The wire shape carries identifiers and an opaque payload — no connection string, no password, nothing the control plane shouldn't be holding in the first place.

// Job is a leased unit of work handed to the agent. IDs only, never secrets.
type Job struct {
	ID           string          `json:"id"`
	Kind         string          `json:"kind"`
	RunID        string          `json:"runId,omitempty"`
	ConnectionID string          `json:"connectionId,omitempty"`
	Payload      json.RawMessage `json:"payload"`
}

There are two ways a job's instructions reach the agent, and the split is intentional:

• Connection-scoped kinds (test, discover, preview) are small and self-contained: the whole spec — provider, config with ${secret.X} refs, and any extras like stream/limit — rides inline in Payload.
• Run-scoped kinds (extract, load) carry an empty payload and fetch a secretless spec with a second authenticated GET — /jobs/{id}/extract-spec or /load-spec. That spec is derived server-side by joining the run → pipeline → connection, so it stays authoritative even if the job sat in the queue for a while.

The same engine, resolved locally

The agent's exec package is a thin adapter over the shared engine. It resolves ${secret.X} against the local store, builds the connector from the same connectors.Default() registry the cloud uses, and drives the identical runner.Copy into a blob-staging sink. An edge extract and a cloud extract are the same function call.

func Extract(ctx context.Context, spec ExtractSpec, secrets map[string]string) (Result, error) {
	resolved, err := ResolveSecrets(spec.Config, secrets) // ${secret.X} → value, locally
	if err != nil {
		return Result{}, err
	}
	reg := connectors.Default()                            // same registry as the cloud worker
	src, err := reg.BuildSource(ctx, spec.Provider, resolved)
	// ... select spec.Streams (or discover all when none named)
	sink := stageblob.NewSink(blb, spec.BlobPrefix)        // stage raw rows to the run's prefix
	stats, err := runner.Copy(ctx, src, sink, runner.Options{
		Mode:    connector.WriteAppend,                    // full-refresh snapshot
		Streams: streams,
	})
	return Result{BlobPrefix: spec.BlobPrefix, Rows: stats.RowsRead}, nil
}

Install and run

The agent is one Go binary, setu, installed from the GitHub Releases (goreleaser publishes win/mac/linux × amd64/arm64 archives; the install scripts resolve the latest release, currently v0.2.1). Two commands matter: enroll once to get a key, then run to start pulling work.

# 0. Install from the public setu-dist releases (pin with SETU_VERSION=v0.2.1). Windows: install.ps1
curl -fsSL https://raw.githubusercontent.com/Abhishekkumar2021/setu-dist/main/install.sh | sh

# 1. Enroll: exchange the one-time token from the console for a permanent key.
export SETU_CONTROL_PLANE_URL=https://app.setu.example
setu enroll <token>            # → "enrolled agent <id> at https://app.setu.example"

# 2. (optional) Store any edge secrets the connection config references.
printf %s 's3cr3t' | setu secret set PGPASSWORD   # value via stdin, not shell history
setu secret import ./edge.env                     # or bulk-import a .env
setu status                                        # config, control plane, secret backend

# 3. Run: long-poll for jobs and execute them until interrupted.
setu run                       # → "agent <id> polling https://app.setu.example"

The binary dispatches on os.Args[1], with grouped, colored output and -h help on every command. The full subcommand set:

setu run is the whole runtime: it loads the config, opens an outbound long-poll to /agent/v1/jobs/next?wait=25, and dispatches each leased job by kind. A clean SIGINT/SIGTERM cancels in-flight work and exits; transport errors back off exponentially (1s → 30s, doubling) and retry. The underlying http.Client has no global timeout — long-poll waits are governed per-call via context, and JobsNext sets its own deadline of wait + 15s so the transport always outlives the server's long-poll window. The mechanics of that loop are in Long-polling & leasing.

The config file

Enrollment writes a small JSON config to ~/.config/setu/config.json (override with SETU_CONFIG, or the legacy SETU_AGENT_CONFIG). It is written 0600 because it can hold the agent key and edge-secret values as a fallback when no OS keychain is available; the parent directory is 0700. When a keychain is present, the agent key and secret values live there instead and the file holds only the non-sensitive secretNames index.

{
  "baseUrl": "https://app.setu.example",
  "agentId": "b1a2...",
  "version": "0.2.1",
  "secretNames": ["PGPASSWORD"]
}

Two environment variables steer the agent, both read only where a config can't yet exist or must be overridden:

Where to go next

• Enrollment — the one-time token → permanent key exchange.
• Long-polling & leasing — how the job loop really works.
• Job lifecycle — states, heartbeat, the reaper, backoff.
• Edge secrets — the trust boundary, in detail.