Edge secrets

A cloud-placed connection resolves its secrets in the cloud (from environment or an encrypted store). An agent-placed connection is different: the cloud must be able to hold and hand out its config without ever holding the secret. Setu does that with a single mechanism — ${secret.NAME} references that are resolved at the data plane, wherever that happens to be.

A config with references, not credentials

In a connection config, a sensitive field holds a reference token instead of a value. The token is parsed by the shared expr engine; a ${secret.NAME} ref is classified as deferred — it can only be resolved at run time, next to a live secret source.

{
  "host": "10.0.4.12",
  "port": 5432,
  "database": "orders",
  "user": "setu_ro",
  "password": "${secret.PGPASSWORD}"
}

The control plane stores this document verbatim and hands it to the agent verbatim — the extract/load spec explicitly returns Config with refs intact and comments "NEVER resolve here". The one field that would be dangerous to store is the one field that is never a real value.

Storing a secret on the agent

You put the plaintext where the plaintext belongs — on the host that can reach the data — with setu secret set. Omit the value and it is read from stdin, so the credential never lands in argv or your shell history. Bulk-load a whole .env with setu secret import, list names with setu secret ls, and remove one with setu secret rm. Nothing is ever transmitted.

printf %s 's3cr3t' | setu secret set PGPASSWORD   # value via stdin, not argv
setu secret import ./edge.env                     # bulk NAME=VALUE lines
setu secret ls                                    # names only — values never printed

The value is stored in the host's OS keychain — macOS Keychain, Windows Credential Manager, or the Linux Secret Service (libsecret / dbus) — via the pure-Go zalando/go-keyring, so the release stays CGO-free. On a headless box with no keychain the store transparently falls back to the owner-only (0600) config file. Either way the list of secret names is kept in the config file (names are not sensitive), which is what lets setu secret ls work without unlocking the keychain. setu status reports the active backend.

// Set stores a secret value: in the OS keychain when one is available (keeping
// the config file free of the value), else in the 0600 config file as a fallback.
// The secret NAME is always recorded in the config so `setu secret ls` works.
func (s *Store) Set(name, value string) error {
	c, err := loadConfig()
	if err != nil {
		return err
	}
	if s.backend == BackendKeychain {
		if err := keyring.Set(service, name, value); err != nil {
			return fmt.Errorf("store secret in keychain: %w", err)
		}
		delete(c.Secrets, name) // keep the file free of the value
	} else {
		if c.Secrets == nil {
			c.Secrets = map[string]string{}
		}
		c.Secrets[name] = value
	}
	c.SecretNames = addName(c.SecretNames, name)
	return config.Save(c)
}

Resolution at the edge: exec.ResolveSecrets

Every job kind that builds a connector — test, discover, preview, extract, load — first calls ResolveSecrets to walk the config JSON and replace each ${secret.NAME} with the value from the agent's local store. A referenced-but-absent secret is a loud error, not a silent empty credential — so a misconfigured agent fails fast instead of connecting as nobody.

// ResolveSecrets walks the connection config JSON and replaces every
// ${secret.NAME} reference with the matching value from the agent's local secret
// store. It errors if a referenced secret is missing. A config with no references
// passes through unchanged.
func ResolveSecrets(config []byte, secrets map[string]string) ([]byte, error) {
	if len(config) == 0 {
		return config, nil
	}
	r := expr.NewResolver(secretMapProvider(secrets))
	out, err := r.ResolveJSON(json.RawMessage(config))
	if err != nil {
		return nil, err
	}
	return []byte(out), nil
}

// secretMapProvider resolves ${secret.NAME} refs from the agent's local store.
// A missing secret returns an error so a bad config fails loudly.
func (m secretMapProvider) Resolve(ref expr.Ref) (any, bool, error) {
	s, ok := ref.(expr.SecretRef)
	if !ok {
		return nil, false, nil
	}
	v, present := m[s.Name]
	if !present {
		return nil, false, fmt.Errorf("missing edge secret %q; set it with `setu secret set %s <value>`", s.Name, s.Name)
	}
	return v, true, nil
}

This is called immediately before the connector is built, so the resolved config exists only for the lifetime of that build. For example, in the extract path:

func Extract(ctx context.Context, spec ExtractSpec, secrets map[string]string) (Result, error) {
	resolved, err := ResolveSecrets(spec.Config, secrets) // ← plaintext, only here
	if err != nil {
		return Result{}, err
	}
	reg := connectors.Default()
	src, err := reg.BuildSource(ctx, spec.Provider, resolved)
	// ... run the copy to blob
}

Two resolvers, one reference grammar

The clever part is that edge and cloud use the same reference grammar (engine/expr) but plug in different providers. On the agent, the provider is a plain in-memory map from the local config. In the cloud, a connection destined for the cloud data plane resolves through an ordered provider chain: environment first, then an optional external vault, then an AES-GCM encrypted per-workspace store.

// NewResolver returns a resolver for connection config: environment first, then
// the encrypted store when both queries and a cipher are available. Store secrets
// are scoped to workspaceID. Unresolved references are errors.
func NewResolver(ctx context.Context, q *sqlc.Queries, cipher *Cipher, workspaceID uuid.UUID) *expr.Resolver {
	providers := []expr.Provider{expr.EnvProvider{}}
	if v := vaultFromEnv(ctx); v != nil { // external vault, if configured
		providers = append(providers, v)
	}
	if q != nil && cipher != nil { // AES-GCM encrypted per-workspace store
		providers = append(providers, StoreProvider{Q: q, Cipher: cipher, Ctx: ctx, WorkspaceID: workspaceID})
	}
	return expr.NewResolver(providers...)
}

The reference's journey

1. 1
   Define. You save an agent-placed connection with ${secret.PGPASSWORD} in its config. The console/apiserver stores the document as-is.
2. 2
   Stash. On the agent host you run setu secret set PGPASSWORD (value via stdin). It lands in the OS keychain — never sent anywhere.
3. 3
   Dispatch. A job is leased; the agent fetches the spec. The Config comes down with the ref still intact — the control plane never saw the password.
4. 4
   Resolve. exec.ResolveSecrets substitutes the local value into the config in memory, just before the connector is built.
5. 5
   Execute. The connector runs against the real source/destination. When the job ends, the resolved config is discarded with the frame.

The trust boundary

Related

• Agent overview — why outbound-only, and the config file.
• Job lifecycle — where in a job's life resolution happens.