Proxy Vhost Sync Loop

The Manager renders the proxy's Caddyfile from its own data and pushes it to the running proxy through Caddy's admin API, without a human in the path. Below: configuring that loop, reading its drift status, and what "sync off" means and looks like. pfy proxy status|diff|sync are the CLI views onto it.

The shape

ent mutation ─► Debouncer ─► [exclusion lock] ─► assemble ─► render ─► POST /load
   startup    ─► Reconcile (force push, no dedup)
   ticker     ─► periodic resync

Mutations to any entity that carries a vhost input (Deployment, Service, Application, Environment, Tenant, Tenant2Application, DataDependency) schedule a push. The hook fires after the mutation succeeds and never returns an error of its own: the sync loop must not be able to fail a user's write.

Writes arrive in bursts (a multi-field edit, a stack apply touching a service and its deployments), and each push is a full Caddy provision cycle. The debouncer coalesces a burst into one push carrying the final state. The quiet window is proxy_sync_debounce, default 2s.

Reconcile force-pushes on cold start, and a periodic resync repairs what hooks cannot see: a push that lost the lock, a write served by another Manager process, a proxy that restarted. Both run under a cross-instance lock; losing that lock is not an error, because the winner renders from the same database and pushes the same bytes.

Split run modes

Reconcile and the resync ticker live in the cron/executor process. An api-only deployment is therefore mutation-driven but never self-healing: it will push when someone writes, and will not repair drift on its own.

Configuration

Key	Env	Required	Meaning
proxy_admin_url	PFY_PROXY_ADMIN_URL	no	Caddy admin base URL, e.g. http://127.0.0.1:2019. Empty disables sync entirely.
proxy_prelude_file	PFY_PROXY_PRELUDE_FILE	when admin URL is set	Path to the operator-owned Caddyfile prelude.
proxy_manager_upstream	PFY_PROXY_MANAGER_UPSTREAM	when admin URL is set	host:port the proxy rewrites /esb/* and /pilet-feed/* to.
proxy_metrics_url	PFY_PROXY_METRICS_URL	no (recommended)	The proxy's metrics base URL, e.g. http://proxy:2112. The Manager reads GET /config-hash there to learn what the proxy is actually running. Without it the "applied" hash is the hash the Manager pushed, so drift detection cannot see a proxy loaded out of band; the dashboard marks the verdict unverified.
proxy_admin_auth_header / proxy_admin_auth_token	PFY_PROXY_ADMIN_AUTH_HEADER / PFY_PROXY_ADMIN_AUTH_TOKEN	no	Authenticate the admin call (and the /config-hash read). Defence in depth.
proxy_sync_debounce	PFY_PROXY_SYNC_DEBOUNCE	no	Burst-coalescing window. Default 2s.
proxy_sync_interval	PFY_PROXY_SYNC_INTERVAL	no	Periodic resync cadence. Default 5m.

The admin endpoint must be bound to the internal network

It applies arbitrary configuration to the proxy that fronts every application and tenant on the install. proxy_admin_auth_header is defence in depth, not a substitute for the binding. The admin URL must not be reachable through any public vhost.

The rendered artifacts enforce this. pfy platform render emits admin {$PFY_ADMIN_LISTEN:127.0.0.1:2019} in the Caddyfile prelude, a loopback default, and sets PFY_ADMIN_LISTEN per target: ${NOMAD_IP_admin}:2019 (the node's own allocated address) in the Nomad proxy job, and 0.0.0.0:2019 in compose, where the admin port is never published and so reaches the compose network only.

Loopback is not available on Nomad: the proxy group is mode = "host" and the Manager group is mode = "bridge", so 127.0.0.1 in the Manager's namespace is the Manager, not the proxy. Set proxy_admin_url to http://<node-ip>:2019.

On a single-NIC host the node's own address is the public one. Set proxy_admin_bind in install.yaml to a private interface, and firewall :2019 either way.

The proxy adapts the prelude itself, and Caddy moves its admin listener on every config change, so a prelude that binds admin 0.0.0.0:2019 re-binds a running proxy to the wildcard the moment the Manager pushes it. Fix the prelude and re-push rather than only restarting.

Why the prelude and the upstream are mandatory

A push replaces the whole Caddyfile. Caddy's /load is atomic and total; there is no partial apply, no merge with what is already there. So everything the proxy needs must be in the bytes the Manager sends.

• proxy_prelude_file holds the global options, the caddy-security portal and the Manager's own vhost. The Manager cannot render it: the prelude is part of the artifact set the Manager is itself deployed by (pfy platform render from install.yaml), so rendering it would make the Manager its own prerequisite. Pushing without it would drop the portal, and every rendered vhost still emits authorize with <policy>, referring to a policy nothing defines. See what a portal-less config actually does below; the short version is that it is not a config the proxy can run.

• proxy_manager_upstream is where each application's /esb/* and /pilet-feed/* routes point. Empty omits those routes from every rendered vhost, breaking ESB and pilet delivery install-wide while producing a config that loads cleanly and looks correct.

Both are checked at startup, and the Manager refuses to boot if proxy_admin_url is set without them. Half-configured sync is the one state worse than no sync: it looks configured, it pushes, and what it pushes is dangerous.

The prelude is read fresh on every render, so re-rendering install.yaml takes effect without restarting the Manager.

What a portal-less config actually does

A portal-less config does not serve applications unauthenticated; it is a config the proxy cannot run at all.

caddy adapt succeeds: a portal-less config is syntactically valid Caddyfile, so nothing at the adapt stage objects. The failure is at provisioning, when caddy-security tries to build the authorization policy the authorize directives name and finds no portal configuration to build it from:

panic: runtime error: invalid memory address or nil pointer dereference
github.com/greenpau/go-authcrunch.(*Config).Validate(0x0)
github.com/greenpau/caddy-security.(*App).Provision(...)

The consequence then depends on how the config arrives, and the two cases are different:

Arrival	Result
Pushed to a running proxy via /load	The panic happens inside the admin API's HTTP handler, where Go's net/http recovers it. The proxy process survives. The push gets no HTTP response at all (the connection is closed mid-request) and the previously loaded configuration keeps serving. Later valid pushes apply normally.
Started with, e.g. the file is on disk and the proxy restarts	Nothing recovers the panic. caddy run exits (code 2) and the proxy does not come up: an install-wide outage until the config is fixed.

So the sync loop's failure mode is the first row, and it is a failed push, not an auth bypass and not a crash: the Manager sees a transport error rather than a rejection status, records it in LastError, and the proxy carries on with its last good config. The dangerous row is the second one, which is why a bad prelude on disk outlives the process that read it.

Either way, a config carrying authorize with <policy> for a policy nothing defines never reaches the point of serving a request.

caddy adapt does not catch this

Adapting is a syntax check; provisioning is where module configuration is validated. A config that adapts cleanly can still be one the proxy cannot run, so adapt passing in CI is not evidence that a config is loadable.

Sync off

An install with no proxy_admin_url registers no hooks, starts no loop, and pushes nothing. This is a supported state (an install that has not adopted the loop must boot, not fail), but it is never presented as a quiet success.

In the startup log:

Proxy vhost sync is OFF (proxy_admin_url unset): the Manager will NOT render or
push proxy config, …

On the dashboard, the panel does NOT say that, and deliberately so. It renders an empty state reading:

Proxy config status is unavailable, and nothing has been compared: this is not a clean result. Either this Manager is not driving the proxy at all (proxy_admin_url is not set, so the proxy keeps whatever configuration it already has), or the status is being withheld from you: install-wide proxy state is visible to system administrators only. The two are deliberately indistinguishable here; a system administrator can tell you which one it is.

The panel refuses to name proxy_admin_url as the cause because doing so would leak the install's state to a caller the admin gate just refused. "Sync is off" and "you may not see this" have to look identical from outside the gate. What the panel keeps is the part that matters either way: this is not a clean result and nothing has been compared.

Off is not "converged"

This is the whole reason the state is spelled out. A disabled loop's push status is the zero value: no hashes, Healthy=false, no error, which is indistinguishable from "configured, and every push has failed since boot". Those two need opposite operator responses, so the status carries an explicit Enabled flag and Healthy() is false whenever it is off. No caller can read a disabled install's empty state as a verdict about the proxy.

When sync is off, vhosts reach the proxy only because somebody ran pfy proxy sync by hand. Between a domain being added in the Manager and someone remembering to run it, the Manager's view and the proxy's view simply disagree.

Drift status

The dashboard's proxy panel (Platform overview → Proxy config status) shows:

Row	Meaning
State	Healthy, Unverified, Drift, Push failed or Unknown, with the age of the last attempt. A hash match on an install with no proxy_metrics_url reports Unverified, never Healthy.
Rendered hash	Hash of the config the Manager last produced.
Applied hash	Hash of the config the proxy is running, or a dash when nothing is known.
Verified	A sentence saying where the applied hash came from: read from the proxy's own /config-hash; assumed from this Manager's last push; the proxy could not be reached; the proxy is running an unstamped config; or not yet determined.
Last error	The render failure or the proxy's rejection body, when there is one.

Sync being off (proxy_admin_url unset) is not a row. The panel renders an empty state instead, and that is deliberately the same thing a non-administrator sees: install-wide proxy state is admin-only, and the gate must not leak the install's state through its own behaviour.

A system administrator can also push from this panel. Sync now renders the current state and pushes it, blocking until the push has settled, then reports which of the three outcomes occurred: pushed, already current, or another instance holds the push lock. It is the same syncProxyConfig mutation pfy proxy sync calls, and it writes the same audit row.

Rendered ≠ applied is drift. The proxy is serving something older than what the Manager can now describe.

Render errors outrank push errors

A config that could not be assembled never reaches the pusher, so a render failure would otherwise leave the last successful push's status standing, a healthy-looking panel while the Manager has silently lost the ability to describe the install. So:

• a render failure makes the status unhealthy even when the last push succeeded;
• the reported error is the render error when there is one, because it is both more recent and more serious: a push error describes something that happened to a config the Manager could still produce, a render error means it can no longer produce one at all.

A rejected push keeps the last good config

Caddy applies atomically. A config it rejects is not partially applied; the proxy keeps serving the previous one. The Manager records the rejection, leaves the old hash as applied, and retries with bounded backoff. Traffic is unaffected throughout; what you lose is freshness, not availability.

That asymmetry is why a render that errors never becomes a push: the worst outcome of not pushing is staleness, and the worst outcome of pushing a half-assembled config is an install-wide authentication bypass.

Metrics

The scrape names carry the executor prefix:

Metric	Notes
pfy_executor_proxy_sync_total{outcome}	outcome distinguishes success, rejection and render error.
pfy_executor_proxy_sync_duration_seconds	Push latency.

Render failures count as outcome=error with no duration sample: they never reached the proxy, so timing them would pollute a push-latency histogram, but not counting them at all would hide the loop's most dangerous state.

Operating it

Sync appears off but the admin URL is set: check which process you are looking at: the status is per-process, and in a split run mode the api process serves mutations while the executor process holds reconcile. Both should report enabled.

Rendered and applied hashes differ and will not converge: read the last error. A push error points at the proxy or the admin endpoint (unreachable, auth rejected, config refused). A render error points at the Manager's data, most often an application whose service routes cannot be assembled.

A vhost never appeared: before suspecting the loop, a service with no route prefixes is internal by design and has no hostname, and a host group with no primary row emits nothing at all. See Service Routing Rules.

Nothing pushes after a write: the hook fires on the process that SERVED the mutation. If that process has no loop, nothing is scheduled until the periodic resync, which only the cron/executor process runs.

Related

• Service Routing Rules
• Anatomy of a Stack
• Proxy Deployment
• Generated Artifacts
• Incident Runbook