Proxy Deployment

The proxy is a custom Caddy build that sits in front of everything: it terminates TLS, logs users in through Pocket ID (OIDC), selects the tenant, and forwards requests to the Manager and to application backends with the identity headers they trust. Below: how to build and run that image, what it reads from the environment, and how to monitor it. For the Caddyfile itself see the Caddyfile reference.

Deployment Overview

The Proxy uses a custom Caddy build with:

• OAuth2/OIDC authentication (caddy-security plugin)
• The productify plugin: identity-header injection, tenant selection, ESB/pilet routing, scale-from-zero activator
• Automatic HTTPS with Let's Encrypt (or Caddy's internal CA)
• A metrics and probe server on port 2112 (/metrics, /healthz, /readyz, /config-hash). Operational surface, never for the internet: bind it with metrics_address, because the plugin's own default is every interface
• A Caddy admin API on port 2019 that the Manager pushes rendered vhosts to

Build Custom Caddy

The proxy requires custom Caddy plugins that must be built from source:

cd proxy
docker build -t ghcr.io/productifyfw/proxy:latest .

The Dockerfile compiles Caddy 2.11.4 (Alpine base) with the caddy-security plugin (OAuth2/OIDC) and the productify plugin from proxy/caddy/productify. Pin the resulting image to a release tag when you deploy it (the install bundle pins ghcr.io/productifyfw/proxy:<version>); latest is a placeholder here.

Docker Deployment

Basic Deployment

docker run -d \
  --name productify-proxy \
  -p 80:80 \
  -p 443:443 \
  -e PFY_MANAGER_TOKEN="$PROXY_SECRET" \
  -e PFY_OAUTH_CLIENT_SECRET="$OIDC_CLIENT_SECRET" \
  -v $PWD/Caddyfile:/etc/caddy/Caddyfile:ro \
  -v caddy_data:/data \
  -v caddy_config:/config \
  ghcr.io/productifyfw/proxy:latest

Do not publish 2112 (metrics/probes) or 2019 (admin API) to the internet; reach them over an internal network only.

Docker Compose

The proxy repository ships a development docker-compose.yml that builds the proxy locally and starts it together with Pocket ID:

volumes:
  pocketid:

networks:
  app:

services:
  pocketid:
    image: ghcr.io/pocket-id/pocket-id:v2.14.0
    env_file: .env.pocketid
    ports:
      - 1411:1411
    networks:
      - app
    volumes:
      - "pocketid:/app/data"

  proxy:
    build:
      context: .
      dockerfile: Dockerfile-local
    ports:
      - "80:80"
      - "443:443"
      - "2112:2112"
    depends_on:
      - pocketid
    networks:
      - app
    volumes:
      - "./Caddyfile:/etc/caddy/Caddyfile"
    # The mounted Caddyfile references these at load time. `:?` makes compose
    # refuse to start with a clear message instead of starting a proxy that
    # rejects every request. Put the values in a local, gitignored .env.
    environment:
      PFY_MANAGER_TOKEN: ${PFY_MANAGER_TOKEN:?set PFY_MANAGER_TOKEN in .env (must match the Manager PFY_PROXY_SECRET)}
      PFY_OAUTH_CLIENT_SECRET: ${PFY_OAUTH_CLIENT_SECRET:?set PFY_OAUTH_CLIENT_SECRET in .env (the Pocket ID OIDC client secret)}
      PFY_ADMIN_LISTEN: ${PFY_ADMIN_LISTEN:-127.0.0.1:2019}

The Caddyfile's productify block names the Manager URL and reads the shared secret as token {$PFY_MANAGER_TOKEN}; the caddy-security block reads client_secret {$PFY_OAUTH_CLIENT_SECRET}. The values come from the environment, never from the file. For a production stack use the install bundle instead of this development compose file.

Nomad Deployment

Job Specification

job "productify-proxy" {
  datacenters = ["dc1"]
  type        = "service"

  group "proxy" {
    count = 1

    network {
      port "http" {
        static = 80
        to     = 80
      }
      port "https" {
        static = 443
        to     = 443
      }
      port "metrics" {
        to = 2112
      }
    }

    service {
      name     = "productify-proxy"
      port     = "https"
      provider = "nomad"
      tags     = ["proxy", "https"]

      check {
        type     = "tcp"
        port     = "https"
        interval = "10s"
        timeout  = "2s"
      }
    }

    service {
      name     = "productify-proxy-metrics"
      port     = "metrics"
      provider = "nomad"
      tags     = ["metrics"]
    }

    task "caddy" {
      driver = "docker"

      config {
        image = "ghcr.io/productifyfw/proxy:1.1.0"
        ports = ["http", "https", "metrics"]

        volumes = [
          "local/Caddyfile:/etc/caddy/Caddyfile",
        ]
      }

      template {
        data = <<EOF
{
  email admin@example.com
  admin off

  security {
    oauth identity provider generic {
      realm generic
      driver generic
      client_id <client-id>
      client_secret {$PFY_OAUTH_CLIENT_SECRET}
      scopes openid email profile
      base_auth_url http://pocketid.localhost
      metadata_url http://pocketid.localhost/.well-known/openid-configuration
    }

    authentication portal pocketportal {
      crypto default token lifetime 3600
      enable identity provider generic
      cookie insecure off

      transform user {
        match realm generic
        action add role user
      }
    }

    authorization policy pocketpolicy {
      set auth url /auth/oauth2/generic
      allow roles user
      validate bearer header
      inject headers with claims
      enable strip token
    }
  }

  productify {
    manager http://172.17.0.1:8080
    # Read from the environment at load time; deliver PFY_MANAGER_TOKEN through
    # a Nomad Variable + `template { env = true }`, never inline it here.
    token {$PFY_MANAGER_TOKEN}
  }
}

# Manager UI/API
manager.example.com {
  # Anonymous browser bootstrap: served with no identity, so ahead of the policy.
  handle /pilet-feed/* {
    reverse_proxy http://172.17.0.1:8080
  }

  handle /language-packs/* {
    reverse_proxy http://172.17.0.1:8080
  }

  handle /auth/* {
    authenticate with pocketportal
  }

  handle {
    authorize with pocketpolicy
    reverse_proxy http://172.17.0.1:8080
  }
}

# Applications
app.example.com {
  vars app_id <application-uuid>

  @auth {
    path /auth/*
  }

  route @auth {
    productify_before_auth with {vars.app_id}
    authenticate with pocketportal
  }

  route /* {
    authorize with pocketpolicy
    productify with {vars.app_id}
    reverse_proxy http://app-backend:8080
  }
}
EOF

        destination = "local/Caddyfile"
      }

      resources {
        cpu    = 200
        memory = 128
      }
    }
  }
}

This job is illustrative: it hand-writes the Caddyfile and keeps the admin API off, so the Manager cannot push vhosts to it. The maintained reference job is proxy/nomad/proxy.nomad, which reads the rendered Caddyfile, manager_token and oauth_client_secret from the Nomad Variable nomad/jobs/proxy; the platform renderer's output is in deployments/examples/qa.productify.dev/rendered/productify-proxy.nomad.

Configuration

Caddyfile Template

See Caddyfile Configuration for complete examples.

Configuration Sources

The proxy is configured through its Caddyfile, and that Caddyfile is rendered, not hand-written. The Caddyfile shown above is illustrative; see Generated Artifacts for the real pipeline.

• Install-level parts (global options, the productify app block with the manager URL and the shared secret, the caddy-security OAuth2/OIDC portal, the Manager's own vhost) render from install.yaml at platform-deploy time (pfy platform render).
• Per-application vhosts (application ID, domains, backend upstream, the productify_* directive chain) render from the Manager's deployment data and are pushed to the running proxy through Caddy's admin API whenever an application or domain changes, with drift detection against the rendered hash (pfy proxy status|diff|sync).

The proxy's admin API must be on and reachable only on the internal network for the push to work; admin off, as in the illustrative job above, prevents it.

Environment variables

The Caddyfile never contains secrets. It reads these at load time ({$VAR}) or request time ({env.VAR}):

Variable	Default	Meaning
PFY_MANAGER_TOKEN	(required)	The shared proxy secret (token in the productify block, and header_up X-Proxy-Auth on Manager-bound routes). Must equal the Manager's PFY_PROXY_SECRET.
PFY_OAUTH_CLIENT_SECRET	(required)	Pocket ID OIDC client secret for the caddy-security portal.
PFY_ADMIN_LISTEN	127.0.0.1:2019	Admin API bind address (admin {$PFY_ADMIN_LISTEN:127.0.0.1:2019} in the rendered prelude). The Manager pushes to it; never expose it publicly.
PFY_METRICS_LISTEN	127.0.0.1:2112	Metrics/probe server bind address (metrics_address in the productify block). Without the directive the plugin binds :2112 on every interface.
PFY_MANAGER_UPSTREAM	(set by the rendered stack)	host:port of the Manager for the Manager vhost's reverse_proxy in the rendered prelude; the compose render sets manager:8080.

The defaults above are the Caddyfile's own fallbacks. A deployment names the addresses it wants: the install bundle's compose file sets both to every interface and publishes neither port. That compose file also exports MANAGER_DOMAIN, ID_DOMAIN, ACME_EMAIL, OIDC_CLIENT_SECRET and CADDY_JWT_SHARED_KEY, which its Caddyfile dereferences. The full list, with security notes, is in the Configuration Reference.

TLS Certificates

Let's Encrypt (Automatic)

{
  email admin@example.com
}

app.example.com {
  # Automatic certificate
  reverse_proxy backend:8080
}

Custom Certificates

app.example.com {
  tls /certs/cert.pem /certs/key.pem
  reverse_proxy backend:8080
}

Mount certificates:

volumes:
  - ./certs:/certs:ro

Monitoring

Prometheus Metrics

The proxy exposes metrics on port 2112:

curl http://proxy:2112/metrics

Prometheus Scrape Config

scrape_configs:
  - job_name: "productify-proxy"
    static_configs:
      - targets: ["proxy:2112"]

Request metrics are recorded only when the productify global block sets enable_metrics (and enable_size_metrics for the size histograms). The metrics server itself always starts on metrics_address.

Available Metrics

• pfy_authentication_awaiting_users - Users in authentication flow
• pfy_total_requests - Total requests per application (a counter without a _total suffix)
• pfy_response_time_seconds - Response time distribution
• pfy_request_size_bytes - Request size distribution
• pfy_response_size_bytes - Response size distribution
• pfy_activator_wakes_total, pfy_activator_timeouts_total, pfy_activator_wait_seconds, pfy_activator_pending_requests, pfy_activator_buffer_depth - Scale-from-zero activator behaviour; the shipped ProxyHighErrorRate alert is built on pfy_activator_timeouts_total. pfy_activator_pending_requests feeds the autoscaler as cold-start demand; the optimizer queries it and adds it to the demand series as an interactive level term, though it is not what wakes a sleeping app (Known Issues)

The same server answers GET /healthz (liveness), GET /readyz (readiness: the app has started and the Manager is reachable) and GET /config-hash (the hash of the config the proxy is running, which the Manager reads through PFY_PROXY_METRICS_URL for drift detection).

Health Checks

TCP Health Check

check {
  type     = "tcp"
  port     = "https"
  interval = "10s"
  timeout  = "2s"
}

HTTP Health Check (probe endpoints)

The metrics server on port 2112 also serves the probes. Use /readyz for a readiness check and /healthz for liveness:

check {
  type     = "http"
  port     = "metrics"
  path     = "/readyz"
  interval = "10s"
  timeout  = "2s"
}

Troubleshooting

Build Failures

Check that:

• Go 1.26+ is available (the Dockerfile builds with golang:1.26-alpine)
• All module dependencies available
• Network access to Go module proxy

Rebuild:

docker build --no-cache -t ghcr.io/productifyfw/proxy:latest ./proxy

Authentication Not Working

Check that:

• Identity provider is accessible from proxy
• OAuth2 client credentials are correct
• Metadata URL returns valid OIDC configuration
• Cookie settings match environment

Test OIDC metadata:

curl https://idp.example.com/.well-known/openid-configuration

Manager Integration Errors

Check the logs:

docker logs productify-proxy

Then check that:

• Manager URL is accessible from proxy
• Manager token is valid
• Application ID exists in Manager database

Test Manager connectivity:

# Manager health endpoint (health server port)
curl http://manager:8081/healthz

Certificate Errors

Let's Encrypt:

• DNS points to proxy
• Ports 80/443 publicly accessible
• Valid email configured
• Not hitting rate limits

Custom certificates:

• Certificate files exist and are readable
• Certificate matches domain
• Private key is not encrypted

High Availability

Multiple Instances

Deploy multiple proxy instances behind a load balancer:

group "proxy" {
  count = 3  # Run 3 instances

  # ... rest of config
}

Load Balancer Configuration

Use external load balancer (e.g., HAProxy, nginx) to distribute traffic:

Internet → Load Balancer → Proxy Instances

Shared State

Caddy stores certificate state in /data. For multiple instances:

1. Use a shared volume (NFS, EFS)
2. Or configure a shared Caddy storage backend so instances coordinate certificate management

Security Considerations

• Keep the admin API on an internal address only. The Manager needs :2019 to push vhosts, so do not turn it off; bind it with PFY_ADMIN_LISTEN to an address only the Manager can reach and firewall it. Caddy's admin API has no authentication of its own.
• Keep the metrics/probe server internal. /readyz and /config-hash are operational endpoints; keep 2112 off the internet.
• Use HTTPS for all production domains.
• Deliver PFY_MANAGER_TOKEN from a secret store.
• Set cookie insecure off in production.
• Configure rate limiting at the load balancer.
• Add security headers to the Caddyfile prelude through extra_caddy_global in install.yaml.

See Also

• Caddyfile Configuration
• Application Routing
• Proxy Vhost Sync Loop: how the Manager pushes vhosts and reports drift
• Configuration Reference