Manager Deployment

The Manager is the control plane: a Go server that serves the GraphQL API and the bundled Vue UI, runs the trigger executor and background sweeps, and stores everything in PostgreSQL. Below: the run modes, the container image, the environment it needs, and how to tell whether it is healthy.

The Manager's own job renders from install.yaml

The platform stack, including the Manager's own Nomad job or Compose service, renders from a single install.yaml via pfy platform render|deploy. The Manager cannot render the artifacts that start it (the bootstrap exception), so these render locally. This page documents the settings that install.yaml drives; the hand-written snippets below show the shape of the result.

Deployment Overview

The Manager supports three run modes:

• API Mode: Handles HTTP/GraphQL requests (stateless, scalable)
• Executor Mode: Runs the trigger execution loop and the background sweeps (stateful, single instance)
• Both Mode: Combined API + Executor in one process (development, or the smallest installs)

Run Modes

The Manager supports three run modes configured via PFY_RUN_MODE environment variable or run_mode in config.yml:

API Mode

run_mode: api

Or with environment variable:

export PFY_RUN_MODE=api
./manager

Characteristics:

• Stateless
• Horizontally scalable
• Load balanced
• No trigger execution

Executor Mode

run_mode: executor
cron:
  metrics_port: 9090 # Prometheus metrics endpoint

Or with environment variable:

export PFY_RUN_MODE=executor
export PFY_CRON_METRICS_PORT=9090
./manager

Characteristics:

• Stateful (cron loop)
• Single instance only (a database advisory lock makes a second instance a no-op)
• Not load balanced
• Runs trigger execution and every background sweep: audit-chain verification and retention, credential-staleness reporting, invite and CLI-login expiry, proxy-sync reconcile, and outbox mail delivery
• Exposes Prometheus metrics on port 9090 (PFY_CRON_METRICS_PORT)

Prometheus Metrics:

When running in executor mode, metrics are exposed at :9090/metrics (PFY_CRON_METRICS_PORT):

• pfy_executor_queue_all_total{app="<app-id>"} - Total triggers queued per app
• pfy_executor_queue_processed_total{app="<app-id>"} - Successfully processed triggers per app
• pfy_executor_queue_waiting{app="<app-id>"} - Current waiting triggers per app
• pfy_executor_queue_process_time_seconds{app="<app-id>"} - Processing duration histogram per app
• pfy_executor_trigger_execution_total / pfy_executor_trigger_execution_errors_total - Trigger executions and failures
• pfy_executor_active_triggers - Currently active/enabled triggers
• pfy_executor_backend_dispatch_total{status="success|error"} / pfy_executor_backend_dispatch_duration_seconds - Backend dispatch results and latency
• pfy_executor_trigger_delivery_failed_total - Trigger deliveries that exhausted their retries
• pfy_executor_next_trigger_run_seconds - Seconds until the next scheduled trigger run
• pfy_executor_audit_chain_verification_failures_total - Audit hash-chain verification failures found by the sweep
• pfy_executor_proxy_sync_total{outcome="success|error"} / pfy_executor_proxy_sync_duration_seconds - Proxy vhost sync pushes

Database-Level Locking:

The executor takes a PostgreSQL advisory lock (trigger_executor) and renews it as a lease. Only the holder processes triggers, so an accidentally started second executor waits instead of dispatching duplicates. Missed cron occurrences during an outage are not replayed; see the runbook.

Both Mode (Development)

run_mode: both

Or with environment variable:

export PFY_RUN_MODE=both
./manager

Characteristics:

• Combined API + Executor in one process
• Fine for development and single-node installs; split the two once you scale the API horizontally, because only one executor may run

Docker Deployment

The published image ships no config file, so the Manager is configured entirely from PFY_* variables. PFY_ENV defaults to production, and the production checks run in every process regardless of run mode: without a DB password, a Pocket ID API key and an encryption key the container exits at startup with invalid configuration: ... on its first line.

Migrate the database first

The Manager never migrates on boot, in any run mode. Run -migrate from this same image first; it carries the migrations:

docker run --rm \
  -e PFY_DB_HOST=postgres \
  -e PFY_DB_PASSWORD="$DB_PASSWORD" \
  -e PFY_DB_SSLMODE=require \
  ghcr.io/productifyfw/manager:2.0.0 /app/manager -migrate

See Database migrations.

API Instance

docker run -d \
  --name manager-api \
  -p 8080:8080 \
  -p 8081:8081 \
  -e PFY_RUN_MODE=api \
  -e PFY_DB_HOST=postgres \
  -e PFY_DB_PASSWORD="$DB_PASSWORD" \
  -e PFY_DB_SSLMODE=require \
  -e PFY_POCKET_ID_HOST=https://auth.example.com \
  -e PFY_POCKET_ID_API_KEY="$POCKET_ID_API_KEY" \
  -e PFY_ENCRYPTION_KEY="$ENCRYPTION_KEY" \
  -e PFY_PROXY_SECRET="$PROXY_SECRET" \
  ghcr.io/productifyfw/manager:2.0.0

PFY_PROXY_SECRET must equal the proxy's PFY_MANAGER_TOKEN. In production an API-serving process refuses to start without it. A value that does not match the proxy's is not caught at startup: every gated request is answered 401 with {"error":"missing or invalid X-Proxy-Auth header: …"}.

Pin the image tag to the release you are deploying (the install bundle pins ghcr.io/productifyfw/manager:<version>); latest is shown here only as a placeholder.

Executor Instance

docker run -d \
  --name manager-executor \
  -p 9090:9090 \
  -e PFY_RUN_MODE=executor \
  -e PFY_CRON_METRICS_PORT=9090 \
  -e PFY_DB_HOST=postgres \
  -e PFY_DB_PASSWORD="$DB_PASSWORD" \
  -e PFY_DB_SSLMODE=require \
  -e PFY_POCKET_ID_API_KEY="$POCKET_ID_API_KEY" \
  -e PFY_ENCRYPTION_KEY="$ENCRYPTION_KEY" \
  ghcr.io/productifyfw/manager:2.0.0

The executor needs the Pocket ID key and the encryption key too. It never calls Pocket ID itself, but Validate() and ConfigureEncryption() run before anything branches on run_mode.

Accessing metrics:

curl http://localhost:9090/metrics

Docker Compose

version: "3.8"

services:
  manager-api:
    image: ghcr.io/productifyfw/manager:2.0.0
    deploy:
      replicas: 3
    ports:
      - "8080-8082:8080"
    environment:
      PFY_RUN_MODE: api
      PFY_DB_HOST: postgres
      PFY_DB_PASSWORD: ${DB_PASSWORD}
      PFY_DB_SSLMODE: require
      PFY_POCKET_ID_HOST: ${POCKET_ID_HOST}
      PFY_POCKET_ID_API_KEY: ${POCKET_ID_API_KEY}
      PFY_ENCRYPTION_KEY: ${ENCRYPTION_KEY}
      # Required in production for every API-serving process; must equal the
      # proxy's PFY_MANAGER_TOKEN.
      PFY_PROXY_SECRET: ${PROXY_SECRET}
    depends_on:
      - postgres
    restart: unless-stopped

  manager-executor:
    image: ghcr.io/productifyfw/manager:2.0.0
    ports:
      - "9090:9090"
    environment:
      PFY_RUN_MODE: executor
      PFY_CRON_METRICS_PORT: 9090
      PFY_DB_HOST: postgres
      PFY_DB_PASSWORD: ${DB_PASSWORD}
      PFY_DB_SSLMODE: require
      # Required in every process, not only the API instances.
      PFY_POCKET_ID_API_KEY: ${POCKET_ID_API_KEY}
      PFY_ENCRYPTION_KEY: ${ENCRYPTION_KEY}
    depends_on:
      - postgres
    restart: unless-stopped

  postgres:
    image: postgres:16
    environment:
      POSTGRES_DB: postgres
      POSTGRES_PASSWORD: ${DB_PASSWORD}
    volumes:
      - postgres_data:/var/lib/postgresql/data
    restart: unless-stopped

volumes:
  postgres_data:

Nomad Deployment

A complete reference job specification is available at manager/nomad/manager.nomad. The example below shows the API and Executor groups:

job "manager" {
  datacenters = ["dc1"]
  type        = "service"

  # API instances (scalable)
  group "api" {
    count = 3

    network {
      port "http" {
        to = 8080
      }
      port "health" {
        to = 8081
      }
    }

    task "manager-api" {
      driver = "docker"

      config {
        image = "ghcr.io/productifyfw/manager:2.0.0"
        ports = ["http", "health"]
      }

      env {
        PFY_ENV         = "production"
        PFY_RUN_MODE    = "api"
        PFY_PORT        = "8080"
        PFY_HEALTH_PORT = "8081"
        PFY_DB_HOST     = "postgres.example.com"
        PFY_DB_PORT     = "5432"
        PFY_DB_USER     = "postgres"
        PFY_DB_NAME     = "productify"
        # Leave PFY_DB_SSLMODE at its "require" default; "disable" alongside
        # env=production makes the Manager exit at startup.
      }

      # Secrets belong in Nomad Variables, not in the job file. All of them are
      # load-bearing: in production the API task refuses to start without the
      # DB password, the Pocket ID API key, the encryption key and the proxy
      # secret.
      template {
        data = <<EOH
{{ with nomadVar "nomad/jobs/manager" -}}
PFY_DB_PASSWORD={{ .db_password }}
PFY_POCKET_ID_HOST={{ .pocket_id_host }}
PFY_POCKET_ID_API_KEY={{ .pocket_id_api_key }}
PFY_ENCRYPTION_KEY={{ .encryption_key }}
PFY_PROXY_SECRET={{ .proxy_secret }}
{{- end }}
EOH
        destination = "secrets/env"
        env         = true
      }

      resources {
        cpu    = 1000
        memory = 512
      }

      service {
        name = "manager-api"
        port = "http"

        check {
          type     = "http"
          port     = "health"
          path     = "/healthz"
          interval = "10s"
          timeout  = "2s"
        }

        check {
          name     = "readiness"
          type     = "http"
          port     = "health"
          path     = "/readyz"
          interval = "10s"
          timeout  = "2s"
        }
      }
    }
  }

  # Executor instance (single)
  group "executor" {
    count = 1

    network {
      port "health" {
        to = 8081
      }
      port "metrics" {
        static = 9090
      }
    }

    task "manager-executor" {
      driver = "docker"

      config {
        image = "ghcr.io/productifyfw/manager:2.0.0"
        ports = ["health", "metrics"]
      }

      env {
        PFY_ENV               = "production"
        PFY_RUN_MODE          = "executor"
        PFY_HEALTH_PORT       = "8081"
        PFY_CRON_METRICS_PORT = "9090"
        PFY_DB_HOST           = "postgres.example.com"
        PFY_DB_PORT           = "5432"
        PFY_DB_USER           = "postgres"
        PFY_DB_NAME           = "productify"
      }

      template {
        data = <<EOH
{{ with nomadVar "nomad/jobs/manager" -}}
PFY_DB_PASSWORD={{ .db_password }}
# Validate() and ConfigureEncryption() run in every process. An executor
# without these two refuses to start, even though it never calls Pocket ID.
PFY_POCKET_ID_API_KEY={{ .pocket_id_api_key }}
PFY_ENCRYPTION_KEY={{ .encryption_key }}
{{- end }}
EOH
        destination = "secrets/env"
        env         = true
      }

      resources {
        cpu    = 500
        memory = 256
      }

      service {
        name = "manager-executor"
        port = "metrics"

        tags = [
          "prometheus",
          "metrics"
        ]

        check {
          type     = "http"
          port     = "health"
          path     = "/healthz"
          interval = "30s"
          timeout  = "5s"
        }
      }
    }
  }
}

Configuration

See Manager Configuration for complete configuration reference.

Minimum required environment variables:

# Database
PFY_DB_HOST=postgres.example.com
PFY_DB_PORT=5432
PFY_DB_USER=manager
PFY_DB_PASSWORD=your-secure-password   # production refuses to start without it
PFY_DB_NAME=manager_prod
PFY_DB_SSLMODE=require                 # "disable" is rejected in production

# PocketID Integration
PFY_POCKET_ID_HOST=https://auth.example.com
PFY_POCKET_ID_API_KEY=your-api-key     # production refuses to start without it

# Secrets at rest and the proxy trust boundary
PFY_ENCRYPTION_KEY=base64-32-bytes     # production refuses to start without it
PFY_PROXY_SECRET=shared-with-the-proxy # production refuses to start without it when the API is served

# Server Configuration
PFY_ENV=production
PFY_RUN_MODE=api  # Options: api, executor, both
PFY_PORT=8080

The database, Pocket ID and encryption settings apply to every process, including run_mode=executor; PFY_PROXY_SECRET is required only where the API is served (api or both). Generate the two keys with openssl rand -base64 32 (PFY_ENCRYPTION_KEY must be standard base64, not URL-safe), and set PFY_PROXY_SECRET to the same value as the proxy's PFY_MANAGER_TOKEN.

Beyond this minimum, three more settings are silently wrong when omitted rather than fatal; see Configuration Reference: PFY_PILETS_SIGNING_PRIVATE_KEY / PFY_PILETS_SIGNING_KEY_ID (pilet uploads are rejected on first use), PFY_ORCHESTRATOR_KIND (scale-from-zero wakes succeed and scale nothing), and the PFY_AUDIT_EXTERNAL_ANCHOR_* pair.

Health Checks

The Manager runs a separate health server (default port 8081, configurable via PFY_HEALTH_PORT or health_port) with liveness and readiness endpoints:

curl http://localhost:8081/healthz

Response:

{ "status": "healthy", "version": "2.0.0", "build_time": "2026-09-05T10:00:00Z" }

/healthz answers 200 whenever the process is up; it does not touch the database. The readiness endpoint pings the database and counts the tenants and services tables, so it also proves the schema is applied:

curl http://localhost:8081/readyz

It answers 200 with {"status":"ready"} (plus a warnings list for conditions such as PFY_ORCHESTRATOR_KIND=noop or a missing pilet signing key), or 503 with {"status":"not ready","reason":"…"}, for example schema_not_applied.

Both live on the health port, never on the API port: :8080/healthz is answered by the UI's catch-all with 200 and index.html, so a check against the API port proves nothing.

Monitoring

Metrics

API Mode Metrics (Future):

• Request rate (per endpoint)
• Response latency (p50, p95, p99)
• Error rate
• Database connection pool usage

Executor Mode Metrics (Available Now):

Prometheus metrics exposed at :9090/metrics:

• Queue Metrics: Triggers queued, processed, and waiting (per app)
• Processing Time: Histogram of trigger execution duration (per app)
• Active Triggers: Count of enabled triggers
• Backend Dispatch: Success/error counts for backend callbacks

Prometheus Scrape Configuration:

scrape_configs:
  - job_name: "manager-executor"
    static_configs:
      - targets: ["manager-executor:9090"]
    scrape_interval: 15s

Example PromQL Queries:

# Triggers waiting per application
pfy_executor_queue_waiting{app="app-uuid"}

# Trigger processing rate (per second)
rate(pfy_executor_queue_processed_total[5m])

# 95th percentile processing time
histogram_quantile(0.95, rate(pfy_executor_queue_process_time_seconds_bucket[5m]))

# Backend dispatch error rate
rate(pfy_executor_backend_dispatch_total{status="error"}[5m])

Logs

The Manager logs to stdout. Ship container stdout to your log aggregator; the shipped Loki/Promtail configuration under deployments/ops/logging/ labels it {service="manager"}; see Architecture: Logging.

Scaling

API Instances

Scale horizontally based on:

• CPU utilization (target 70%)
• Request rate
• Response latency

Nomad scaling policy:

scaling {
  min = 2
  max = 10

  policy {
    evaluation_interval = "30s"
    cooldown = "60s"
  }
}

Executor Instance

Do not scale it: exactly one instance must run.

Troubleshooting

API Not Responding

Check that:

• Container is running
• Port is accessible
• Database connection is valid
• Health check passes

Container exits immediately with invalid configuration:

Validate() runs before the health server binds, so a rejected setting shows up as a container that never becomes ready. The message names the setting; the common ones under PFY_ENV=production are a missing db.password, a missing pocket_id.api_key, db.sslmode=disable, and a missing PFY_ENCRYPTION_KEY (the last is reported separately by ConfigureEncryption). All of them apply to the executor as well as the API.

Everything returns 401

PFY_PROXY_SECRET does not match the proxy's PFY_MANAGER_TOKEN (or, in development, is unset). The Manager fails the trust boundary closed before any handler runs; the 401 carries a JSON body whose error field names which of the two it is, and /healthz stays green. See Configuration Reference.

Queries fail with relation "..." does not exist

The database was never migrated. The Manager does not migrate on boot in any run mode, and the shipped image cannot run -migrate; see Database migrations.

Executor Not Running Triggers

Check that:

• Mode is set to executor or both
• Database connection is valid
• Triggers are enabled
• Cron schedules are valid

Database Connection Issues

Check that:

• Connection string is correct
• Database is accessible from Manager
• Credentials are valid
• SSL/TLS configured properly

See Also

• Deployment Overview
• Docker Compose Deployment: the install bundle
• Configuration Reference: every PFY_* variable
• Architecture