Productify

Appearance

Nomad Platform Deployment ​

Complete guide for deploying Productify on HashiCorp Nomad with detailed setup instructions and monitoring.

These jobs are rendered, not hand-authored

Since the Wave 3 release train the platform stack renders from a single install.yaml via pfy platform render|deploy, and application jobs render from a deployment spec via pfy nomad render|deploy. Do not hand-edit the job files — hand edits are drift and are overwritten. The HCL below is shown as reference output: it is what the renderer produces, useful for understanding the shape of a job and for the nomad cluster setup itself. Start from Generated Artifacts.

Prerequisites ​

Required Software ​

  • Nomad: 1.6+ cluster (server and client nodes)
  • PostgreSQL: 16+ database
  • Docker: Runtime on all Nomad client nodes

System Requirements ​

  • Memory: 8 GB RAM minimum per node (16 GB recommended)
  • CPU: 4 cores minimum per node
  • Storage: 50 GB free disk space per node
  • Network: Low latency between nodes (< 10ms for same datacenter)

Nomad Installation ​

Linux (AMD64) ​

bash
# Download Nomad
wget https://releases.hashicorp.com/nomad/1.7.3/nomad_1.7.3_linux_amd64.zip

# Extract
unzip nomad_1.7.3_linux_amd64.zip

# Move to system path
sudo mv nomad /usr/local/bin/

# Verify installation
nomad version

macOS (Homebrew) ​

bash
# Install Nomad
brew install nomad

# Verify installation
nomad version

Other Platforms ​

See the official Nomad installation guide for other platforms and installation methods.

Nomad Agent Setup ​

Development Mode (Single Node) ​

For local testing and development:

bash
# Start Nomad in development mode (server + client)
sudo nomad agent -dev

# In another terminal, verify
nomad node status
nomad server members

# Access Web UI
open http://localhost:4646

Important Nomad Ports:

  • 4646: HTTP API and Web UI
  • 4647: RPC (Remote Procedure Call)
  • 4648: Serf WAN (gossip protocol)

Production Cluster ​

For production, separate server and client nodes are recommended.

Server Node:

bash
sudo nomad agent -config=/etc/nomad.d/server.hcl

server.hcl:

hcl
datacenter = "dc1"
data_dir = "/var/lib/nomad"

server {
  enabled = true
  bootstrap_expect = 3
}

Client Node:

bash
sudo nomad agent -config=/etc/nomad.d/client.hcl

client.hcl:

hcl
datacenter = "dc1"
data_dir = "/var/lib/nomad"

client {
  enabled = true
}

plugin "docker" {
  config {
    allow_privileged = false
  }
}

Complete Job Specification ​

Manager Job ​

A complete reference job (including a PostgreSQL group and a frontend group) is available at manager/nomad/manager.nomad.

hcl
job "manager" {
  datacenters = ["dc1"]
  type = "service"

  group "api" {
    count = 3

    update {
      max_parallel = 1
      health_check = "checks"
      min_healthy_time = "10s"
      healthy_deadline = "3m"
      auto_revert = true
    }

    network {
      port "http" {
        to = 8080
      }
      port "health" {
        to = 8081
      }
    }

    service {
      name     = "manager-api"
      port     = "http"
      provider = "nomad"
      tags     = ["productify", "api"]

      check {
        type = "http"
        port = "health"
        path = "/healthz"
        interval = "10s"
        timeout = "2s"
      }
    }

    task "server" {
      driver = "docker"

      config {
        image = "ghcr.io/productifyfw/manager:latest"
        ports = ["http", "health"]
      }

      env {
        PFY_ENV         = "production"
        PFY_RUN_MODE    = "api"
        PFY_PORT        = "8080"
        PFY_HEALTH_PORT = "8081"
        PFY_DB_SSLMODE  = "disable"
      }

      # Read secrets from Nomad Variables
      template {
        data = <<EOH
{{ with nomadVar "nomad/jobs/manager" -}}
PFY_DB_HOST={{ .db_host }}
PFY_DB_PORT={{ .db_port }}
PFY_DB_USER={{ .db_user }}
PFY_DB_PASSWORD={{ .db_password }}
PFY_DB_NAME={{ .db_name }}

PFY_POCKET_ID_HOST={{ .pocket_id_host }}
PFY_POCKET_ID_API_KEY={{ .pocket_id_api_key }}
{{- end }}
EOH
        destination = "secrets/env"
        env = true
      }

      resources {
        cpu    = 500
        memory = 512
      }
    }
  }

  group "executor" {
    count = 1

    constraint {
      operator = "distinct_hosts"
      value = "true"
    }

    network {
      port "metrics" {
        to = 9090
      }
    }

    service {
      name     = "manager-executor"
      port     = "metrics"
      provider = "nomad"
      tags     = ["productify", "executor"]
    }

    task "executor" {
      driver = "docker"

      config {
        image = "ghcr.io/productifyfw/manager:latest"
        ports = ["metrics"]
      }

      env {
        PFY_ENV               = "production"
        PFY_RUN_MODE          = "executor"
        PFY_CRON_METRICS_PORT = "9090"
      }

      template {
        data = <<EOH
{{ with nomadVar "nomad/jobs/manager" -}}
PFY_DB_HOST={{ .db_host }}
PFY_DB_PORT={{ .db_port }}
PFY_DB_USER={{ .db_user }}
PFY_DB_PASSWORD={{ .db_password }}
PFY_DB_NAME={{ .db_name }}
{{- end }}
EOH
        destination = "secrets/env"
        env = true
      }

      resources {
        cpu    = 200
        memory = 256
      }
    }
  }
}

Optimizer Job ​

hcl
job "optimizer" {
  datacenters = ["dc1"]
  type = "service"

  group "optimizer" {
    count = 2

    network {
      port "http" {
        to = 8015
      }
    }

    service {
      name     = "optimizer"
      port     = "http"
      provider = "nomad"
      tags     = ["productify", "autoscaler"]

      check {
        type = "http"
        path = "/health"
        interval = "10s"
        timeout = "2s"
      }
    }

    task "server" {
      driver = "docker"

      config {
        image = "ghcr.io/productifyfw/optimizer:latest"
        ports = ["http"]
      }

      env {
        CONFIG_PATH = "${NOMAD_TASK_DIR}/config.ini"
        PORT        = "${NOMAD_PORT_http}"
      }

      template {
        data = <<EOF
[main]
loglevel=info
api_loglevel=warning
only_test_data=false
enable_test_metrics=false
prometheus_url=http://prometheus:9090
port={{ env "NOMAD_PORT_http" }}
token=<optimizer-token>
EOF
        destination = "${NOMAD_TASK_DIR}/config.ini"
      }

      resources {
        cpu    = 1000
        memory = 1024
      }
    }
  }
}

Proxy Job ​

hcl
job "proxy" {
  datacenters = ["dc1"]
  type = "system"  # Deploy on all nodes

  group "caddy" {
    network {
      port "http" {
        static = 80
        to = 80
      }
      port "https" {
        static = 443
        to = 443
      }
      port "metrics" {
        to = 2112
      }
    }

    service {
      name     = "proxy"
      port     = "http"
      provider = "nomad"
      tags     = ["productify", "proxy"]

      check {
        type = "http"
        port = "metrics"
        path = "/metrics"
        interval = "10s"
        timeout = "2s"
      }
    }

    task "caddy" {
      driver = "docker"

      config {
        image = "ghcr.io/productifyfw/proxy:latest"
        ports = ["http", "https", "metrics"]

        volumes = [
          "local/Caddyfile:/etc/caddy/Caddyfile"
        ]
      }

      template {
        data = <<EOH
{
  admin off
  email admin@example.com
}

manager.example.com {
  reverse_proxy {
    {{- range nomadService "manager-api" }}
    to {{ .Address }}:{{ .Port }}
    {{- end }}

    lb_policy least_conn
    health_interval 10s
  }
}
EOH
        destination = "local/Caddyfile"
        change_mode = "script"
        change_script {
          command       = "/usr/bin/caddy"
          args          = ["reload", "--config", "/local/Caddyfile", "--force"]
          timeout       = "5s"
          fail_on_error = false
        }
      }

      resources {
        cpu    = 200
        memory = 256
      }
    }
  }
}

A production-grade proxy job — including host volumes for certificate storage, Pocket ID, the security (OAuth2/OIDC) configuration, and the productify app block — is available at deployments/proxy.nomad (and proxy/nomad/proxy.nomad).

Autoscaler Plugin Job ​

The ghcr.io/productifyfw/nomadscaler image bundles the Nomad Autoscaler agent with the productify-scaler strategy plugin preinstalled at /plugins/productify-scaler. A complete reference job (agent + Optimizer + Prometheus) is available at autoscaler/nomadscaler/config/autoscaler.hcl.

hcl
job "nomad-autoscaler" {
  datacenters = ["dc1"]
  type = "service"

  group "autoscaler" {
    count = 1

    network {
      port "http" {
        to = 8080
      }
    }

    service {
      name     = "nomad-autoscaler"
      port     = "http"
      provider = "nomad"

      check {
        type = "http"
        path = "/v1/health"
        interval = "10s"
        timeout = "2s"
      }
    }

    task "autoscaler" {
      driver = "docker"

      config {
        image   = "ghcr.io/productifyfw/nomadscaler:latest"
        command = "nomad-autoscaler"
        args    = ["agent", "-config", "${NOMAD_TASK_DIR}/config.hcl"]
      }

      # Autoscaler configuration
      template {
        data = <<EOH
plugin_dir = "/plugins"

nomad {
  address = "http://{{ env "attr.unique.network.ip-address" }}:4646"
}

apm "nomad" {
  driver = "nomad-apm"
  config = {
    address = "http://{{ env "attr.unique.network.ip-address" }}:4646"
  }
}

strategy "productify-scaler" {
  driver = "productify-scaler"

  config = {
    optimizer_url   = "http://{{- with nomadService "optimizer" }}{{ with index . 0 }}{{ .Address }}:{{ .Port }}{{ end }}{{ end }}"
    optimizer_token = "<optimizer-token>"
  }
}

http {
  bind_address = "0.0.0.0"
  bind_port    = 8080
}

telemetry {
  prometheus_metrics = true
  prometheus_retention_time = "24h"
}
EOH
        destination = "local/config.hcl"
      }

      resources {
        cpu    = 200
        memory = 256
      }
    }
  }
}

Scaling policies are embedded in the target job's scaling block (not deployed with the agent). For example, in the manager job's api group:

hcl
scaling {
  enabled = true
  min     = 2
  max     = 10

  policy {
    evaluation_interval = "10s"
    cooldown            = "30s"

    check "productify-scale-check" {
      source = "nomad-apm"
      query  = "avg_cpu-allocated"

      strategy "productify-scaler" {
        min             = 2
        max             = 10
        metric_app_name = "manager-api"
        cache_size      = 10
      }
    }
  }
}

Monitoring ​

Monitor the Manager Executor, the Nomad Autoscaler, and Nomad itself using Prometheus:

yaml
# prometheus.yml
scrape_configs:
  - job_name: "manager-executor"
    static_configs:
      - targets: ["manager-executor:9090"]

  - job_name: "nomad-autoscaler"
    metrics_path: /v1/metrics
    params:
      format: ["prometheus"]
    static_configs:
      - targets: ["nomad-autoscaler:8080"]

  - job_name: "nomad"
    metrics_path: /v1/metrics
    params:
      format: ["prometheus"]
    static_configs:
      - targets: ["localhost:4646"]

Deployment Commands ​

Deploy Manager Component ​

bash
cd manager/nomad

# Validate job specification
nomad job validate manager.nomad

# Plan the deployment (dry run)
nomad job plan manager.nomad

# Deploy the job
nomad job run manager.nomad

# Check job status
nomad job status manager

# View allocations (running instances)
nomad alloc status <alloc-id>

# Follow logs for a specific allocation
nomad alloc logs -f <alloc-id> server

Manager Job Components:

  • API Group: 3 instances (horizontally scaled)
    • Serves GraphQL API
    • Handles HTTP requests
    • Registers with Nomad service discovery
  • Executor Group: 1 instance
    • Runs scheduled jobs and triggers
    • Processes background tasks

Deploy Proxy and Authentication ​

bash
cd proxy/nomad

# The proxy job includes both the Caddy proxy and Pocket ID
nomad job validate proxy.nomad
nomad job run proxy.nomad

# Check status
nomad job status proxy

# View service endpoints
nomad service list

Deploy Autoscaler Stack ​

bash
cd autoscaler/nomadscaler/config

# Deploy autoscaler with optimizer and Prometheus
nomad job validate autoscaler.hcl
nomad job run autoscaler.hcl

# Check status
nomad job status autoscaler

# Find the dynamically assigned Prometheus port
nomad alloc status <autoscaler-alloc-id> | grep -A 10 "Ports"

# Access Prometheus UI (using the assigned port)
open http://localhost:<prometheus-port>

# Verify metrics collection
curl http://localhost:<prometheus-port>/api/v1/targets

Autoscaler Components:

  • Nomad Autoscaler Plugin: Evaluates scaling policies
  • Optimizer Service: Provides ML-based scaling decisions (Python/FastAPI)
  • Prometheus: Collects and stores metrics

Deploy All Jobs (Complete Stack) ​

bash
# Deploy in order
nomad job run manager/nomad/manager.nomad
nomad job run proxy/nomad/proxy.nomad
nomad job run autoscaler/nomadscaler/config/autoscaler.hcl

# Verify all jobs are running
nomad job status

Rolling Update ​

bash
# Update Manager
nomad job run -check-index $(nomad job inspect manager | jq .JobModifyIndex) manager.nomad

# Monitor deployment
nomad deployment status <deployment-id>

# Watch deployment progress
watch -n 2 'nomad job status manager | head -20'

Rollback ​

bash
# View job versions
nomad job history manager

# Revert to previous version
nomad job revert manager <version>

# Stop a failed deployment
nomad deployment fail <deployment-id>

Monitoring and Verification ​

Nomad Web UI ​

Access the Nomad UI at http://localhost:4646:

  • Jobs: View all running jobs, their status, and allocations
  • Allocations: See details of each running container instance
  • Nodes: Infrastructure information and resource usage
  • Topology: Visual overview of the cluster state
  • Evaluations: View scheduling decisions

Check Job Health ​

bash
# Job overview
nomad job status manager

# Specific allocation details
nomad alloc status <alloc-id>

# Service health checks
nomad service info manager-api

# Recent job history
nomad job history manager

# View deployment status
nomad deployment status <deployment-id>

Prometheus Metrics ​

Access Prometheus (on its assigned port) and query metrics:

bash
# Manager API availability
curl 'http://localhost:9090/api/v1/query?query=up{job="manager"}'

# CPU usage across cluster
curl 'http://localhost:9090/api/v1/query?query=nomad_client_host_cpu_user'

# Memory usage per allocation
curl 'http://localhost:9090/api/v1/query?query=nomad_client_alloc_memory_usage'

# HTTP request rate (if exposed by app)
curl 'http://localhost:9090/api/v1/query?query=rate(http_requests_total[5m])'

# Active connections
curl 'http://localhost:9090/api/v1/query?query=nomad_client_alloc_network_rx_bytes'

Test Autoscaler ​

Verify autoscaler functionality by generating load:

bash
# Install Apache Bench (if not already installed)
# Ubuntu/Debian: apt-get install apache2-utils
# macOS: brew install ab

# Generate load
ab -n 10000 -c 100 http://localhost:8080/

# Watch scaling events in real-time
nomad job history manager

# Monitor allocation count changes
watch -n 2 'nomad job status manager | grep "Allocations"'

# View autoscaler decision logs
nomad alloc logs -f <autoscaler-alloc-id> autoscaler

# Check optimizer service health
curl http://localhost:8015/health

Service Discovery Verification ​

bash
# List all registered services
nomad service list

# Get specific service information
nomad service info manager-api

High Availability ​

Executor Instance

Only one Executor instance should run. The Executor uses database-level locking to ensure single-instance execution across the cluster.

Increase the count for API instances to scale horizontally:

hcl
group "api" {
  count = 5  # Scale as needed
  # ...
}

Troubleshooting ​

Job Won't Start ​

bash
# Validate job file syntax
nomad job validate <job-file>.hcl

# Get detailed error message
nomad job status <job-name>

# Check allocation-level errors
nomad alloc status <alloc-id>

# View allocation logs
nomad alloc logs <alloc-id> <task-name>

# Check Docker driver status
docker ps
docker images

Network Connectivity Issues ​

bash
# Check service discovery
nomad service list
nomad service info <service-name>

# Verify port mappings
nomad alloc status <alloc-id> | grep -A 10 "Ports"

# Enter container for debugging
nomad alloc exec -task <task-name> <alloc-id> /bin/sh

# Test connectivity from inside container
nomad alloc exec <alloc-id> wget -O- http://localhost:8081/healthz
nomad alloc exec <alloc-id> ping postgres

Database Connection Problems ​

bash
# Test database connectivity
nomad alloc exec <postgres-alloc-id> psql -U postgres -d productify -c "SELECT version();"

# Check database logs
nomad alloc logs <postgres-alloc-id>

Autoscaler Not Scaling ​

bash
# Check scaling policies
nomad scaling policy list

# View policy details
nomad scaling policy info <policy-id>

# Test manual scaling
nomad job scale manager 5

# Verify Prometheus targets
curl http://localhost:9090/api/v1/targets

# Check optimizer logs
nomad alloc logs -f <optimizer-alloc-id> optimizer

# Verify autoscaler plugin
nomad alloc logs -f <autoscaler-alloc-id> autoscaler

High Resource Usage ​

bash
# Check cluster resources
nomad node status

# View resource allocation
nomad status

# Check specific node usage
nomad node status <node-id>

# See resource constraints
nomad job inspect manager | jq '.Job.TaskGroups[].Tasks[].Resources'

Check Job Status ​

bash
nomad job status manager
nomad alloc status <alloc-id>
nomad alloc logs <alloc-id>

Networking Issues ​

bash
# Check allocations
nomad alloc status -verbose <alloc-id>

# Exec into container
nomad alloc exec <alloc-id> sh

Advanced Configuration ​

Auto-Revert on Failure ​

hcl
update {
  max_parallel = 1
  health_check = "checks"
  min_healthy_time = "10s"
  healthy_deadline = "5m"
  auto_revert = true
  auto_promote = false
}

Canary Deployments ​

hcl
update {
  max_parallel = 1
  canary = 1
  min_healthy_time = "30s"
  healthy_deadline = "10m"
  auto_promote = false
  auto_revert = true
}

Resource Limits ​

hcl
resources {
  cpu    = 1000  # MHz
  memory = 1024  # MB

  memory_max = 2048  # Max memory before OOM
}

Spread Across Nodes ​

hcl
spread {
  attribute = "${node.unique.id}"
  weight    = 100
}

See Also ​