Productify

Appearance

Nomad Platform Deployment ​

Nomad is the scale-up path from the single-node Compose bundle: the same pinned platform versions, scheduled by a HashiCorp Nomad cluster. Below: installing Nomad, what the platform jobs look like, how to migrate the database before the Manager starts, and how to deploy, monitor and troubleshoot the jobs.

These jobs are rendered, not hand-authored

The platform stack (Manager, proxy, Pocket ID, Postgres, and the Caddyfile prelude) renders from a single install.yaml with pfy platform render --target nomad / pfy platform deploy, and application jobs render from a deployment spec with pfy nomad render|deploy. Do not hand-edit job files: hand edits are drift and are overwritten.

The HCL on this page is illustrative. It shows the shape of each job and the environment the Manager needs; it is not byte-for-byte renderer output. The real rendered platform jobs for a sample install are committed in deployments/examples/qa.productify.dev/rendered/ (productify-manager.nomad, productify-proxy.nomad, productify-pocket-id.nomad, productify-postgres.nomad and the Caddyfile), and deployments/NOMAD.md walks the single-node render + deploy. Start from Generated Artifacts.

Prerequisites ​

Required Software ​

  • Nomad: 1.6+ cluster (server and client nodes)
  • PostgreSQL: 16+ database
  • Docker: Runtime on all Nomad client nodes

System Requirements ​

  • Memory: 8 GB RAM minimum per node (16 GB recommended)
  • CPU: 4 cores minimum per node
  • Storage: 50 GB free disk space per node
  • Network: Low latency between nodes (< 10ms for same datacenter)

Nomad Installation ​

Linux (AMD64) ​

bash
# Download Nomad
wget https://releases.hashicorp.com/nomad/1.7.3/nomad_1.7.3_linux_amd64.zip

# Extract
unzip nomad_1.7.3_linux_amd64.zip

# Move to system path
sudo mv nomad /usr/local/bin/

# Verify installation
nomad version

macOS (Homebrew) ​

bash
# Install Nomad
brew install nomad

# Verify installation
nomad version

Other Platforms ​

See the official Nomad installation guide for other platforms and installation methods.

Nomad Agent Setup ​

Development Mode (Single Node) ​

For local testing and development:

bash
# Start Nomad in development mode (server + client)
sudo nomad agent -dev

# In another terminal, verify
nomad node status
nomad server members

# Access Web UI
open http://localhost:4646

Important Nomad Ports:

  • 4646: HTTP API and Web UI
  • 4647: RPC (Remote Procedure Call)
  • 4648: Serf WAN (gossip protocol)

Production Cluster ​

For production, separate server and client nodes are recommended.

Server Node:

bash
sudo nomad agent -config=/etc/nomad.d/server.hcl

server.hcl:

hcl
datacenter = "dc1"
data_dir = "/var/lib/nomad"

server {
  enabled = true
  bootstrap_expect = 3
}

Client Node:

bash
sudo nomad agent -config=/etc/nomad.d/client.hcl

client.hcl:

hcl
datacenter = "dc1"
data_dir = "/var/lib/nomad"

client {
  enabled = true
}

plugin "docker" {
  config {
    allow_privileged = false
  }
}

Complete Job Specification ​

Manager Job ​

A maintained reference job with an api group and an executor group is at manager/nomad/manager.nomad; it takes its image and database settings as Nomad variables. The matching one-shot migration job is manager/nomad/manager-migrate.nomad (see Database migrations).

hcl
job "manager" {
  datacenters = ["dc1"]
  type = "service"

  group "api" {
    count = 3

    update {
      max_parallel = 1
      health_check = "checks"
      min_healthy_time = "10s"
      healthy_deadline = "3m"
      auto_revert = true
    }

    network {
      port "http" {
        to = 8080
      }
      port "health" {
        to = 8081
      }
    }

    service {
      name     = "manager-api"
      port     = "http"
      provider = "nomad"
      tags     = ["productify", "api"]

      check {
        type = "http"
        port = "health"
        path = "/healthz"
        interval = "10s"
        timeout = "2s"
      }
    }

    task "server" {
      driver = "docker"

      config {
        image = "ghcr.io/productifyfw/manager:2.0.0"
        ports = ["http", "health"]
      }

      env {
        PFY_ENV         = "production"
        PFY_RUN_MODE    = "api"
        PFY_PORT        = "8080"
        PFY_HEALTH_PORT = "8081"

        # PFY_DB_SSLMODE is NOT set here on purpose. It defaults to "require",
        # and setting it to "disable" alongside PFY_ENV=production makes the
        # Manager exit at startup:
        #   invalid configuration: db.sslmode=disable is not allowed in
        #   production (use require or stronger)
        # Set it to verify-full once the database presents a verifiable cert.
      }

      # Read secrets from Nomad Variables
      template {
        data = <<EOH
{{ with nomadVar "nomad/jobs/manager" -}}
PFY_DB_HOST={{ .db_host }}
PFY_DB_PORT={{ .db_port }}
PFY_DB_USER={{ .db_user }}
PFY_DB_PASSWORD={{ .db_password }}
PFY_DB_NAME={{ .db_name }}

PFY_POCKET_ID_HOST={{ .pocket_id_host }}
PFY_POCKET_ID_API_KEY={{ .pocket_id_api_key }}

# Trust boundary. Must equal the proxy's PFY_MANAGER_TOKEN. The API task refuses
# to start in production without it; a wrong value is not caught at startup and
# answers every request with 401 {"error": "missing or invalid X-Proxy-Auth ..."}.
PFY_PROXY_SECRET={{ .proxy_secret }}

# At-rest encryption of endpoint/backend secrets. Required in production; the
# Manager refuses to start without it.
PFY_ENCRYPTION_KEY={{ .encryption_key }}

# Pilet provenance signing. Not checked at startup -- unset means the first
# pilet upload is rejected instead.
PFY_PILETS_SIGNING_PRIVATE_KEY={{ .pilet_signing_key }}
PFY_PILETS_SIGNING_KEY_ID={{ .pilet_signing_key_id }}

# Scale-from-zero. The default kind is "noop", which answers every wake
# successfully and scales nothing.
PFY_ORCHESTRATOR_KIND=nomad
PFY_ORCHESTRATOR_NOMAD_ADDRESS={{ .nomad_address }}
PFY_ORCHESTRATOR_NOMAD_TOKEN={{ .nomad_token }}
{{- end }}
EOH
        destination = "secrets/env"
        env = true
      }

      resources {
        cpu    = 500
        memory = 512
      }
    }
  }

  group "executor" {
    count = 1

    constraint {
      operator = "distinct_hosts"
      value = "true"
    }

    network {
      port "metrics" {
        to = 9090
      }
    }

    service {
      name     = "manager-executor"
      port     = "metrics"
      provider = "nomad"
      tags     = ["productify", "executor"]
    }

    task "executor" {
      driver = "docker"

      config {
        image = "ghcr.io/productifyfw/manager:2.0.0"
        ports = ["metrics"]
      }

      env {
        PFY_ENV               = "production"
        PFY_RUN_MODE          = "executor"
        PFY_CRON_METRICS_PORT = "9090"
        # Same rule as the api group: leave PFY_DB_SSLMODE at its "require"
        # default. "disable" plus env=production is a fatal startup error.
      }

      template {
        data = <<EOH
{{ with nomadVar "nomad/jobs/manager" -}}
PFY_DB_HOST={{ .db_host }}
PFY_DB_PORT={{ .db_port }}
PFY_DB_USER={{ .db_user }}
PFY_DB_PASSWORD={{ .db_password }}
PFY_DB_NAME={{ .db_name }}

# Validate() and ConfigureEncryption() run in EVERY process, not just the API.
# An executor missing either of these refuses to start.
PFY_POCKET_ID_API_KEY={{ .pocket_id_api_key }}
PFY_ENCRYPTION_KEY={{ .encryption_key }}
{{- end }}
EOH
        destination = "secrets/env"
        env = true
      }

      resources {
        cpu    = 200
        memory = 256
      }
    }
  }
}

Optimizer Job ​

The optimizer image ghcr.io/productifyfw/autoscaler/optimizer:<version> is a private ghcr package until it is made public; the Nomad client's Docker driver needs registry credentials (auth in the task config, or a docker config on the node) to pull it. ghcr.io/productifyfw/autoscaler-nomadscaler is public.

hcl
job "optimizer" {
  datacenters = ["dc1"]
  type = "service"

  group "optimizer" {
    count = 2

    network {
      port "http" {
        to = 8015
      }
    }

    service {
      name     = "optimizer"
      port     = "http"
      provider = "nomad"
      tags     = ["productify", "autoscaler"]

      check {
        type = "http"
        path = "/health"
        interval = "10s"
        timeout = "2s"
      }
    }

    task "server" {
      driver = "docker"

      config {
        image = "ghcr.io/productifyfw/autoscaler/optimizer:latest"
        ports = ["http"]
      }

      env {
        CONFIG_PATH = "${NOMAD_TASK_DIR}/config.ini"
        PORT        = "${NOMAD_PORT_http}"
      }

      template {
        data = <<EOF
[main]
loglevel=info
api_loglevel=warning
only_test_data=false
enable_test_metrics=false
prometheus_url=http://prometheus:9090
port={{ env "NOMAD_PORT_http" }}
token=<optimizer-token>
EOF
        destination = "${NOMAD_TASK_DIR}/config.ini"
      }

      resources {
        cpu    = 1000
        memory = 1024
      }
    }
  }
}

Proxy Job ​

The job below is the smallest thing that runs: a hand-written Caddyfile with the admin API off and no login portal, fronting the Manager. It is enough to see how the proxy is scheduled, and not enough for a real install: without the caddy-security portal nobody can log in, and with admin off the Manager cannot push application vhosts. Use the rendered productify-proxy.nomad (see the tip at the top) or proxy/nomad/proxy.nomad, which reads the rendered Caddyfile, manager_token and oauth_client_secret from the Nomad Variable nomad/jobs/proxy.

hcl
job "proxy" {
  datacenters = ["dc1"]
  type = "system"  # Deploy on all nodes

  group "caddy" {
    network {
      port "http" {
        static = 80
        to = 80
      }
      port "https" {
        static = 443
        to = 443
      }
      port "metrics" {
        to = 2112
      }
    }

    service {
      name     = "proxy"
      port     = "http"
      provider = "nomad"
      tags     = ["productify", "proxy"]

      check {
        type = "http"
        port = "metrics"
        path = "/metrics"
        interval = "10s"
        timeout = "2s"
      }
    }

    task "caddy" {
      driver = "docker"

      config {
        image = "ghcr.io/productifyfw/proxy:1.1.0"
        ports = ["http", "https", "metrics"]

        volumes = [
          "local/Caddyfile:/etc/caddy/Caddyfile"
        ]
      }

      template {
        data = <<EOH
{
  admin off
  email admin@example.com
}

manager.example.com {
  reverse_proxy {
    {{- range nomadService "manager-api" }}
    to {{ .Address }}:{{ .Port }}
    {{- end }}

    lb_policy least_conn
    health_interval 10s
  }
}
EOH
        destination = "local/Caddyfile"
        change_mode = "script"
        change_script {
          command       = "/usr/bin/caddy"
          args          = ["reload", "--config", "/local/Caddyfile", "--force"]
          timeout       = "5s"
          fail_on_error = false
        }
      }

      resources {
        cpu    = 200
        memory = 256
      }
    }
  }
}

proxy/nomad/proxy.nomad is the maintained reference: it runs the proxy in host networking with the rendered Caddyfile, plus a pocketid group. The renderer's output for a sample install is deployments/examples/qa.productify.dev/rendered/productify-proxy.nomad.

Autoscaler Plugin Job ​

The ghcr.io/productifyfw/autoscaler-nomadscaler image (published by the autoscaler release workflow) bundles the Nomad Autoscaler agent with the productify-scaler strategy plugin preinstalled at /plugins/productify-scaler. A complete reference job (agent + Optimizer + Prometheus) is at autoscaler/nomadscaler/config/autoscaler.hcl; that file uses locally built productify/scaler:local and productify/optimizer:local images (from nomadscaler/build.sh), so substitute the published tags when you deploy it.

hcl
job "nomad-autoscaler" {
  datacenters = ["dc1"]
  type = "service"

  group "autoscaler" {
    count = 1

    network {
      port "http" {
        to = 8080
      }
    }

    service {
      name     = "nomad-autoscaler"
      port     = "http"
      provider = "nomad"

      check {
        type = "http"
        path = "/v1/health"
        interval = "10s"
        timeout = "2s"
      }
    }

    task "autoscaler" {
      driver = "docker"

      config {
        image   = "ghcr.io/productifyfw/autoscaler-nomadscaler:latest"
        command = "nomad-autoscaler"
        args    = ["agent", "-config", "${NOMAD_TASK_DIR}/config.hcl"]
      }

      # Autoscaler configuration
      template {
        data = <<EOH
plugin_dir = "/plugins"

nomad {
  address = "http://{{ env "attr.unique.network.ip-address" }}:4646"
}

apm "nomad" {
  driver = "nomad-apm"
  config = {
    address = "http://{{ env "attr.unique.network.ip-address" }}:4646"
  }
}

strategy "productify-scaler" {
  driver = "productify-scaler"

  config = {
    optimizer_url   = "http://{{- with nomadService "optimizer" }}{{ with index . 0 }}{{ .Address }}:{{ .Port }}{{ end }}{{ end }}"
    optimizer_token = "<optimizer-token>"
  }
}

http {
  bind_address = "0.0.0.0"
  bind_port    = 8080
}

telemetry {
  prometheus_metrics = true
  prometheus_retention_time = "24h"
}
EOH
        destination = "local/config.hcl"
      }

      resources {
        cpu    = 200
        memory = 256
      }
    }
  }
}

Scaling policies are embedded in the target job's scaling block (not deployed with the agent). For example, in the manager job's api group:

hcl
scaling {
  enabled = true
  min     = 2
  max     = 10

  policy {
    evaluation_interval = "10s"
    cooldown            = "30s"

    check "productify-scale-check" {
      source = "nomad-apm"
      query  = "avg_cpu-allocated"

      strategy "productify-scaler" {
        min             = 2
        max             = 10
        metric_app_name = "manager-api"
        # The Nomad identifiers, which are a different identity from the `app`
        # label above: the nomad_client_allocs_* series that measure per-replica
        # capacity are exported by the Nomad client and carry the job's OWN
        # names. This policy lives in the `api` group, whose task is `server`.
        group           = "api"
        task            = "server"
        # Seconds each optimizer response covers. Above the optimizer's step
        # (15s by default) it holds a multi-step plan, so the count can ramp
        # between calls instead of moving one step per call.
        cache_size      = 30
      }
    }
  }
}

Monitoring ​

Monitor the Manager Executor, the Nomad Autoscaler, and Nomad itself using Prometheus:

yaml
# prometheus.yml
scrape_configs:
  - job_name: "manager-executor"
    static_configs:
      - targets: ["manager-executor:9090"]

  - job_name: "nomad-autoscaler"
    metrics_path: /v1/metrics
    params:
      format: ["prometheus"]
    static_configs:
      - targets: ["nomad-autoscaler:8080"]

  - job_name: "nomad"
    metrics_path: /v1/metrics
    params:
      format: ["prometheus"]
    static_configs:
      - targets: ["localhost:4646"]

Deployment Commands ​

Database migrations ​

Run this before the Manager job, on a fresh database and on every upgrade that ships a migration. The Manager never migrates on boot in any run mode, and the API group will start against an unmigrated database and then fail on every query.

Simplest path: a batch job running manager -migrate from the image you are deploying. It applies the SQL compiled into that binary through the Atlas Go SDK, so nothing is mounted and nothing is fetched, and the migration set cannot drift from the binary:

hcl
job "manager-migrate" {
  datacenters = ["dc1"]
  type        = "batch"

  group "migrate" {
    task "migrate" {
      driver = "docker"

      config {
        image = "ghcr.io/productifyfw/manager:2.0.0"
        args  = ["/app/manager", "-migrate"]
      }

      template {
        data        = <<EOT
{{ with nomadVar "nomad/jobs/manager" }}PFY_DB_HOST={{ .db_host }}
PFY_DB_PORT={{ .db_port }}
PFY_DB_USER={{ .db_user }}
PFY_DB_PASSWORD={{ .db_password }}
PFY_DB_NAME={{ .db_name }}{{ end }}
PFY_DB_SSLMODE=require
EOT
        destination = "secrets/env"
        env         = true
      }
    }
  }
}

Applying is idempotent, so the batch job can stay in the deploy pipeline, and concurrent runs are serialised by a Postgres advisory lock.

Or from a checkout on any host that can reach the database:

bash
cd manager
git checkout <the-tag-you-are-deploying>   # migrations must match the image
make migrate-apply DB_URL='postgres://user:pass@db.internal:5432/productify?sslmode=require'

The Atlas CLI path still works and writes the same revision history, so an existing pipeline built on it needs no change. manager/nomad/manager-migrate.nomad takes it, using the upstream Atlas image with the migration files delivered by artifact (run it with -var db_host=… -var db_password=… and a pinned -var ref=<tag>; the Nomad client needs git on its PATH for the git artifact). The rendered platform stack instead runs migrate as a prestart task from a pinned tarball named in install.yaml (deployments/NOMAD.md, "Database schema"). Abbreviated:

hcl
job "manager-migrate" {
  datacenters = ["dc1"]
  type        = "batch"

  group "migrate" {
    task "atlas" {
      driver = "docker"

      config {
        image   = "arigaio/atlas:latest"
        work_dir = "/local"
        args = [
          "migrate", "apply",
          "--dir", "file:///local/migrations",
          "--url", "${DB_URL}",
        ]
      }

      artifact {
        # The migration set for the exact manager version being deployed.
        source      = "git::https://github.com/ProductifyFW/manager.git//ent/migrate/migrations?ref=v2.0.0"
        destination = "local/migrations"
      }

      template {
        data        = "DB_URL={{ with nomadVar \"nomad/jobs/manager\" }}postgres://{{ .db_user }}:{{ .db_password }}@{{ .db_host }}:{{ .db_port }}/{{ .db_name }}?sslmode=require{{ end }}"
        destination = "secrets/env"
        env         = true
      }
    }
  }
}

atlas migrate apply is idempotent too. Its apply path takes no advisory lock, so do not run it alongside a manager -migrate.

Also run the derived-name audit before an upgrade

manager -audit-derived-names reports deployments whose derived Nomad upstream or proxy hostname collides, and exits non-zero when any do. Run it after the migration and before starting the new server; a collision that predates the uniqueness rule will otherwise block every deployment write. Like -migrate, it runs from the shipped image: it needs only the binary and the database.

Deploy Manager Component ​

bash
cd manager/nomad

# Validate job specification
nomad job validate manager.nomad

# Plan the deployment (dry run). The job takes its image and database settings
# as variables (see the `variable` blocks at the top of the file).
nomad job plan -var image=ghcr.io/productifyfw/manager:<tag> -var db_host=... manager.nomad

# Deploy the job
nomad job run -var image=ghcr.io/productifyfw/manager:<tag> -var db_host=... manager.nomad

# Check job status
nomad job status manager

# View allocations (running instances)
nomad alloc status <alloc-id>

# Follow logs for a specific allocation
nomad alloc logs -f <alloc-id> server

Manager Job Components:

  • API Group: 3 instances (horizontally scaled)
    • Serves GraphQL API
    • Handles HTTP requests
    • Registers with Nomad service discovery
  • Executor Group: 1 instance
    • Runs scheduled jobs and triggers
    • Processes background tasks

Deploy Proxy and Authentication ​

bash
cd proxy/nomad

# The proxy job includes both the Caddy proxy and Pocket ID
nomad job validate proxy.nomad
nomad job run proxy.nomad

# Check status
nomad job status proxy

# View service endpoints
nomad service list

Deploy Autoscaler Stack ​

bash
cd autoscaler/nomadscaler/config

# Deploy autoscaler with optimizer and Prometheus
nomad job validate autoscaler.hcl
nomad job run autoscaler.hcl

# Check status
nomad job status autoscaler

# Find the dynamically assigned Prometheus port
nomad alloc status <autoscaler-alloc-id> | grep -A 10 "Ports"

# Access Prometheus UI (using the assigned port)
open http://localhost:<prometheus-port>

# Verify metrics collection
curl http://localhost:<prometheus-port>/api/v1/targets

Autoscaler Components:

  • Nomad Autoscaler Plugin: Evaluates scaling policies
  • Optimizer Service: Forecasts demand and solves for the replica count (Python/FastAPI)
  • Prometheus: Collects and stores metrics

Deploy All Jobs (Complete Stack) ​

The rendered path does the ordering for you:

bash
pfy platform render --target nomad --out-dir ./out   # inspect the four jobs + Caddyfile
pfy platform deploy                                   # render, run migrations, run each job

Hand-run equivalent, in order (migrations first; see Database migrations):

bash
nomad job run manager/nomad/manager-migrate.nomad
nomad job run manager/nomad/manager.nomad
nomad job run proxy/nomad/proxy.nomad
nomad job run autoscaler/nomadscaler/config/autoscaler.hcl

# Verify all jobs are running
nomad job status

Rolling Update ​

bash
# Update Manager
nomad job run -check-index $(nomad job inspect manager | jq .JobModifyIndex) manager.nomad

# Monitor deployment
nomad deployment status <deployment-id>

# Watch deployment progress
watch -n 2 'nomad job status manager | head -20'

Rollback ​

bash
# View job versions
nomad job history manager

# Revert to previous version
nomad job revert manager <version>

# Stop a failed deployment
nomad deployment fail <deployment-id>

Monitoring and Verification ​

Nomad Web UI ​

Access the Nomad UI at http://localhost:4646:

  • Jobs: View all running jobs, their status, and allocations
  • Allocations: See details of each running container instance
  • Nodes: Infrastructure information and resource usage
  • Topology: Visual overview of the cluster state
  • Evaluations: View scheduling decisions

Check Job Health ​

bash
# Job overview
nomad job status manager

# Specific allocation details
nomad alloc status <alloc-id>

# Service health checks
nomad service info manager-api

# Recent job history
nomad job history manager

# View deployment status
nomad deployment status <deployment-id>

Prometheus Metrics ​

Access Prometheus (on its assigned port) and query metrics:

bash
# Manager API availability
curl 'http://localhost:9090/api/v1/query?query=up{job="manager"}'

# CPU usage across cluster
curl 'http://localhost:9090/api/v1/query?query=nomad_client_host_cpu_user'

# Memory usage per allocation
curl 'http://localhost:9090/api/v1/query?query=nomad_client_alloc_memory_usage'

# HTTP request rate (if exposed by app)
curl 'http://localhost:9090/api/v1/query?query=rate(http_requests_total[5m])'

# Active connections
curl 'http://localhost:9090/api/v1/query?query=nomad_client_alloc_network_rx_bytes'

Test Autoscaler ​

Verify autoscaler functionality by generating load:

bash
# Install Apache Bench (if not already installed)
# Ubuntu/Debian: apt-get install apache2-utils
# macOS: brew install ab

# Generate load
ab -n 10000 -c 100 http://localhost:8080/

# Watch scaling events in real-time
nomad job history manager

# Monitor allocation count changes
watch -n 2 'nomad job status manager | grep "Allocations"'

# View autoscaler decision logs
nomad alloc logs -f <autoscaler-alloc-id> autoscaler

# Check optimizer service health
curl http://localhost:8015/health

Service Discovery Verification ​

bash
# List all registered services
nomad service list

# Get specific service information
nomad service info manager-api

High Availability ​

Executor Instance

Only one Executor instance should run. The Executor uses database-level locking to ensure single-instance execution across the cluster.

Increase the count for API instances to scale horizontally:

hcl
group "api" {
  count = 5  # Scale as needed
  # ...
}

Troubleshooting ​

Job Won't Start ​

bash
# Validate job file syntax
nomad job validate <job-file>.hcl

# Get detailed error message
nomad job status <job-name>

# Check allocation-level errors
nomad alloc status <alloc-id>

# View allocation logs
nomad alloc logs <alloc-id> <task-name>

# Check Docker driver status
docker ps
docker images

Network Connectivity Issues ​

bash
# Check service discovery
nomad service list
nomad service info <service-name>

# Verify port mappings
nomad alloc status <alloc-id> | grep -A 10 "Ports"

# Enter container for debugging
nomad alloc exec -task <task-name> <alloc-id> /bin/sh

# Test connectivity from inside container
nomad alloc exec <alloc-id> wget -O- http://localhost:8081/healthz
nomad alloc exec <alloc-id> ping postgres

Database Connection Problems ​

bash
# Test database connectivity
nomad alloc exec <postgres-alloc-id> psql -U postgres -d productify -c "SELECT version();"

# Check database logs
nomad alloc logs <postgres-alloc-id>

Autoscaler Not Scaling ​

bash
# Check scaling policies
nomad scaling policy list

# View policy details
nomad scaling policy info <policy-id>

# Test manual scaling
nomad job scale manager 5

# Verify Prometheus targets
curl http://localhost:9090/api/v1/targets

# Check optimizer logs
nomad alloc logs -f <optimizer-alloc-id> optimizer

# Verify autoscaler plugin
nomad alloc logs -f <autoscaler-alloc-id> autoscaler

High Resource Usage ​

bash
# Check cluster resources
nomad node status

# View resource allocation
nomad status

# Check specific node usage
nomad node status <node-id>

# See resource constraints
nomad job inspect manager | jq '.Job.TaskGroups[].Tasks[].Resources'

Check Job Status ​

bash
nomad job status manager
nomad alloc status <alloc-id>
nomad alloc logs <alloc-id>

Networking Issues ​

bash
# Check allocations
nomad alloc status -verbose <alloc-id>

# Exec into container
nomad alloc exec <alloc-id> sh

Advanced Configuration ​

Auto-Revert on Failure ​

hcl
update {
  max_parallel = 1
  health_check = "checks"
  min_healthy_time = "10s"
  healthy_deadline = "5m"
  auto_revert = true
  auto_promote = false
}

Canary Deployments ​

hcl
update {
  max_parallel = 1
  canary = 1
  min_healthy_time = "30s"
  healthy_deadline = "10m"
  auto_promote = false
  auto_revert = true
}

Resource Limits ​

hcl
resources {
  cpu    = 1000  # MHz
  memory = 1024  # MB

  memory_max = 2048  # Max memory before OOM
}

Spread Across Nodes ​

hcl
spread {
  attribute = "${node.unique.id}"
  weight    = 100
}

See Also ​