Productify

Appearance

Autoscaler Deployment ​

The autoscaler decides how many replicas each application should run. It has two parts: a Go strategy plugin for the HashiCorp Nomad Autoscaler, and a Python optimizer service the plugin asks for replica counts. Both run on a Nomad cluster; none of this applies to Compose installs.

Components ​

  • Nomadscaler Plugin - Go-based Nomad Autoscaler strategy plugin (productify-scaler, source in autoscaler/nomadscaler)
  • Optimizer Service - Python (FastAPI) service that forecasts demand and solves a MILP for the replica count (source in autoscaler/optimizer)

Optimizer Deployment ​

The Optimizer is configured through a config.ini file (see Optimizer Configuration). The container reads the config path from the CONFIG_PATH environment variable and the listen port from PORT (default: 8015).

The release workflow publishes the optimizer as ghcr.io/productifyfw/autoscaler/optimizer:<version> and :latest. That package is private: an anonymous pull is refused with 401, so docker login ghcr.io with a token that can read ProductifyFW packages first, until the package is made public. (ghcr.io/productifyfw/autoscaler-nomadscaler is public.)

The image ships with a baked-in development config.ini (test data and test metrics enabled), so it starts without any mount:

bash
docker run --rm -p 8015:8015 ghcr.io/productifyfw/autoscaler/optimizer:latest

Docker ​

For production, mount your own config.ini and point CONFIG_PATH at it (it overrides the baked-in config):

bash
docker run -d \
  --name optimizer \
  -p 8015:8015 \
  -e CONFIG_PATH=/app/config.ini \
  -v $PWD/config.ini:/app/config.ini:ro \
  ghcr.io/productifyfw/autoscaler/optimizer:latest

Docker Compose ​

yaml
services:
  optimizer:
    image: ghcr.io/productifyfw/autoscaler/optimizer:latest
    ports:
      - "8015:8015"
    environment:
      CONFIG_PATH: /app/config.ini
    volumes:
      - ./config.ini:/app/config.ini:ro
    restart: unless-stopped

Nomad ​

hcl
job "optimizer" {
  datacenters = ["dc1"]
  type = "service"

  group "optimizer" {
    count = 2  # For high availability

    network {
      port "http" {
        to = 8015
      }
    }

    service {
      name = "optimizer"
      port = "http"

      tags = ["productify", "autoscaler"]

      check {
        type = "http"
        path = "/health"
        interval = "10s"
        timeout = "2s"
      }
    }

    task "server" {
      driver = "docker"

      config {
        image = "ghcr.io/productifyfw/autoscaler/optimizer:latest"
        ports = ["http"]
      }

      env {
        CONFIG_PATH = "${NOMAD_TASK_DIR}/config.ini"
        PORT        = "${NOMAD_PORT_http}"
      }

      template {
        data = <<EOF
[main]
loglevel=info
api_loglevel=warning
only_test_data=false
enable_test_metrics=false
prometheus_url=http://prometheus:9090
port={{ env "NOMAD_PORT_http" }}
token=<optimizer-token>
EOF
        destination = "${NOMAD_TASK_DIR}/config.ini"
      }

      resources {
        cpu    = 1000
        memory = 1024
      }
    }
  }
}

Nomadscaler Plugin Deployment ​

Installation ​

bash
# Build plugin (produces ./bin/productify-scaler and the productify/scaler:local Docker image)
cd autoscaler/nomadscaler
./build.sh

# Install to Nomad Autoscaler
sudo cp ./bin/productify-scaler /opt/nomad-autoscaler/plugins/
sudo chmod +x /opt/nomad-autoscaler/plugins/productify-scaler

Alternatively, use the prebuilt image ghcr.io/productifyfw/autoscaler-nomadscaler:latest, which bundles the Nomad Autoscaler agent with the plugin preinstalled at /plugins/productify-scaler.

Nomad Autoscaler Configuration ​

hcl
# autoscaler.hcl
plugin_dir = "/opt/nomad-autoscaler/plugins"

nomad {
  address = "http://localhost:4646"
}

apm "nomad" {
  driver = "nomad-apm"
  config = {
    address = "http://localhost:4646"
  }
}

strategy "productify-scaler" {
  driver = "productify-scaler"

  config = {
    optimizer_url   = "http://optimizer:8015"
    optimizer_token = "<optimizer-token>"
  }
}

http {
  bind_address = "0.0.0.0"
  bind_port    = 8080
}

log {
  level = "INFO"
}

Scaling Policy ​

Scaling policies are embedded in the target job's scaling block and use the productify-scaler strategy:

hcl
group "frontend" {
  count = 3

  scaling {
    enabled = true
    min     = 2
    max     = 20

    policy {
      evaluation_interval = "10s"
      cooldown            = "30s"

      check "productify-scale-check" {
        source = "nomad-apm"
        query  = "avg_cpu-allocated"

        strategy "productify-scaler" {
          min             = 2
          max             = 20
          metric_app_name = "web-app"  # the `app` label on the pfy_* series
          group           = "frontend" # the Nomad task group this scaling block lives in
          task            = "web"      # the task inside it, i.e. the task block below
          cache_size      = 30 # seconds the response covers; above step_seconds gives a multi-step ramp
        }
      }
    }
  }

  task "web" {
    # ...
  }
}

optimizer_url and optimizer_token can be set either in the agent-level strategy block (as above) or per check in the strategy "productify-scaler" config. See Scaling Policies for the full parameter reference.

Running Nomad Autoscaler ​

bash
nomad-autoscaler agent \
  -config=/etc/nomad-autoscaler/autoscaler.hcl \
  -plugin-dir=/opt/nomad-autoscaler/plugins

Complete Stack Deployment ​

Docker Compose ​

yaml
services:
  optimizer:
    image: ghcr.io/productifyfw/autoscaler/optimizer:latest
    ports:
      - "8015:8015"
    environment:
      CONFIG_PATH: /app/config.ini
    volumes:
      - ./config.ini:/app/config.ini:ro
    restart: unless-stopped

  nomad-autoscaler:
    image: ghcr.io/productifyfw/autoscaler-nomadscaler:latest
    volumes:
      - ./autoscaler.hcl:/etc/autoscaler.hcl:ro
    command: agent -config=/etc/autoscaler.hcl
    environment:
      NOMAD_ADDR: http://nomad:4646
    depends_on:
      - optimizer
    restart: unless-stopped

Nomad Job (Complete) ​

A complete reference job that runs the Nomad Autoscaler agent, the Optimizer, and a Prometheus instance in a single group is at autoscaler/nomadscaler/config/autoscaler.hcl. That file uses the locally built productify/scaler:local and productify/optimizer:local images; the abbreviated version below substitutes the published ones:

hcl
job "autoscaler" {
  datacenters = ["dc1"]

  group "autoscaler" {
    count = 1

    network {
      port "optimizer_port" {}
      port "optimizer_port_prometheus" {}
    }

    task "autoscaler" {
      driver = "docker"

      config {
        image   = "ghcr.io/productifyfw/autoscaler-nomadscaler:latest"
        command = "nomad-autoscaler"
        args    = ["agent", "-config", "${NOMAD_TASK_DIR}/config.hcl"]
      }

      template {
        data = <<EOF
plugin_dir = "/plugins"

nomad {
  address = "http://{{env "attr.unique.network.ip-address" }}:4646"
}
apm "nomad" {
  driver = "nomad-apm"
  config = {
    address = "http://{{env "attr.unique.network.ip-address" }}:4646"
  }
}
strategy "productify-scaler" {
  driver = "productify-scaler"

  config = {
    {{with nomadVar "nomad/jobs/autoscaler/autoscaler/autoscaler" -}}
    optimizer_token = "{{.TOKEN}}"
    {{- end}}
    optimizer_url   = "http://{{env "NOMAD_IP_optimizer_port"}}:{{env "NOMAD_PORT_optimizer_port"}}"
  }
}
EOF
        destination = "${NOMAD_TASK_DIR}/config.hcl"
      }

      resources {
        cpu    = 200
        memory = 256
      }
    }

    task "optimizer" {
      driver = "docker"

      config {
        image = "ghcr.io/productifyfw/autoscaler/optimizer:latest"
        ports = ["optimizer_port"]
      }

      env {
        CONFIG_PATH = "${NOMAD_TASK_DIR}/config.ini"
        PORT        = "${NOMAD_PORT_optimizer_port}"
      }

      template {
        data = <<EOF
[main]
loglevel=info
api_loglevel=warning
only_test_data=false
enable_test_metrics=false
prometheus_url=http://{{env "NOMAD_IP_optimizer_port_prometheus"}}:{{env "NOMAD_PORT_optimizer_port_prometheus"}}
port={{env "NOMAD_PORT_optimizer_port"}}
{{ with nomadVar "nomad/jobs/autoscaler/autoscaler/optimizer" -}}
token={{.TOKEN}}
{{- end }}
EOF
        destination = "${NOMAD_TASK_DIR}/config.ini"
      }

      resources {
        cpu    = 768
        memory = 512
      }
    }

    # A prometheus task also runs in this group; see
    # autoscaler/nomadscaler/config/autoscaler.hcl for the full job.
  }
}

Configuration ​

Optimizer Configuration ​

See Optimizer Configuration for details.

Key config.ini settings ([main] section):

  • port - HTTP listen port (default: 8015)
  • prometheus_url - Prometheus server to read metrics from
  • token - Shared token required on /optimize requests
  • loglevel / api_loglevel - Logging levels
  • only_test_data - Use synthetic test data instead of Prometheus
  • enable_test_metrics - Serve a synthetic test-metrics endpoint (port set by metrics_port, default: 8017)

Plugin Configuration ​

See Scaling Policies for details.

Key settings:

  • optimizer_url - Optimizer service URL
  • optimizer_token - Token for authenticating with the Optimizer
  • min / max - Replica bounds
  • metric_app_name - The app label on the pfy_* demand series (an application UUID for anything pfy nomad render emits). Not a Nomad identifier
  • group - Nomad task group of the scaling block (recommended; the optimizer's task_group label)
  • task - Nomad task inside that group (the optimizer's task label on the nomad_client_allocs_* capacity queries). Omitted, it falls back to metric_app_name, which measures nothing unless the task is named after the app label
  • cache_size - Seconds each optimizer response covers (default: 10). The response holds ceil(cache_size / step_seconds) values; the MILP plans milp_startup_delay steps further and returns the actionable tail. At or below step_seconds (15 by default) you get one value per call, so the count moves at most one scale step between calls; see Known Issues. It also bounds how long a scale-down has to pay for itself: a replica is released only once the replica_cost saved over the actionable horizon reaches milp_shutdown_cost, so a short cache_size on a small allocation holds the count; see Configuration
  • evaluation_interval - How often to evaluate (policy-level)
  • cooldown - Minimum time between scaling actions (policy-level)

The task key is not optional in practice. The optimizer selects the nomad_client_allocs_* capacity series by the Nomad task name it carries; a strategy block that omits task falls back to metric_app_name, which is the app label (an application UUID on rendered jobs), so the capacity query matches nothing and the app is either answered 503 metrics_unavailable on every evaluation (replicas running) or sized from the fallback constants (at zero replicas). Set task = "<the Nomad task name>" on every policy, and re-render any job rendered without it. See Known Issues.

Health Checks ​

Optimizer Health ​

bash
curl http://localhost:8015/health

Nomad Autoscaler Status ​

bash
curl http://localhost:8080/v1/health

Policy Status ​

bash
nomad scaling policy list

Monitoring ​

Optimizer Metrics ​

The Optimizer exposes its own always-on Prometheus metrics at GET /metrics on the API port (default 8015: request counts, solve behaviour, cost breakdowns). Separately, when enable_test_metrics=true is set in config.ini, it serves a synthetic test-metrics endpoint on metrics_port (default 8017) that generates load-pattern data for development and testing. Never enable it in production.

Nomad Autoscaler Metrics ​

With telemetry { prometheus_metrics = true } in the agent configuration, the Nomad Autoscaler exposes Prometheus metrics on its HTTP endpoint:

bash
curl http://localhost:8080/v1/metrics?format=prometheus

Troubleshooting ​

Optimizer Not Responding ​

Check that:

  • Service is running
  • Port 8015 (or configured port) is accessible
  • Health endpoint returns 200
  • Logs for errors

Plugin Not Loading ​

Check that:

  • Plugin binary in correct directory
  • Binary has execute permissions
  • Plugin version matches autoscaler version
  • Logs show plugin registration

No Scaling Actions ​

Check that:

  • Policy is enabled
  • Metrics are being collected
  • Optimizer is reachable
  • Cooldown period hasn't been triggered
  • Min/max bounds allow scaling

See Also ​