Quick Start

Run the two autoscaler components on your machine and make a first scaling request by hand. No Nomad cluster is needed for the first half; the Deploy to Nomad section needs one.

Prerequisites

• Python 3.12+ and Poetry, to run the optimizer from source
• Go 1.25.10 or newer, to build the plugin
• Docker, for the container images (nomadscaler/build.sh builds an image, so it needs Docker too)
• A Nomad cluster with the Nomad Autoscaler, only for the deployment step
• Prometheus, only for real metrics; the optimizer ships a test-data mode

1. Run the optimizer

From source

cd autoscaler/optimizer
poetry install

# Solve the bundled demand fixture once and print the result
poetry run testcalc
# Desired replicas: [0, 1, 0, 0, 1, 3, 4, 6, 4, 2, 0, 2, 4, 6, 8, 10, 12, 14, 16, 18, 20]

# Start the HTTP service (reads ./config.ini; listens on port 8015)
poetry run web

The committed optimizer/config.ini is a development config: only_test_data=true (canned metrics, no Prometheus needed), enable_test_metrics=true (a fake pfy_* metrics generator on port 8017) and token=supersecrettoken. The service logs a warning for each of these at startup; that is expected in development.

With Docker

cd autoscaler/optimizer
docker build -t productify/optimizer:local .
docker run --rm -p 8015:8015 productify/optimizer:local

The image bakes in the same development config.ini, so it starts without a mount. The container reads its listen port from the PORT environment variable (default 8015), not from config.ini. For production, mount your own config and point CONFIG_PATH at it; see Autoscaler Deployment.

The productify/optimizer:local tag is what the reference Nomad job (nomadscaler/config/autoscaler.hcl) expects. The release workflow publishes the optimizer as ghcr.io/productifyfw/autoscaler/optimizer:<version> and :latest (current release 1.2.0). That package is private: an anonymous docker pull is refused with a 401 until the package is made public (Known Issues). Log in to ghcr.io with a GitHub personal access token that has the read:packages scope, then pull and retag:

echo "$GITHUB_TOKEN" | docker login ghcr.io -u <github-username> --password-stdin
docker pull ghcr.io/productifyfw/autoscaler/optimizer:1.2.0
docker tag ghcr.io/productifyfw/autoscaler/optimizer:1.2.0 productify/optimizer:local

2. Make a first request

Check the service is up:

curl http://localhost:8015/health
# {"response":"ok"}

Ask for a scaling decision. The token goes in an Authorization: Bearer header:

curl -s http://localhost:8015/optimize \
  -H 'Content-Type: application/json' \
  -H 'Authorization: Bearer supersecrettoken' \
  -d '{
    "check": {"name": "demo", "source": "nomad-apm", "group": "demo", "metric_app_name": "test", "task": "webapp"},
    "current_replicas": 3,
    "min_replicas": 1,
    "max_replicas": 10,
    "cache_size": 10
  }'

Expected shape (values differ):

{
  "desired": [3],
  "step_seconds": 15,
  "cost_breakdown": {
    "replica_cost": 5.1,
    "expected_shortfall": 342.0,
    "backlog_carry_cost": 0.0,
    "scale_event_cost": 0.0
  },
  "engine": "milp",
  "fallback": false
}

cache_size = 10 with the default 15-second step gives one value, covering the next 15 seconds. The MILP plans one step past the one it pins to current_replicas and returns the tail, so the value is a decision, but a single value is also a single step of ramp: the count moves by at most milp_max_scale_up / milp_max_scale_down per call. Try "cache_size": 45 to get three values and watch a multi-step plan. See cache_size.

First request is slow

The first /optimize call for an application fits the SARIMAX model. On the bundled test data this takes well over a minute; later calls reuse the cached fit and return in about two seconds. The Nomad plugin gives each attempt two seconds, so it falls back (hold current count) until the fit is cached. This is expected on a cold start.

3. Build the Nomadscaler plugin

cd autoscaler/nomadscaler
./build.sh

build.sh compiles the plugin to ./bin/productify-scaler and then builds the productify/scaler:local Docker image: the hashicorp/nomad-autoscaler:0.5.0 image with the plugin installed at /plugins/productify-scaler. The published equivalent is ghcr.io/productifyfw/autoscaler-nomadscaler:1.2.0 (public, no login needed).

To build only the binary (no Docker):

cd autoscaler/nomadscaler
CGO_ENABLED=0 go build -o bin/productify-scaler .

The plugin is a Nomad Autoscaler plugin, not a standalone program: running the binary directly only prints a go-plugin handshake error. It is loaded by the nomad-autoscaler agent from plugin_dir.

4. Configure the Nomad Autoscaler agent

The agent needs a strategy "productify-scaler" block naming the plugin. optimizer_url and optimizer_token can live here (shared by every policy) or in each policy's check. The reference job nomadscaler/config/autoscaler.hcl renders this config:

plugin_dir = "/plugins"

nomad {
  address = "http://nomad-address:4646"
}

apm "nomad" {
  driver = "nomad-apm"
  config = {
    address = "http://nomad-address:4646"
  }
}

strategy "productify-scaler" {
  driver = "productify-scaler"

  config = {
    optimizer_token = "supersecrettoken"
    optimizer_url   = "http://optimizer:8015"
    # metrics_addr  = "0.0.0.0:9101"   # optional: expose the plugin's own /metrics
  }
}

optimizer_token and optimizer_url can also come from the OPTIMIZER_TOKEN / OPTIMIZER_URL environment variables of the agent process.

5. Add a scaling policy to your job

Add a scaling block to the task group you want to scale. This is nomadscaler/config/test.hcl, trimmed:

job "webapp" {
  datacenters = ["dc1"]

  group "demo" {
    count = 3

    network {
      port "webapp_http" {}
    }

    scaling {
      enabled = true
      min     = 0
      max     = 10

      policy {
        evaluation_interval = "3s"
        cooldown            = "10s"

        check "productify-scale-check" {
          source = "nomad-apm"
          query  = "avg_cpu-allocated"

          strategy "productify-scaler" {
            min             = 0
            max             = 10
            metric_app_name = "test"   # the `app` label on the pfy_* series
            group           = "demo"   # the Nomad task group this scaling block lives in
            task            = "webapp" # the task inside it, i.e. the task block below
            cache_size      = 10
          }
        }
      }
    }

    task "webapp" {
      driver = "docker"

      config {
        image = "hashicorp/demo-webapp-lb-guide"
        ports = ["webapp_http"]
      }

      resources {
        cpu    = 100
        memory = 16
      }
    }
  }
}

Three names appear in that strategy block, and they are three different identities. metric_app_name (test here) must match the app label on the application's pfy_* metrics, or an app_aliases entry on the optimizer; it is what the demand forecast is built from, and for an application job rendered by pfy nomad render it is the application UUID, not a name you choose. group (demo) is the Nomad task group the scaling block lives in. task (webapp) is the Nomad task inside that group whose allocated CPU/memory the optimizer reads, from nomad_client_allocs_*{task_group="<group>", task="<task>"}; those series come from the Nomad client and carry the job's own names, never the app label.

group and task are both optional, and both degrade rather than fail: without group the optimizer matches by task name alone, and without task it falls back to metric_app_name. The plugin warns once per check for each. See Scaling Policies for every field.

min = 0 in this fixture does not give you scale-to-zero

min = 0 permits zero replicas in the strategy's clamp, and the optimizer can return zero for an idle app, but only when releasing the replica pays for itself: a replica is released once the replica_cost it saves over the actionable horizon reaches milp_shutdown_cost (0.3), and at the shipped cost_cpu_mhz_hour = 0.001 a small allocation does not clear that in one step. With this fixture's fallback_cpu_alloc = 100 MHz, an idle app at 1 replica answers [1] at cache_size = 10 and [1, 1] at 30, and reaches [0, 0, 0] only at 45. See min and max.

A working scale-to-zero cycle needs more than this key. On the Manager-managed path min: 0 cannot be set, and with this policy's source = "nomad-apm" the app is not evaluated at all once it reaches zero replicas: the APM returns metric not found with no running allocation and the strategy plugin is never called. Read Known Issues and the APM blocker before relying on min = 0.

Deploy to Nomad

nomadscaler/config/autoscaler.hcl runs the autoscaler agent, the optimizer and a Prometheus in one job. It uses the locally built productify/scaler:local and productify/optimizer:local images, so build (or retag) them on every client node first, and set the TOKEN item in the Nomad variables nomad/jobs/autoscaler/autoscaler/autoscaler and nomad/jobs/autoscaler/autoscaler/optimizer (the job templates read the token from them).

cd autoscaler/nomadscaler
nomad job run config/autoscaler.hcl
nomad job run config/test.hcl        # the demo job with the scaling policy above

Then watch the agent:

nomad alloc logs -f <autoscaler-alloc-id> autoscaler

You should see the productify-scaler plugin load, then one POST /optimize per check when its cache is empty or time-exhausted, and debug lines like serving cached value in between.

Troubleshooting

Optimizer does not start

• config.ini not found: the optimizer looks for optimizer/optimizer/config.ini, then optimizer/config.ini, unless CONFIG_PATH points elsewhere. Run from autoscaler/optimizer or set CONFIG_PATH.
• prometheus_url is unset or malformed: with only_test_data=false a valid http(s):// prometheus_url is required at startup.
• Configured engine ... is unknown: the engine key must be one of the names the service lists in the error.
• Port 8015 (or your port) already in use.

Every request is 401

The optimizer denies all requests when token is empty or unset. Set a non-empty token in config.ini, and send the same value as Authorization: Bearer <token> (or optimizer_token on the plugin side).

Plugin does not load

• The binary must be in the agent's plugin_dir (/plugins in the image) and executable.
• The agent config must contain a strategy "productify-scaler" { driver = "productify-scaler" } block.
• Read the agent log: nomad alloc logs <alloc-id> autoscaler.

Policy evaluates but nothing scales

• missing required field in the agent log: metric_app_name is required per check; optimizer_url/optimizer_token must be set per check, in the agent's strategy config, or as environment variables.
• optimizer request failed, falling back: the optimizer is unreachable, slow (first call after startup), or returned an error. The reason is in the log line.
• APM returned no metrics; skipping evaluation: the check's query returned nothing; check the APM plugin and query.
• The plugin clamps every decision into the strategy's [min, max], and Nomad enforces the scaling { min max } block afterwards. Keep the two in agreement.
• Nomad's cooldown blocks actions for the configured period after each scaling action.

Next steps

• Architecture: how forecasting and the MILP fit together
• Configuration: all settings for both components
• Scaling Policies: policy reference and tuning
• Autoscaler Deployment: production deployment