Nomadscaler Plugin

productify-scaler is a Nomad Autoscaler strategy plugin written in Go (autoscaler/nomadscaler/). A strategy plugin's job is to turn the metrics an APM plugin collected into a desired replica count. This one delegates the decision to the Optimizer Service and caches the answer so it does not call the optimizer on every evaluation. See Configuration for the full key reference.

What it does

On every policy check evaluation the plugin:

1. Resolves its settings: optimizer_url, optimizer_token, min, max, metric_app_name, group, task, cache_size, from the check's strategy block, then the agent-level plugin config, then (URL and token only) the OPTIMIZER_URL / OPTIMIZER_TOKEN environment variables; group falls back to the check block's own group attribute last. task has no such last resort: the SDK carries no task name, and a scaling policy attaches to a task group that may hold several tasks, so only the policy can say which one is being sized. Missing optimizer_url, optimizer_token or metric_app_name, min < 0, max < 0, or min > max when max > 0 fail the evaluation with an error. An empty group or task is allowed but warned once per check.
2. Skips the evaluation with a warning if the APM returned no samples.
3. Looks up the cache for this check (keyed by check name + source + resolved group + metric_app_name + task + query, so policies never share decisions, not even two jobs with copy-pasted checks, and not two checks differing only in which Nomad task they size).
4. If the cache still covers the current moment, serves the cached value. Otherwise sends POST <optimizer_url>/optimize with the check details (name, source, group, metric_app_name, task, query; task is omitted from the body when unset), current_replicas, min_replicas, max_replicas, the APM samples as metrics (RFC 3339 UTC timestamps) and cache_size.
5. Stores the returned desired array (truncated to cache_size values, warned once per check, if the optimizer returned more) with step_seconds and a timestamp, and serves desired[elapsed / step_seconds].
6. On failure, serves the cache if possible, otherwise the fallback policy (below).
7. Clamps the result into [min, max] and returns it with the scale direction (up, down, or none).

Build and install

cd autoscaler/nomadscaler
./build.sh

build.sh needs Go 1.25.10 or newer and Docker. It writes the binary to ./bin/productify-scaler and builds the image productify/scaler:local: hashicorp/nomad-autoscaler:0.5.0 with the plugin at /plugins/productify-scaler. To build only the binary:

CGO_ENABLED=0 go build -o bin/productify-scaler .

The binary is loaded by the Nomad Autoscaler agent from its plugin_dir; it is not a standalone program. The reference Nomad job config/autoscaler.hcl runs the image with nomad-autoscaler agent -config <rendered config.hcl>. To run the image by hand with your own agent config:

docker run --rm -v "$PWD/config.hcl:/config.hcl:ro" --entrypoint nomad-autoscaler \
  productify/scaler:local agent -config /config.hcl

Published images: tagged releases push ghcr.io/productifyfw/autoscaler-nomadscaler:<version> and :latest (GoReleaser, linux/amd64; current release 1.2.0). That package is public and can replace productify/scaler:local above. CI also pushes ghcr.io/productifyfw/autoscaler/nomadscaler:nightly and one tag per commit SHA from main, but that package is private and needs a docker login ghcr.io with a token carrying read:packages.

Configuration

Agent config (config.hcl), as rendered by config/autoscaler.hcl:

plugin_dir = "/plugins"

nomad {
  address = "http://nomad-address:4646"
}

apm "nomad" {
  driver = "nomad-apm"
  config = {
    address = "http://nomad-address:4646"
  }
}

strategy "productify-scaler" {
  driver = "productify-scaler"

  config = {
    optimizer_token = "supersecrettoken"
    optimizer_url   = "http://optimizer:8015"
    # metrics_addr  = "0.0.0.0:9101"   # optional: plugin /metrics listener
  }
}

Per-check strategy block (inside a job's scaling { policy { check { ... } } }):

strategy "productify-scaler" {
  min             = 1        # optional, default 1
  max             = 10       # optional, default 10; 0 means no ceiling
  metric_app_name = "my-app" # required: the `app` label on the pfy_* series
  group           = "web"    # recommended: the Nomad task group this scaling block lives in
  task            = "server" # recommended: the Nomad task inside it, for the alloc queries
  cache_size      = 30       # optional, default 10 (seconds), max 600; above step_seconds gives a multi-step ramp
  # optimizer_url / optimizer_token if not set in the agent config
}

Key	Required	Default	Notes
optimizer_url	yes	none	check, agent config, or OPTIMIZER_URL
optimizer_token	yes	none	check, agent config, or OPTIMIZER_TOKEN; sent as Authorization: Bearer
metric_app_name	yes	none	app label of the pfy_* series. Not a Nomad identifier; for anything pfy nomad render emits it is the application UUID
task	no (recommended)	empty; the optimizer falls back to metric_app_name	check or agent config; the Nomad task, sent to the optimizer as task (its task label on the nomad_client_allocs_* queries). No fallback to an SDK field, because a policy attaches to a task group that may hold several tasks. Empty: the plugin warns once per check and the optimizer selects by metric_app_name instead
group	no (recommended)	check block's group attribute, else empty	check or agent config; the Nomad task group, sent to the optimizer as group (its task_group label). Empty: the optimizer matches by task only and the plugin warns once per check
min	no	1	>= 0; <= max when max > 0. 0 permits zero replicas here, but is not sufficient for scale-to-zero; see Known Issues
max	no	10	0 means no ceiling; negative values are rejected
cache_size	no	10	seconds; clamped to 600. At or below step_seconds the response is a single value, so the count moves by at most one scale step per optimizer call; see Cache behaviour
metrics_addr	no	unset	agent config only

The strategy's min/max and the job's scaling { min max } are independent: the plugin clamps to its own bounds, then Nomad applies the scaling block's. Keep them in agreement.

Cache behaviour

• The optimizer returns desired (one count per step) and step_seconds (seconds each count covers, 15 by default). The plugin caps step_seconds at 3600 so a bad value cannot silence re-polling for days.
• At most cache_size values are kept: a response carrying more desired values than requested is truncated (warned once per check), so an over-long response cannot silence re-polling either.
• Value served at elapsed time e since the response: desired[e / step_seconds], capped at the last index.
• The cache is time-exhausted when e >= len(desired) × step_seconds. Only then does the plugin call the optimizer again; there is no fixed use count, because Nomad's evaluation_interval is configurable.
• Clamping into [min, max] happens at serve time, so lowering max in the policy takes effect on the next evaluation even for cached values.
• A cache entry that has not been used for 24 hours (a deleted or renamed policy) is evicted.

Example, cache_size = 45, step_seconds = 15, response [3, 5, 7]:

Elapsed (s)  | 0–14 | 15–29 | 30–44 | 45+
Cache index  |  0   |  1    |  2    | time-exhausted
Count served |  3   |  5    |  7    | fresh request (or fallback)

With the defaults (cache_size = 10, 15-second step) each response holds one value that is served for 15 seconds. The optimizer plans past the step it pins to the current count and returns the actionable tail, so that one value is a decision; it is also a single step of ramp per optimizer call, so the count moves by at most milp_max_scale_up / milp_max_scale_down (2 each by default) before the next call, and a scale-down has to beat milp_shutdown_cost over that one step, which an idle app under 300 MHz of allocated CPU does not. See Cost coefficients.

Failure handling

• Each optimizer request gets 2 attempts with a 2-second timeout and a 200 ms backoff. Only network errors and 5xx responses are retried; 4xx responses (bad token, validation errors) are not.
• After a failed request, or a 200 with an empty desired, the plugin serves the cache if it still covers the current moment. If not, it applies the fallback policy: if any APM sample in this evaluation is > 0 and current_replicas == 0, use 1; otherwise hold current_replicas; then clamp into [min, max].
• The fallback never forces at least one replica, so it does not itself override a min = 0 bound. The same policy is implemented in the optimizer (optimizer/optimizer/fallback.py). It only runs when the optimizer cannot be reached or returns nothing usable; on the normal path the MILP decides, and min = 0 alone is not enough to make scale-to-zero work; see Known Issues.

Plugin metrics

When metrics_addr is set in the agent-level plugin config, the plugin serves Prometheus metrics at http://<metrics_addr>/metrics:

Metric	Type	Meaning
productify_scaler_optimizer_requests_total	counter	Optimizer request attempts
productify_scaler_optimizer_request_duration_seconds	histogram	Latency of each attempt
productify_scaler_cache_hits_total	counter	Decisions served from the cache
productify_scaler_fallbacks_total	counter	Times the plugin fell back after a failed or empty response

Without metrics_addr the counters exist but nothing can scrape them; the plugin logs prometheus metrics not exposed at startup.

Choosing the check query

The plugin works with any APM plugin. The samples are used only (a) to detect whether demand is present for the fallback policy and (b) forwarded to the optimizer, which appends them to the request-rate series it forecasts only when the check's query names that series (request_rate_metric, default pfy_total_requests). The plugin sends the check's query verbatim, so a query that returns the application's request rate is the best fit.

Nomad APM (source = "nomad-apm") task-group queries have the form <op>_<metric> with op one of sum, avg, max, min and metric one of cpu, cpu-allocated, memory, memory-allocated:

check "cpu" {
  source = "nomad-apm"
  query  = "avg_cpu"
}

Prometheus APM (source = "prometheus"), for example the proxy's request counter for the app. The proxy registers the counter as pfy_total_requests and exposes it verbatim, without the _total suffix a Prometheus counter usually carries, and the optimizer queries the same name:

check "requests" {
  source = "prometheus"
  query  = "sum(rate(pfy_total_requests{app=\"my-app\"}[30s]))"
}

Troubleshooting

Symptom	What to check
Plugin does not load	The binary must be in the agent's plugin_dir and executable, and the agent config must contain a strategy "productify-scaler" { driver = "productify-scaler" } block. Set log_level = "DEBUG" in the agent config and read nomad alloc logs <alloc-id> autoscaler
missing required field	metric_app_name must be in the check; optimizer_url and optimizer_token must be in the check, the agent config, or the environment
no `group` configured for check	Neither the strategy block, the agent config nor the check block sets group. The evaluation still runs, but the optimizer matches the task's allocations by task name only; set group to the Nomad task group
no `task` configured for check	Neither the strategy block nor the agent config sets task. The evaluation still runs, but the optimizer selects nomad_client_allocs_* by metric_app_name instead, which matches a real series only where the Nomad task happens to be named after the app label. Set task to the Nomad task name; see Known Issues
optimizer request failed, falling back	The log line carries the error: connection refused / timeout (optimizer down, wrong URL, or the slow first solve after optimizer startup), optimizer returned status 401 (token mismatch), 422 (invalid bounds), 503 (optimizer cannot read Prometheus). Test the URL from the agent's network: curl http://<optimizer>:8015/health
desired count outside [min,max], clamping	The optimizer or cache produced a value outside the strategy bounds; the plugin clamped it. Usually a min/max mismatch between the strategy and the request
Nothing scales although the plugin returns a new count	Nomad's cooldown or the job's scaling { min max } block may be blocking the action

See also

• Architecture
• Configuration
• Optimizer Service
• Scaling Policies