Productify Autoscaler

The Productify Autoscaler decides how many replicas a Nomad job should run. Instead of reacting to a CPU threshold, it forecasts demand a few steps ahead and picks the cheapest replica count that still meets that demand. Use it when you run Productify applications on Nomad and want proactive, cost-aware scaling.

It has two components, both in the autoscaler/ repository:

• The Nomadscaler plugin (nomadscaler/, Go) is a Nomad Autoscaler strategy plugin named productify-scaler. Nomad calls it on every policy evaluation; it asks the optimizer what to do and caches the answer.
• The optimizer service (optimizer/, Python) is an HTTP service. It reads application metrics from Prometheus, forecasts demand with a SARIMAX time-series model, and solves a small mixed-integer program (MILP) to choose replica counts.

The current release is v1.2.0 for both components.

How a scaling decision is made

1. The Nomad Autoscaler evaluates a scaling policy and calls the productify-scaler strategy plugin with the current replica count.
2. If the plugin's cache for that check still covers the current moment, it serves the cached value (see step 6).
3. Otherwise the plugin sends POST /optimize to the optimizer with the check details, the current/min/max replica counts, the freshest metric samples it received, and cache_size (how many seconds ahead to plan; default 10).
4. The optimizer loads the last training_window_minutes (default 240) of pfy_* metrics for the application from Prometheus, forecasts the request rate, converts the forecast into a demand series, and solves the MILP. It plans milp_startup_delay + ceil(cache_size / step_seconds) steps and returns the actionable tail of that plan: ceil(cache_size / step_seconds) values.
5. It returns one desired replica count per step plus step_seconds, the number of seconds each value covers. step_seconds is derived from the Prometheus scrape interval (default 15).
6. The plugin stores the values with a timestamp and, on each evaluation, serves the value at index elapsed / step_seconds. When elapsed time passes len(values) × step_seconds, the cache is time-exhausted and the plugin requests fresh values.
7. The plugin clamps the chosen count into the strategy's [min, max] and returns it. Nomad's target plugin applies the count to the job.

With the defaults (cache_size = 10, step_seconds = 15) each optimizer call returns a single value that is served for 15 seconds. With cache_size = 45 you get three values, one per 15-second step. A single value is also a single step of ramp, so the count moves by at most milp_max_scale_up / milp_max_scale_down per call; see cache_size.

Architecture

┌───────────────────────────────────────────────────┐
│         Nomad Autoscaler agent                    │
│  ┌────────────────────────────────────────────┐   │
│  │   productify-scaler (strategy plugin, Go)  │   │
│  │   - receives policy check evaluations      │   │
│  │   - keeps a per-check prediction cache     │   │
│  │   - returns the desired replica count      │   │
│  └────────────────────────────────────────────┘   │
└───────────────────────────────────────────────────┘
                    │ HTTP  POST /optimize
                    ▼
┌───────────────────────────────────────────────────┐
│         Optimizer service (Python, port 8015)     │
│  - SARIMAX forecast of the request rate           │
│  - MILP solve for replica counts per step         │
│  - GET /health, GET /metrics                      │
└───────────────────────────────────────────────────┘
                    │ HTTP  /api/v1/query_range, /api/v1/query
                    ▼
          ┌───────────────────┐
          │    Prometheus     │
          │ pfy_* app metrics │
          │ nomad_client_*    │
          └───────────────────┘

Request and response

The plugin sends (token in an Authorization: Bearer <token> header):

{
  "check": {
    "name": "productify-scale-check",
    "source": "nomad-apm",
    "group": "demo",
    "metric_app_name": "my-app",
    "task": "webapp"
  },
  "current_replicas": 3,
  "min_replicas": 1,
  "max_replicas": 10,
  "metrics": [{ "timestamp": "2026-01-01T12:00:00Z", "value": 42.0 }],
  "cache_size": 45
}

check.group and check.task are the strategy's group and task keys: the Nomad task group the scaling block lives in, and the task inside it. The optimizer uses them as the task_group and task labels when it reads that task's allocated CPU and memory. Neither is the app label: metric_app_name selects the pfy_* demand series and is the application UUID for anything pfy renders. group may be empty (the match then spans task groups) and task may be empty (the optimizer then falls back to metric_app_name, which selects a real series only when a task is named after the app label). Set both.

The optimizer answers (here for cache_size = 45, three 15-second steps):

{
  "desired": [3, 5, 7],
  "step_seconds": 15,
  "cost_breakdown": {
    "replica_cost": 25.5,
    "expected_shortfall": 630.0,
    "backlog_carry_cost": 0.0,
    "scale_event_cost": 2.0
  },
  "engine": "milp",
  "fallback": false
}

The plugin then serves desired[0] for 0 to 14 s, desired[1] for 15 to 29 s and desired[2] for 30 to 44 s after the response, and requests fresh values from 45 s on.

What happens when something fails

When the optimizer is unreachable or returns 5xx, the plugin retries once (2-second timeout per attempt, 200 ms backoff). If that fails it serves the cache if it still covers the current moment; otherwise it applies the fallback policy: hold the current count, or go to 1 replica if the check's metric shows demand and no replica is running, then clamp into [min, max]. 4xx responses (bad token, validation error) are not retried.

When the optimizer cannot reach Prometheus, it returns 503 {"error": "metrics_unavailable"} and the plugin falls back as above.

When the MILP solve fails inside the optimizer, the optimizer applies the same fallback policy itself and returns the result with "fallback": true.

The fallback never forces at least one replica, so it does not itself override a min = 0 bound. The policy is defined once in optimizer/optimizer/fallback.py and mirrored in nomadscaler/plugin/web.go. It is a failure path only; min = 0 on the normal path is not enough to make scale-to-zero work. See Known Issues.

Requirements

• A Nomad cluster running the Nomad Autoscaler (the plugin image is built on hashicorp/nomad-autoscaler:0.5.0)
• Go 1.25.10 or newer to build the plugin
• Python 3.12+ and Poetry to run the optimizer outside Docker
• Prometheus scraping the pfy_* application metrics and Nomad's /v1/metrics (the optimizer has a test-data mode that needs neither)

Next steps

1. Quick Start: build both components and make a first /optimize call
2. Architecture: data flow, forecasting and the MILP model in detail
3. Configuration: every plugin and optimizer setting
4. Scaling Policies: the scaling block for your Nomad job
5. Nomadscaler Plugin and Optimizer Service: component reference
6. Autoscaler Deployment: running it in production