Autoscaler Architecture

A scaling decision travels from Prometheus metrics to a new replica count through the components below. File paths refer to the autoscaler/ repository.

System overview

┌────────────────────────────────────────────────────────┐
│              Nomad Autoscaler agent                    │
│                                                        │
│  ┌──────────────────────────────────────────────────┐  │
│  │  APM plugin (nomad-apm or prometheus)            │  │
│  │  - runs the check's `query`                      │  │
│  │  - hands the samples to the strategy             │  │
│  └──────────────────────────────────────────────────┘  │
│                       │                                │
│                       ▼                                │
│  ┌──────────────────────────────────────────────────┐  │
│  │  Strategy plugin: productify-scaler (Go)         │  │
│  │  - POST /optimize when its cache is exhausted    │  │
│  │  - serves cached values by elapsed time          │  │
│  │  - clamps into [min, max], returns the count     │  │
│  └──────────────────────────────────────────────────┘  │
│                       │                                │
│                       ▼                                │
│  ┌──────────────────────────────────────────────────┐  │
│  │  Target plugin (nomad-target)                    │  │
│  │  - scales the task group to the returned count   │  │
│  └──────────────────────────────────────────────────┘  │
└────────────────────────────────────────────────────────┘
                       │ HTTP
                       ▼
┌────────────────────────────────────────────────────────┐
│           Optimizer service (Python / FastAPI)         │
│                                                        │
│   Prometheus ──► metrics window ──► SARIMAX forecast   │
│                        │                  │            │
│                        ▼                  ▼            │
│              demand conversion ──► MILP solve          │
│                                       │                │
│                                       ▼                │
│                     desired[] + step_seconds           │
└────────────────────────────────────────────────────────┘

The strategy plugin never talks to Nomad itself. It receives the current count from the agent and returns the desired count; the agent's target plugin applies it.

Components

Nomadscaler plugin (nomadscaler/)

A Nomad Autoscaler strategy plugin (plugin/plugin.go, plugin/web.go). One plugin process serves every policy that uses the productify-scaler strategy. For each check it:

• reads the strategy config (optimizer_url, optimizer_token, min, max, metric_app_name, group, task, cache_size), falling back to the agent-level plugin config and to the OPTIMIZER_URL / OPTIMIZER_TOKEN environment variables for the URL and token;
• keeps one prediction cache per check identity (check name + source + resolved group + metric_app_name + task + query). task is part of that identity because two checks differing only in it describe different Nomad tasks, and so different per-replica capacities;
• sends POST /optimize when the cache is empty or time-exhausted, retrying once on network errors and 5xx;
• clamps every count it returns (fresh, cached or fallback) into [min, max];
• optionally exposes its own Prometheus metrics (metrics_addr).

Optimizer service (optimizer/)

A FastAPI service (optimizer/web.py) with five routes: POST /optimize, GET /models, GET /serving/{group}/{app}, GET /health, GET /metrics. Per /optimize request it:

1. authenticates the bearer token (constant-time compare; an unset server token denies everything);
2. resolves the decision engine (engine request field, else the canary route, else the engine config key, else milp) and rejects unknown names with 400;
3. loads the metrics window from Prometheus (optimizer/metrics.py) or, with only_test_data=true, from fixtures;
4. forecasts the request rate and converts it to demand (optimizer/demand.py);
5. reads allocated CPU/memory for the task and derives per-replica capacity and cost (optimizer/costs.py);
6. runs the engine (optimizer/engines/) through the safety shell (optimizer/engines/safety.py);
7. returns desired, step_seconds, cost_breakdown, engine, fallback, drift_score, regime.

Data flow

1. Metrics collection

Proxy / Manager ──► Prometheus ◄── Nomad /v1/metrics

The optimizer queries Prometheus over GET /api/v1/query_range (application series) and GET /api/v1/query (allocation series). The application queries, all wrapped in sum by (app) (...) and filtered to app="<metric_app_name>" (or an alias from app_aliases):

Series	PromQL used	Feeds
pfy_total_requests	rate(...[2×scrape_interval])	forecast target (request rate)
pfy_response_time_seconds_sum / _count	rate(sum) / rate(count)	demand: per-request work
pfy_executor_queue_all_total	rate(...)	two-stream mode: queue arrivals
pfy_executor_queue_processed_total	rate(...)	loaded, not used in the demand formula
pfy_executor_queue_waiting	gauge	demand: backlog pressure / initial backlog
pfy_executor_queue_process_time_seconds_sum / _count	rate(sum) / rate(count)	demand: per-item queue work
pfy_authentication_awaiting_users	gauge	demand: users waiting to log in
pfy_activator_pending_requests	gauge	demand: requests held through a cold start (additive interactive level, weight 1.0)

Two naming conventions meet in that table. pfy_total_requests has no _total suffix: prometheus/client_golang does not append one to an explicitly named counter, and the proxy's promhttp.Handler() emits the registered name verbatim. The Manager's queue counters are registered with _total in their own names, so those rows keep the suffix. Every name is a constant in optimizer/optimizer/metrics.py, the fake exporter imports the same constants, and a test fails if a query names a series no producer registers.

The allocation queries use two Nomad identifiers from the check, group (the Nomad task group) and task (the Nomad task), and neither of them is the app label. The Nomad client exports these series labelled with the job's own task_group/task names, which never carry the application UUID. When group is empty the task_group matcher is omitted and the optimizer warns once per task that the policy should set group:

nomad_client_allocs_cpu_allocated{task_group="<group>", task="<task>"}      # MHz
nomad_client_allocs_memory_allocated{task_group="<group>", task="<task>"}   # bytes

task is optional on the wire. A policy that omits it falls back to metric_app_name, which selects a real series only where the Nomad task happens to be named after the app label. It is not, for anything pfy renders: there metric_app_name is an application UUID and no task is named after it. See Known Issues.

An empty allocation result while replicas are running is treated as metrics unavailable (503), never as zero capacity. At current_replicas = 0 the series legitimately do not exist: the optimizer then uses the last-known allocation for the (group, task) (cached in-process from its most recent successful query), else the configured fallback_cpu_alloc / fallback_mem_alloc (defaults 100 MHz / 16 MiB), so scale-from-zero can proceed.

That gate decides how a mis-selected task shows up. With replicas running it is loud: the empty result re-raises and the request is a 503, so the optimization is lost and the plugin falls back to its own heuristic. Only at zero replicas is it silent, and only there do the fallback constants size anything.

The window is the last training_window_minutes (default 240). The range-query step is max(scrape_interval, ceil(window_seconds / 10000)), 15 seconds by default, and the same value is returned to the plugin as step_seconds.

2. Policy evaluation

The Nomad Autoscaler runs the check's query through the APM plugin every evaluation_interval and calls the strategy with the samples and the current count. The plugin skips the evaluation (with a warning) if the APM returned no samples.

At zero replicas the strategy is never reached

With the shipped source = "nomad-apm" / query = "avg_cpu-allocated", the APM walks the job's allocations and skips every one that is not running. At zero replicas it therefore finds nothing and returns metric not found, and the Nomad Autoscaler's check runner aborts on that error before calling the strategy plugin, so the plugin's own "no samples" guard above never runs, and the optimizer is never asked. See Known Issues.

3. Prediction request

productify-scaler ──► POST /optimize ──► optimizer

POST /optimize
Authorization: Bearer SECRET
Content-Type: application/json

{
  "check": {
    "name": "productify-scale-check",
    "source": "nomad-apm",
    "group": "demo",
    "metric_app_name": "my-app",
    "task": "webapp"
  },
  "current_replicas": 3,
  "min_replicas": 1,
  "max_replicas": 10,
  "metrics": [{ "timestamp": "2026-01-01T12:00:00Z", "value": 42.0 }],
  "cache_size": 10
}

• metrics are the APM samples from this evaluation. The optimizer appends their values to the request-rate series as the freshest observations only when check.query names the request-rate series (see below).
• cache_size (1 to 3600) is how many seconds ahead to plan. min_replicas > max_replicas (when max_replicas > 0), a cache_size outside 1 to 3600, a group / task / metric_app_name that is not [A-Za-z0-9_.:-]+ (group and task may be empty), a check.name that is empty, longer than 256 characters or contains control characters, or an unknown or out-of-range overrides key is rejected with 422.
• Optional: check.query (the check's APM query line), overrides (per-request MILP parameters, typed and bounded) and engine (per-request engine name). A body token field is accepted but deprecated; use the header.

The optimizer appends metrics[].value to the request-rate series it forecasts only when check.query contains the configured request_rate_metric (default pfy_total_requests). For any other query, such as the reference policy's source = "nomad-apm" with query = "avg_cpu-allocated", whose values are allocated MHz rather than requests per second, they are ignored. The plugin sends the check's query verbatim (check.query). On the plugin side the samples are still used to test whether demand is present for the fallback policy.

4. Forecasting

metrics window ──► resample to step ──► SARIMAX fit (cached) ──► forecast mean + P90 upper band

• The target series is the request rate, resampled to step_seconds and gap-filled (forward, backward, then zeros).
• The model is SARIMAX(order=sarimax_order) (default 1,1,1) with enforce_stationarity=False, enforce_invertibility=False. Order selection is offline: optimizer/analysis/order_selection.py compares candidate orders by AIC on recorded traces; the request path never loops over orders.
• Exogenous inputs (fourier_only strategy, the default): deterministic Fourier terms for daily and weekly seasonality (2 harmonics each). The metric columns do not enter the forecast, so no future-unknown value is frozen into it.
• Fits are cached in-process per (group, metric_app_name). A cached fit is reused and extended with the new tail of the window for 300 seconds, as long as the mean of the last 10 observations is within 50 % of the fitted mean; otherwise it is refit. These two thresholds are constants in optimizer/demand.py, not config keys.
• With fewer than 8 observations, or if the fit or forecast fails or produces non-finite values, the optimizer uses a persistence forecast (repeat the last value).
• The upper band is the 90th percentile of the forecast interval (forecast_quantile = 0.90).

5. Demand conversion

The forecast (and its upper band) is turned into a scalar demand per step. Each term uses its own metric averaged over the last three samples and its own weight from optimizer/weights.py:

demand[t] = forecast_rate[t] × avg_response_time × 0.05      # per-request work
          + queue_waiting × 0.1                                # backlog pressure
          + avg_processing_time × 0.1                          # per-item queue work
          + authentication_awating_users × 0.2
          + activator_pending_requests × 1.0                   # held through a cold start

Every term after the first is a level read from a gauge and added to each step from a recent mean, never forecast and never rate()d. That matters most for activator_pending_requests: it is the only interactive demand that registers while an app is at zero or mid-drain, and it arrives as a short burst at the very end of a multi-hour training window, where a one-step-ahead forecast would barely move.

With two_stream=true the queue terms are removed from demand and become a separate deferrable stream: arrivals per step from rate(pfy_executor_queue_all_total) × step_seconds and an initial backlog from pfy_executor_queue_waiting.

6. MILP optimization

demand[], upper[] ──► OR-Tools SCIP ──► replicas per step

optimizer/opt.py::autoscale_milp. Variables per step t over horizon T:

• x[t] integer replicas in [min_replicas, max_replicas]
• up[t] in [0, max_scale_up], down[t] in [0, max_scale_down] (integer)
• soft[t], hard[t] ≥ 0: shortfall split into a cheap soft tier (degraded / deferrable work) and an expensive hard tier (unserved interactive demand)
• no_rep[t] binary, 1 when x[t] = 0
• two-stream mode only: backlog[t], qserved[t] ≥ 0

Objective, with w(t) = time_discount^t (1 when time_discount = 1):

minimize  Σ_t  replica_cost × x[t]
            + w(t) × sla_soft × soft[t]
            + w(t) × sla_hard × ( hard[t] + no_rep[t] × [demand[t] > 0] )
            + startup_cost × up[t] + shutdown_cost × down[t]
            + carry_cost × backlog[t]                    (two-stream mode; carry_cost defaults to sla_soft)

Constraints:

x[0] = current_replicas (clamped into [min, max]),  down[0] = 0
capacity × x[t] − qserved[t] + soft[t] + hard[t] ≥ coverage[t]      coverage = upper band if risk_aware, else demand
soft[t] ≤ soft_fraction × coverage[t]
x[t] = x[t−1] + Σ up[k] (k activating at t) − down[t]              a scale-up at k activates at k + startup_delay
down[t] ≤ x[t−1]
qserved[t] ≤ capacity × x[t];  backlog[t] = backlog[t−1] + arrivals[t] − qserved[t]   (two-stream)
Σ_t replica_cost × x[t] × step_seconds / 3600 ≤ cost_budget_per_hour                (only if set)
x[T−1] ≥ min(ceil(demand[T−1] / capacity), max_replicas, reachable)                 (terminal condition, if it exceeds min)

• capacity = cpu_alloc × cost_cpu_capacity + (mem_alloc / 1024³) × cost_mem_capacity; replica_cost = cpu_alloc × cost_cpu_mhz_hour + (mem_alloc / 1024³) × cost_mem_gb_hour, from the allocation metrics (or, at current_replicas = 0, the last-known / fallback allocation described under Metrics collection). cpu_alloc is MHz and mem_alloc is bytes, while both memory coefficients are per GB, so Costs converts at the single boundary where they are applied. See Cost coefficients.
• The solver is SCIP via OR-Tools with a time limit of milp_time_limit_ms (default 1000); a FEASIBLE solution is accepted, and the previous solution is passed as a warm-start hint.
• max_replicas = 0 in the request means "no ceiling"; the engine substitutes a bound the per-step ramp cannot reach.
• Because x[0] is pinned and every other constraint is guarded by t >= 1, a plan of T == 1 has no free variable at all. The optimizer therefore plans T = milp_startup_delay + ceil(cache_size / step_seconds) steps and returns plan[milp_startup_delay:]; the offset is milp_startup_delay because capacity from a scale-up decided now first exists that many steps later.

Every engine's output, including MILP's, then passes the safety shell: per-step changes are clamped to ±max_scale_up / max_scale_down relative to the previous value, values are clamped into [min, max], and empty, non-finite or too-short output triggers a fresh MILP solve (counted in pfy_ml_engine_fallback_total{engine}).

7. Response

{
  "desired": [3, 5, 7],
  "step_seconds": 15,
  "cost_breakdown": {
    "replica_cost": 25.5,
    "expected_shortfall": 630.0,
    "backlog_carry_cost": 0.0,
    "scale_event_cost": 2.0
  },
  "engine": "milp",
  "fallback": false,
  "drift_score": 0.03,
  "regime": "diurnal"
}

desired has ceil(cache_size / step_seconds) entries. cost_breakdown is present for the MILP-family engines (expected_shortfall is soft + hard shortfall in demand units); the same numbers are exported as pfy_optimizer_*{app} gauges on GET /metrics. fallback is true when the safety shell or the hold-current policy produced the result. drift_score and regime come from the live serving loop and are ignored by the plugin.

Error responses always carry an error field: 401 invalid/unset token, 400 unknown_engine, 422 validation, 413 body over 10 MB, 503 {"error": "metrics_unavailable"} when Prometheus cannot be read, 503 {"error": "invalid_capacity"} when the derived capacity_per_replica is not a positive finite number, 500 {"error": "internal error"} otherwise (details stay in the server log).

8. Cache and scaling execution

The plugin stores desired with a timestamp and step_seconds (capped at 3600). On each evaluation:

• if elapsed < len(desired) × step_seconds, serve desired[elapsed / step_seconds];
• otherwise request fresh values, and on failure fall back (below);
• clamp the result into [min, max] and return it with the scale direction.

The agent then applies the count through the Nomad target plugin, subject to the job's own scaling { min max } and cooldown.

Failure modes

Situation	Behaviour
Optimizer unreachable, timeout, or 5xx	Plugin retries once (2 s per attempt, 200 ms backoff), then serves the cache if it still covers the current moment, else applies the fallback policy.
Optimizer returns 4xx (bad token, validation)	No retry; same fallback as above. The cause is in the agent log.
Optimizer returns an empty desired	Treated like a failure: cache, else fallback policy.
Prometheus unreachable or a series is empty (an empty allocation series at current_replicas = 0 is not an error; see Metrics collection)	Optimizer returns 503 {"error": "metrics_unavailable"}; plugin falls back.
Derived capacity_per_replica is zero, negative or non-finite	Optimizer returns 503 {"error": "invalid_capacity"} before any engine runs; plugin falls back.
MILP solve fails	Optimizer applies the fallback policy itself and returns "fallback": true.
Engine output anomalous (empty, NaN, too short)	Safety shell re-solves with MILP; fallback: true.
Cache time-exhausted during an outage	Fallback policy, not the stale last value.

The fallback policy (optimizer/optimizer/fallback.py, mirrored in nomadscaler/plugin/web.go): if demand is present and no replica is running, 1; otherwise hold the current count; then clamp into [min, max]. It never forces a floor of one, so it does not itself override a min = 0 bound. It is a failure path only; on the normal path the MILP decides. See Known Issues.

See also

• Quick Start: run both components
• Configuration: every key referenced above
• Nomadscaler Plugin: cache and fallback details
• Optimizer Service: API reference and engines
• Scaling Policies: policy examples