Configuration

Reference for every setting of the two autoscaler components: the productify-scaler strategy plugin (configured in HCL, inside Nomad scaling policies and the autoscaler agent config) and the optimizer service (configured in config.ini).

Nomadscaler plugin

Where settings live

The plugin reads settings from these places, in this order:

1. the strategy "productify-scaler" { ... } block inside a policy check (per check);
2. the config = { ... } map of the strategy "productify-scaler" block in the autoscaler agent config (shared by every check);
3. for optimizer_url and optimizer_token only: the OPTIMIZER_URL and OPTIMIZER_TOKEN environment variables of the agent process.

Per-check policy:

scaling {
  enabled = true
  min     = 1
  max     = 10

  policy {
    evaluation_interval = "3s"   # how often the check runs
    cooldown            = "10s"  # minimum time between scaling actions

    check "productify-scale-check" {
      source = "nomad-apm"
      query  = "avg_cpu-allocated"

      strategy "productify-scaler" {
        min             = 1
        max             = 10
        metric_app_name = "my-app"  # the `app` label on the pfy_* series
        group           = "demo"    # recommended: the Nomad task group this scaling block lives in
        task            = "webapp"  # recommended: the Nomad task inside it, for the alloc queries
        cache_size      = 30        # optional, default 10; above step_seconds gives a multi-step ramp
        # optimizer_url / optimizer_token may be set here instead of in the agent config
      }
    }
  }
}

Agent-level plugin config:

strategy "productify-scaler" {
  driver = "productify-scaler"

  config = {
    optimizer_token = "supersecrettoken"
    optimizer_url   = "http://optimizer:8015"
    # min          = "1"
    # max          = "10"
    # group        = "demo"
    # metrics_addr = "0.0.0.0:9101"
  }
}

Parameters

Key	Where	Required	Default	Description
optimizer_url	check, agent, OPTIMIZER_URL env	yes	none	Base URL of the optimizer; the plugin posts to <url>/optimize (a trailing slash is stripped)
optimizer_token	check, agent, OPTIMIZER_TOKEN env	yes	none	Sent as Authorization: Bearer <token>. The plugin also sends it as the deprecated body token field
metric_app_name	check	yes	none	Must match the app label of the pfy_* metrics (or an app_aliases entry). For anything pfy nomad render emits this is the application UUID, not a Nomad identifier; the nomad_client_allocs_* queries use task instead
task	check, agent	no (recommended)	empty; the optimizer falls back to metric_app_name	The Nomad task name, used as the task label of the nomad_client_allocs_* queries that measure per-replica capacity. Read from the check's strategy config, then the plugin's global config; there is no SDK last resort, because a scaling policy attaches to a task group that may hold several tasks. Sent on the wire only when set, so the optimizer owns the fallback. Unset logs no `task` configured for check once per check. The fallback selects a real series only when the task is named after the app label, which no rendered job is, so a policy without task measures nothing and is sized from fallback_cpu_alloc / fallback_mem_alloc at zero replicas or answers 503 above zero
group	check, agent	no (recommended)	the check block's own group attribute, else empty	The Nomad task group the scaling block lives in. Sent to the optimizer as group, which uses it as the task_group label of the nomad_client_allocs_* queries. Empty is allowed: the optimizer then matches allocations by task name only and the plugin warns no group configured for check once per check
min	check, agent	no	1	Lower bound for the returned count. Must be >= 0 and, when max > 0, <= max; otherwise the evaluation fails with an error. 0 permits zero replicas here, but does not by itself give a working scale-to-zero cycle; see Known Issues
max	check, agent	no	10	Upper bound for the returned count. 0 means no ceiling (only min is enforced, matching the optimizer's max_replicas); a negative value fails the evaluation
cache_size	check	no	10	Seconds of decisions to request per optimizer call; values above 600 are clamped to 600, invalid values fall back to the default. The default is below the default step_seconds (15), so the response is a single value and the count moves at most one scale step per call; see Request parameters
metrics_addr	agent only	no	unset	Address for the plugin's own Prometheus /metrics listener, e.g. "0.0.0.0:9101". Unset disables it. Changing it requires a plugin restart

Unknown keys in a check's strategy block are logged at warning level (once per check and key) and ignored. Check the agent log for unknown key in check strategy config after a typo.

The strategy's min/max and the job's scaling { min max } block are independent. The plugin clamps its decision into its own bounds; Nomad then enforces the scaling block's bounds. Keep them equal.

Fixed behaviour (not configurable): 2 attempts per optimizer request, 2-second timeout each, 200 ms backoff; step_seconds from the optimizer is capped at 3600; a response carrying more desired values than cache_size is truncated to cache_size (warned once per check); a per-check cache entry untouched for 24 hours is evicted.

Optimizer service

config.ini

All keys live in a single [main] section. The committed optimizer/config.ini is a development config; a minimal production config looks like this:

[main]
loglevel=info
api_loglevel=info
only_test_data=false
enable_test_metrics=false
prometheus_url=http://prometheus:9090
token=CHANGE_ME

Loading

The optimizer reads the first file that exists, in this order:

1. the path in the CONFIG_PATH environment variable, if set;
2. optimizer/optimizer/config.ini (next to the package code);
3. optimizer/config.ini (the project root, where the committed file lives).

If none exists the process exits with FileNotFoundError: config.ini not found. The file must contain a [main] section.

export CONFIG_PATH=/etc/productify/optimizer.ini
poetry run web

Startup validation

The service refuses to start (RuntimeError) when:

• engine names an engine that is not registered;
• only_test_data=false and prometheus_url is missing or not an http(s):// URL.

It logs a warning, but starts, when only_test_data=true, when enable_test_metrics=true, when token is empty (every request will get 401), or when token is one of the committed development tokens (supersecrettoken, dev-only-local-token).

Service keys

Key	Type	Default	Description
token	string	unset	Bearer token clients must send. Unset or empty denies all requests
host	string	0.0.0.0	Listen address for poetry run web
port	int	8015	Listen port for poetry run web
loglevel	string	info	Log level for the optimizer's own loggers (debug, info, warning, error)
api_loglevel	string	info	Log level passed to uvicorn
only_test_data	bool	false	Serve canned fixture metrics instead of querying Prometheus. Development only
enable_test_metrics	bool	false	Start a fake pfy_* metrics generator on metrics_port. It fabricates pfy_total_requests; never enable it against a real Prometheus
metrics_port	int	8017	Port of the fake metrics generator (only with enable_test_metrics=true)
prometheus_url	string	unset	Prometheus base URL; required unless only_test_data=true
prometheus_timeout	float	10	HTTP timeout in seconds for Prometheus queries

Docker image

The image starts uvicorn optimizer.web:app --host 0.0.0.0 --port $PORT directly, so inside the container host and port are ignored and the listen port comes from the PORT environment variable (default 8015). metrics_port is still read from config.ini.

Metrics and forecasting keys

Key	Type	Default	Description
scrape_interval	int	15	Prometheus scrape interval in seconds. Sets the rate() window (2 × scrape_interval) and the floor of step_seconds
training_window_minutes	int	240	How much history to load per request. step_seconds = max(scrape_interval, ceil(window_seconds / 10000))
app_aliases	string	empty	Comma-separated a=b pairs; a metric_app_name equal to either side matches series labelled with either value, in every query. Both metric producers (the proxy and the Manager's queue executor) stamp the application UUID into the app label, so this bridges nothing by default. It is an escape hatch for a hand-written job whose metric_app_name is not the app UUID, which otherwise matches no series and raises no error. Leave it empty for CLI-rendered jobs
request_rate_metric	string	pfy_total_requests	Series name that identifies a request-rate check. A request's metrics are ingested as request-rate observations only when its check.query contains this name
fallback_cpu_alloc	float	100	Allocated CPU (MHz) assumed for a scaled-to-zero app when no nomad_client_allocs_* series exist and no allocation has been seen yet for the (group, task); set it to the task's resources { cpu }. Reached only at current_replicas = 0; with replicas running an empty result is a 503 instead
fallback_mem_alloc	float	16777216	Same for memory, in bytes (default 16 MiB); set it to the task's resources { memory } in bytes
sarimax_order	string	1,1,1	SARIMAX p,d,q. Malformed values fall back to 1,1,1. Choose it offline with optimizer/analysis/order_selection.py
risk_aware	bool	true	Provision capacity against the forecast's P90 upper band instead of the mean
two_stream	bool	false	Model queue work as a separate deferrable backlog stream (see Architecture)
engine	string	milp	Default decision engine: milp, milp_hybrid, meta, or dl1 / dl2 when their optional ML dependencies are installed. A request's engine field overrides it
meta_policy	string	empty	Routing policy for engine=meta, as comma-separated regime=engine pairs over the regimes idle, diurnal and burst, e.g. burst=milp_hybrid,diurnal=milp,idle=milp. Regimes not named stay on milp, so an empty value makes meta behave exactly like milp; the selector logs the regime and the engine it resolved to on every decision. An unknown regime, an unknown engine or regime=meta is logged as an error and the whole key is ignored
hybrid_demand_scale	float	1.0	Learned demand-shape correction for engine=milp_hybrid: the demand and its upper band are multiplied by hybrid_demand_scale × hybrid_per_item_work before the solve. Derive it offline from a recorded realized/forecast pair (optimizer/analysis/training_runbook.md); it is not fitted online
hybrid_churn_cost_scale	float	1.0	Multiplier on startup_cost and shutdown_cost for engine=milp_hybrid
hybrid_per_item_work	float	1.0	Second demand multiplier for engine=milp_hybrid. With all three hybrid_* keys at 1.0 the engine decides identically to milp and warns so at construction

Cost coefficients

Keys prefixed cost_. Unparsable values fall back to the default.

Key	Default	Description
cost_cpu_mhz_hour	0.001	Price per allocated CPU MHz per hour
cost_mem_gb_hour	0.00001	Price per allocated GB of memory per hour. nomad_client_allocs_memory_allocated is in bytes; the allocation is divided by 1024³ before this coefficient is applied
cost_scale_event	0.5	Cost per scale-up / scale-down decision; the default for milp_startup_cost and milp_shutdown_cost when those keys are unset
cost_sla_soft	1.0	Cost per unit of soft shortfall (degraded or deferrable work)
cost_sla_hard	10.0	Cost per unit of hard shortfall (unserved interactive demand); also applied when zero replicas face positive demand
cost_cpu_capacity	0.5	Capacity per allocated CPU MHz
cost_mem_capacity	0.0001	Capacity per allocated GB of memory, with the same byte-to-GB conversion
cost_budget_per_hour	empty	Optional hard cap on hourly replica spend per solve. Empty = no cap

capacity_per_replica = cpu × cost_cpu_capacity + (mem / 1024³) × cost_mem_capacity and replica_cost = cpu × cost_cpu_mhz_hour + (mem / 1024³) × cost_mem_gb_hour, where cpu (MHz) and mem (bytes) come from the Nomad allocation metrics. Both conversions happen inside Costs, where the per-GB coefficients are applied, so callers pass the raw metric through unmodified.

replica_cost and milp_shutdown_cost together set the scale-down threshold. Releasing a replica costs milp_shutdown_cost once and saves replica_cost on every step of the actionable horizon, so the count only falls when the saving over that horizon reaches the shutdown cost. At the shipped values (cost_cpu_mhz_hour = 0.001, milp_shutdown_cost = 0.3) and one actionable step that is 300 MHz of allocated CPU; below it an idle app holds its count until cache_size covers enough steps. See min and max.

The coefficient values are uncalibrated

The four cost_* resource coefficients are inherited placeholders, not derived prices (optimizer/docs/cost-calibration.md shows how to derive them from an instance price). At the shipped defaults the memory terms are negligible: at a 500 MHz / 512 MB allocation memory is 0.00002% of capacity_per_replica and 0.001% of replica_cost, so the model is effectively CPU-only. Whether memory should be modelled at all, and what cost_cpu_mhz_hour and milp_shutdown_cost should be relative to each other, are pricing decisions for the operator.

MILP parameters

Keys prefixed milp_. Each key can be overridden per request through the overrides object of /optimize, using the name without the milp_ prefix. Resolution: request override, then config key, then built-in default. Overrides are typed and bounded (last column); an out-of-range value or an unknown key is rejected with 422.

Key	Default	Override bounds	Description
milp_startup_cost	cost_scale_event (0.5)	0 to 1e6	Cost per replica added
milp_shutdown_cost	cost_scale_event (0.5; the committed config.ini sets 0.3)	0 to 1e6	Cost per replica removed. Paid once against the replica_cost saved over the actionable horizon, so it sets the scale-down threshold; see Cost coefficients
milp_max_scale_up	2	1 to 1000	Max replicas added per step (also the safety-shell clamp)
milp_max_scale_down	2	0 to 1000	Max replicas removed per step (also the safety-shell clamp)
milp_soft_fraction	1.0	0 to 1	Fraction of a step's demand that may be soft-costed shortfall
milp_startup_delay	1	0 to 3600 (0 behaves as 1)	Steps between a scale-up decision and the capacity becoming available (minimum 1). Also the number of leading plan steps dropped from the response
milp_time_discount	1.0	(0, 1]	Weight time_discount^t on later steps' shortfall; 1.0 disables discounting
milp_time_limit_ms	1000	1 to 60000	Solver time limit per solve
milp_terminal_condition	true	config only	Force the last step to cover the last step's demand (bounded by what is reachable). Not overridable per request

{ "overrides": { "max_scale_up": 5, "startup_cost": 0.2 } }

Recording and shadow mode

These keys are read on every /optimize call.

Key	Default	Description
trace_recording	false	Append the metrics window, demand and decision of each request to a CSV for offline benchmarking
trace_path	trace.csv	Trace file path
trace_max_rows	10000	Row cap for the trace, shadow and decision-record files
decision_record_recording	false	Log fully-labelled decision records (context, engine, schedule, realized demand) for training
decision_record_path	decisions.parquet	Decision-record path (.csv when no parquet engine is installed)
shadow_mode	false	Also run the engines in shadow_engines on each request and log their counterfactual cost; the configured engine's decision is still the one applied
shadow_engines	empty	Comma-separated engine names to shadow
shadow_log_path	shadow.csv	Shadow log path

Live serving loop

These seven keys are read once at startup, so changing one needs a service restart; only a model promotion or rollback applies without one. They drive model resolution, drift monitoring and canary rollout in the live /optimize path; see Live serving loop. Their defaults keep the loop idle (no candidate, so no canary is built). Retraining is an offline runbook step (optimizer/analysis/training_runbook.md); the live service never triggers it. The live forecaster's own refit thresholds (300 s, 50% drift) are fixed constants in demand.py, not configurable.

Key	Default	Description
model_registry_root	models	Filesystem root learned engines (dl1/dl2) resolve their production version from. Created if missing
drift_threshold	0.2	PSI drift-score threshold on the trailing 96 points of the observed demand window (exported as pfy_ml_drift_score); also the canary's drift-based rollback threshold
drift_rolling_error_threshold	1e9	Rolling forecast-error threshold (exported as pfy_ml_rolling_error)
canary_candidate	empty	Learned engine (dl1 or dl2) to canary. Empty disables the canary; milp is always the incumbent
canary_fraction	0.1	Fraction of a deployment's decisions routed to the candidate
canary_cost_regression_tol	0.1	Rollback threshold: candidate's mean realized cost exceeding the incumbent's by more than this fraction. Requires decision_record_recording=true, because realized cost reaches the canary only via the decision-record backfill
canary_min_samples	5	Realized-cost samples required on both arms before the cost comparison is made at all. A drift breach is never gated on it

Request parameters of /optimize

Field	Type	Default	Notes
check.name	string	required	The check's label; free-form, 1 to 256 printable characters (no control characters). Only logged
check.group	string	""	The Nomad task group; must match ^[A-Za-z0-9_.:-]*$. May be empty; the allocation query then selects by task name only and the optimizer warns once per task
check.metric_app_name	string	required	The app label of the pfy_* series; must match ^[A-Za-z0-9_.:-]+$
check.task	string	""	The Nomad task, used as the task label of the allocation query; must match ^[A-Za-z0-9_.:-]*$. Empty falls back to metric_app_name and warns once per app
check.source	string	required	The APM plugin name; informational
check.query	string	""	The check's APM query line (up to 4096 characters); decides whether metrics are request-rate samples (below)
current_replicas	int ≥ 0	0	Current count; the MILP starts from it
min_replicas	int ≥ 0	0	Lower bound
max_replicas	int ≥ 0	10	Upper bound; 0 means no ceiling. Must be ≥ min_replicas when > 0
metrics	list of {timestamp, value}	[]	Up to 10000 entries; appended to the request-rate series only when check.query contains request_rate_metric, otherwise ignored
cache_size	int, 1 to 3600	10	Seconds the response covers; it carries ceil(cache_size / step_seconds) values. The MILP plans milp_startup_delay + ceil(cache_size / step_seconds) steps and returns the actionable tail
overrides	object	{}	MILP parameter overrides, see above; typed and bounded, unknown keys are a 422
engine	string	""	Engine name; empty uses the config default
token	string	""	Deprecated; use the Authorization: Bearer header

Larger cache_size means fewer optimizer calls and steadier counts; smaller means quicker reaction to new metrics. The plugin caps it at 600.

The MILP pins the first step of every plan to the count the job is already running (x[0] = clamp(current_replicas, min, max)), and capacity from a scale-up decided now first exists milp_startup_delay steps later. The plan is therefore milp_startup_delay + ceil(cache_size / step_seconds) steps long and the response is plan[milp_startup_delay:], so the values the plugin receives are decisions rather than its own input echoed back.

A cache_size at or below step_seconds still costs two things: one actionable step per call, so the count moves by at most milp_max_scale_up / milp_max_scale_down (2 each) between calls; and no visibility into the plan beyond the next step. Raising it to 2 × step_seconds or more buys a smoother ramp and, for scale-down, more steps over which a released replica's saving can accumulate.

Development mode

[main]
only_test_data=true       # canned metrics, no Prometheus needed
enable_test_metrics=true  # fake pfy_* generator on metrics_port for a local Prometheus to scrape
token=supersecrettoken

docker-compose.test.yml at the repository root starts the optimizer with this config plus a Prometheus (config/prometheus.test.yml) that scrapes the fake generator on port 8017.

See also

• Nomadscaler Plugin
• Optimizer Service
• Scaling Policies
• Architecture