Optimizer Service

The optimizer is a Python HTTP service (autoscaler/optimizer/) that answers one question: how many replicas should this application run over the next few steps? It forecasts the request rate from Prometheus metrics with a SARIMAX model and solves a mixed-integer linear program (MILP) that trades replica cost against the cost of unmet demand. The Nomadscaler plugin calls it; you can also call it directly for testing.

Requirements: Python 3.12+ and Poetry (or Docker).

Run it

cd autoscaler/optimizer
poetry install

poetry run testcalc   # solve the bundled demand fixture and print the replica schedule
poetry run web        # start the HTTP service on host/port from config.ini (default 0.0.0.0:8015)
poetry run test       # run the test suite (pytest)

Docker:

cd autoscaler/optimizer
docker build -t productify/optimizer:local .
docker run --rm -p 8015:8015 productify/optimizer:local

The image bakes in the development config.ini (only_test_data=true, enable_test_metrics=true, token=supersecrettoken) so it starts without a mount. It runs uvicorn optimizer.web:app --host 0.0.0.0 --port $PORT (PORT defaults to 8015). For production mount your own config and set CONFIG_PATH; see Autoscaler Deployment.

Published images all live in the ghcr.io/productifyfw/autoscaler/optimizer package. Tagged releases push :<version> and :latest (current release 1.2.0, linux/amd64); CI pushes from main add :<sha> (slim) and :latest-ml / :<sha>-ml (with the optional ML dependencies; see Live serving loop). The package is private: an anonymous pull gets a 401 until it is made public (Known Issues), so log in first:

echo "$GITHUB_TOKEN" | docker login ghcr.io -u <github-username> --password-stdin   # token with read:packages
docker pull ghcr.io/productifyfw/autoscaler/optimizer:1.2.0

Configuration

The service reads config.ini ([main] section) from CONFIG_PATH, else optimizer/optimizer/config.ini, else optimizer/config.ini. A minimal production file:

[main]
loglevel=info
api_loglevel=info
only_test_data=false
enable_test_metrics=false
prometheus_url=http://prometheus:9090
token=CHANGE_ME

• token: required for any request to succeed; an empty or missing token denies everything with 401. Keep it out of version control.
• prometheus_url: required unless only_test_data=true; validated at startup.
• loglevel / api_loglevel: optimizer and uvicorn log levels.

Every key, with defaults, is in Configuration.

API

POST /optimize

Returns the desired replica count per step.

Authentication: Authorization: Bearer <token>. A token field in the body is accepted but deprecated. A wrong token, or a server with no token configured, gives 401 {"error": "Invalid token", "detail": "Invalid token"}.

Request:

POST /optimize
Authorization: Bearer SUPER_SECRET_TOKEN
Content-Type: application/json

{
  "check": {
    "name": "productify-scale-check",
    "source": "nomad-apm",
    "group": "demo",
    "metric_app_name": "my-app",
    "task": "webapp"
  },
  "current_replicas": 3,
  "min_replicas": 1,
  "max_replicas": 10,
  "metrics": [{ "timestamp": "2026-01-01T12:00:00Z", "value": 42.0 }],
  "cache_size": 45,
  "overrides": { "max_scale_up": 5 },
  "engine": "milp"
}

Field	Notes
check.name	the check's label; free-form, 1 to 256 printable characters (no control characters). Only logged, never interpolated
check.group	Nomad task group; used as the task_group label when reading allocated CPU/memory. Must match ^[A-Za-z0-9_.:-]*$ and may be empty (the plugin sends "" for a check without group): the allocation query then matches on task alone and the optimizer logs a warning once per task
check.metric_app_name	app label of the pfy_* series (or an app_aliases entry); must match ^[A-Za-z0-9_.:-]+$. Not a Nomad identifier; for anything pfy renders it is the application UUID
check.task	optional Nomad task name, used as the task label of the nomad_client_allocs_* queries; must match ^[A-Za-z0-9_.:-]*$ and may be empty (the plugin omits it for a check without task), in which case it falls back to metric_app_name and the optimizer warns once per app
check.query	optional, the check's APM query line (up to 4096 characters). Only used to decide whether metrics are request-rate samples
current_replicas, min_replicas, max_replicas	integers ≥ 0; max_replicas = 0 means no ceiling; min_replicas must not exceed a positive max_replicas
metrics	optional, up to 10000 {timestamp, value} pairs. Appended to the request-rate series as the newest observations only when check.query contains the configured request_rate_metric (default pfy_total_requests); otherwise ignored. The plugin sends the check's query verbatim
cache_size	optional, default 10, range 1 to 3600. Seconds the response covers; it carries ceil(cache_size / step_seconds) values. The MILP plans milp_startup_delay + ceil(cache_size / step_seconds) steps and returns the actionable tail
overrides	optional MILP parameter overrides (startup_cost, shutdown_cost, max_scale_up, max_scale_down, soft_fraction, startup_delay, time_discount, time_limit_ms). Typed and bounded; see the bounds in Configuration. An out-of-range value or an unknown key is a 422
engine	optional engine name; empty uses the engine config key (default milp)

Response (for cache_size = 45 with the default 15-second step):

{
  "desired": [3, 5, 7],
  "step_seconds": 15,
  "cost_breakdown": {
    "replica_cost": 25.5,
    "expected_shortfall": 630.0,
    "backlog_carry_cost": 0.0,
    "scale_event_cost": 2.0
  },
  "engine": "milp",
  "fallback": false,
  "drift_score": 0.03,
  "regime": "diurnal"
}

• desired: one replica count per step.
• step_seconds: seconds each value covers: max(scrape_interval, ceil(training_window_seconds / 10000)), 15 by default.
• cost_breakdown: the solved objective's parts, for the MILP-family engines: replica_cost (sum of replica cost over the horizon), expected_shortfall (soft + hard shortfall, in demand units), backlog_carry_cost (two-stream mode), scale_event_cost (startup + shutdown costs). Empty for other engines.
• engine: the engine that decided: the request's engine override, else the canary's routed engine (when canary_candidate is active and loadable), else the config default. This is also what the decision record stores.
• fallback: true when the result came from the safety shell's MILP re-solve or from the hold-current fallback policy rather than the engine's own output.
• drift_score / regime: PSI drift score of the current demand window against its reference (a learned model's manifest window, else the app's first training window) and the coarse regime label (idle / diurnal / burst). Always present; ignored by the Nomadscaler plugin. See Live serving loop.

Errors (always a JSON body with an error field):

Status	Body	Cause
400	{"error": "unknown_engine", "detail": "..."}	engine (request or config) is not registered; detail lists the valid names
401	{"error": "Invalid token", ...}	wrong token, or no token configured
413	{"error": "request body too large"}	body over 10 MB
422	FastAPI validation detail	out-of-range fields, min_replicas > max_replicas, bad names, unknown or out-of-range overrides keys
503	{"error": "metrics_unavailable"}	Prometheus query failed, or an allocation query returned no series while replicas are running (at current_replicas = 0 the scale-from-zero fallback applies, see Operations)
503	{"error": "invalid_capacity"}	the derived capacity_per_replica is not a positive finite number (zero allocations or mis-set cost_*_capacity coefficients); no engine runs
500	{"error": "internal error"}	anything else; details are in the server log only

GET /models

Authentication: Authorization: Bearer <token>, same as /optimize.

Registry versions and the loaded production model per learned engine (dl1, dl2):

{
  "dl1": {
    "versions": {
      "1": {"stage": "archived", "feature_spec_hash": "0d323f14b34d07234ac4c4ff54c1535add34c158f715ebc1f208f66ca1865d08"},
      "2": {"stage": "production", "feature_spec_hash": "0d323f14b34d07234ac4c4ff54c1535add34c158f715ebc1f208f66ca1865d08"}
    },
    "loaded_version": "2",
    "load_reason": null
  },
  "dl2": {
    "versions": {},
    "loaded_version": null,
    "load_reason": null
  }
}

Versions are the registry's own sequential ids ("1", "2", …), assigned by Registry.register as max(existing) + 1, and every entry carries the feature_spec_hash the version was registered with (a version whose hash does not match the live serving spec never loads). load_reason is a string when a production version exists but failed to load (missing optional dependency, unpicklable artifact, or a feature-spec mismatch); the engine then serves modelless and the safety shell falls back to MILP. An error key, per engine and at the top level, appears only when that engine's registry index file could not be read. With no usable registry root the whole response is {"error": "no model registry configured"}. See Live serving loop.

GET /serving/{group}/{app}

Authentication: Authorization: Bearer <token>, same as /optimize.

Drift, canary and per-engine model state for one deployment; 404 {"error": "unknown_serving_state"} before its first /optimize request:

{
  "group": "demo",
  "app": "my-app",
  "drift": {
    "drift_score": 0.03,
    "rolling_error": 1.2,
    "regime": "diurnal",
    "regime_code": 1.0,
    "drift": false
  },
  "canary": {
    "candidate": "dl1",
    "incumbent": "milp",
    "fraction": 0.1,
    "candidate_version": "2",
    "candidate_samples": 4,
    "incumbent_samples": 36,
    "rolled_back": false,
    "rollback_reason": null,
    "inactive_reason": null
  },
  "models": {
    "dl1": {"loaded": true, "reason": null},
    "dl2": {"loaded": false, "reason": null}
  }
}

canary is null when canary_candidate is unset. See Live serving loop.

GET /health

Returns {"response": "ok"}. No authentication.

GET /metrics

Prometheus exposition of the service's own metrics, always on, no authentication:

Metric	Labels	Meaning
pfy_optimizer_replica_cost	app	cost_breakdown.replica_cost of the last solve
pfy_optimizer_expected_shortfall	app	cost_breakdown.expected_shortfall of the last solve
pfy_optimizer_backlog_carry_cost	app	cost_breakdown.backlog_carry_cost of the last solve
pfy_optimizer_scale_event_cost	app	cost_breakdown.scale_event_cost of the last solve
pfy_optimizer_alloc_fallback_used_total	group, task	Solves whose per-replica capacity came from fallback_cpu_alloc / fallback_mem_alloc rather than a measurement. Non-zero for a genuinely scaled-to-zero task; also non-zero when group/task select no series that exists, which is the one case where a mis-selected task is not already a 503
pfy_ml_engine_fallback_total	engine	Times an engine's output was anomalous and the safety shell re-solved with MILP (appears after the first fallback)
pfy_ml_drift_score	app	Current PSI drift score of the observed demand window against its reference
pfy_ml_rolling_error	app	Current rolling forecast/decision error
pfy_ml_regime	app, regime	1 for the app's current coarse regime (idle/diurnal/burst), 0 for the others
pfy_ml_canary_rollbacks_total	app, candidate, reason	Canary rollbacks to the incumbent engine (reason is drift or cost_regression)

Test metrics generator

With enable_test_metrics=true the service also starts a fake pfy_* metrics generator (optimizer/testmetrics.py) on metrics_port (default 8017). It fabricates every series the optimizer queries (pfy_total_requests, pfy_response_time_seconds, pfy_authentication_awaiting_users, pfy_activator_pending_requests and the pfy_executor_queue_* family) every 5 seconds so a local Prometheus has something to scrape. The names come from the same constants the queries are built from, so the fixture cannot validate a query no producer answers. It publishes to its own registry, so it cannot mix fabricated load into the optimizer's real pfy_optimizer_* / pfy_ml_* metrics. Never enable it against a production Prometheus: it pollutes the very series the optimizer forecasts.

Forecasting

optimizer/demand.py::create_demand_series.

1. The request-rate series (rate(pfy_total_requests)) from the last training_window_minutes (default 240) is resampled to step_seconds and gap-filled (forward, backward, then zeros).

2. A SARIMAX model with the configured order is fitted:

   model = SARIMAX(
       y,                 # request rate, one row per step
       exog=exog_train,   # Fourier seasonality terms (default strategy)
       order=order,       # from the sarimax_order key, default (1, 1, 1)
       enforce_stationarity=False,
       enforce_invertibility=False,
   )

3. Exogenous inputs (fourier_only, the default): sin/cos terms for daily (86400 s) and weekly (604800 s) periods, 2 harmonics each. They are deterministic, so the forecast never depends on a future-unknown metric value. The metric columns (avg_response_time, authentication_awating_users, queue_waiting, avg_processing_time) do not enter the model; they feed the demand conversion. (A legacy freeze strategy exists for benchmarking only.)

4. Fitted models are cached in-process per (group, metric_app_name). A cached fit is extended with the new tail of the window (no re-estimation) while it is younger than 300 seconds and the mean of the last 10 observations is within 50 % of the fitted mean; otherwise the model is refit. These thresholds are constants in demand.py.

5. The forecast runs to the plan horizon, milp_startup_delay + ceil(cache_size / step_seconds) steps, not just the ceil(cache_size / step_seconds) steps the response carries; the upper band is the 90th percentile of the forecast interval.

6. If fewer than 8 observations exist, or fitting/forecasting fails or yields non-finite values, a persistence forecast (repeat the last value) is used instead.

Order selection is an offline step: optimizer/analysis/order_selection.py compares candidate (p, d, q) orders by AIC on recorded traces and recommends the value to put in sarimax_order.

Demand conversion

The forecast (and, when risk_aware=true, its upper band) becomes a scalar demand per step. Each term uses its own metric, averaged over its last 3 samples, and a weight from optimizer/weights.py:

demand[t] = forecast_rate[t] * avg_response_time * 0.05   # per-request work
          + queue_waiting * 0.1                            # backlog pressure
          + avg_processing_time * 0.1                      # per-item queue work
          + authentication_awating_users * 0.2
          + activator_pending_requests * 1.0               # requests held through a cold start

The upper band is never allowed below the mean. With two_stream=true the two queue terms are removed here and modelled as a separate backlog stream inside the MILP; the held-request term is never removed, because a request blocked on a cold start is interactive by definition.

activator_pending_requests is weighted 1.0 rather than the auth gauge's 0.2 because it is not a proxy for pressure: it is an exact count of interactive requests currently unserved on live connections. It is also why the request-rate term alone is not enough for a waking app: that term is multiplicative, and the proxy's response-time histogram is only observed when a handler returns, so avg_response_time reads 0, and the whole product with it, for exactly as long as the requests are held. The weight is an operator-tunable product decision; see Known Issues for what the term does and does not buy.

MILP optimization

optimizer/opt.py::autoscale_milp, called through optimizer/engines/milp.py. Solver: OR-Tools with SCIP, time limit milp_time_limit_ms (default 1000 ms), FEASIBLE solutions accepted, previous solution passed as a warm-start hint.

from ortools.linear_solver import pywraplp

solver = pywraplp.Solver.CreateSolver("SCIP")

Variables (per step t, horizon T)

x[t]        # integer replicas, min_replicas..max_replicas
up[t]       # integer scale-up decisions, 0..max_scale_up
down[t]     # integer scale-down decisions, 0..max_scale_down
soft[t]     # continuous >= 0: soft shortfall (degraded / deferrable work)
hard[t]     # continuous >= 0: hard shortfall (unserved interactive demand)
no_rep[t]   # binary: 1 when x[t] == 0
backlog[t], qserved[t]   # continuous >= 0, two-stream mode only

Objective

def w(t):                       # later steps may be discounted
    return time_discount ** t if 0 < time_discount <= 1 else 1.0

total_cost = (
    solver.Sum(replica_cost * x[t] for t in range(T))
    + solver.Sum(w(t) * sla_soft * soft[t] for t in range(T))
    + solver.Sum(w(t) * (sla_hard * hard[t]
                         + sla_hard * no_rep[t] * (1 if demand[t] > 0 else 0))
                 for t in range(T))
    + solver.Sum(startup_cost * up[t] + shutdown_cost * down[t] for t in range(T))
    + solver.Sum(carry * backlog[t] for t in range(T))   # two-stream mode; carry = sla_soft unless set
)
solver.Minimize(total_cost)

Constraints

coverage = demand_upper if risk_aware else demand   # provision against the P90 band by default

solver.Add(x[0] == clamp(current_replicas, min_replicas, max_replicas))
solver.Add(down[0] == 0)

for t in range(T):
    cap = x[t] * capacity_per_replica - (qserved[t] if two_stream else 0)
    solver.Add(cap + soft[t] + hard[t] >= coverage[t])            # meet demand or pay shortfall
    solver.Add(soft[t] <= soft_fraction * coverage[t])            # cap on the cheap tier
    if two_stream:
        solver.Add(qserved[t] <= x[t] * capacity_per_replica)
        solver.Add(backlog[t] == backlog[t-1] + queue_arrivals[t] - qserved[t])
    if t >= 1:
        solver.Add(down[t] <= x[t-1])                              # cannot stop more than you run
        # a scale-up decided at step k adds capacity at k + startup_delay
        solver.Add(x[t] == x[t-1] + sum(up[k] for k in activating_at[t]) - down[t])
    solver.Add(x[t] <= max_replicas * (1 - no_rep[t]))            # link no_rep to x
    solver.Add(x[t] >= 1 - no_rep[t])

if cost_budget_per_hour is not None:                               # optional spend cap
    solver.Add(sum(replica_cost * x[t]) * step_seconds / 3600 <= cost_budget_per_hour)

if terminal_condition and T > 1 and demand[T-1] > 0:               # avoid a spurious end-of-horizon drop
    need = min(ceil(demand[T-1] / capacity_per_replica), max_replicas, reachable)
    if need > min_replicas:
        solver.Add(x[T-1] >= need)

x[0] is constrained to clamp(current_replicas, min_replicas, max_replicas), and every other constraint above is guarded by t >= 1, so a one-step plan has no free variable and can only return the count it was handed. The plan horizon and the response length are therefore separate quantities: T = milp_startup_delay + ceil(cache_size / step_seconds), and the response is plan[milp_startup_delay:], offset by milp_startup_delay because capacity from a scale-up decided now first exists that many steps later (activate_at). At the shipped milp_startup_delay = 1 that is exactly "drop the pinned step". The response is still ceil(cache_size / step_seconds) values long.

capacity_per_replica and replica_cost are derived per request from the task's allocated CPU (MHz) and memory (bytes, converted to GB before the per-GB cost_mem_* coefficients are applied; at current_replicas = 0, from the last-known or fallback allocation, see Operations) and the cost_* coefficients; the other parameters come from the milp_* keys or the request's overrides. Defaults: startup_cost and shutdown_cost both cost_scale_event (0.5; the committed config.ini sets milp_shutdown_cost=0.3), max_scale_up 2, max_scale_down 2, soft_fraction 1.0, startup_delay 1, time_discount 1.0, sla_soft 1.0, sla_hard 10.0. See Configuration.

Decision Engines

The decision step is pluggable (optimizer/engines/). The engine config key, or the request's engine field, selects one:

Engine	What it is	Availability
milp	The MILP above. Default, explainable baseline, and the fallback for every other engine	always
milp_hybrid	The same MILP solve, with learned corrections from the hybrid_demand_scale, hybrid_per_item_work and hybrid_churn_cost_scale config keys applied to its inputs; still emits a cost breakdown. With all three at their default 1.0 it decides identically to milp and logs a warning saying so	always
meta	Classifies the demand window into a regime (idle, diurnal, burst) and routes to the engine the meta_policy config key names for it; regimes not named stay on milp, so with the key empty it behaves exactly like milp. It logs the regime and the engine it resolved to on every decision	always
dl1	Imitation-learned policy (PyTorch)	only with poetry install --with ml
dl2	Offline-RL policy (PyTorch)	only with poetry install --with ml

Every engine's output passes the same safety shell (optimizer/engines/safety.py): per-step changes are clamped to ±max_scale_up / max_scale_down from the previous value, values are clamped into [min_replicas, max_replicas], and empty, non-numeric, non-finite or too-short output triggers a fresh MILP solve, counted in pfy_ml_engine_fallback_total{engine} and reported as "fallback": true. An engine that raises is treated the same way.

shadow_mode=true additionally runs each engine in shadow_engines on every request and logs its counterfactual cost to shadow_log_path, while the configured engine's decision is the one returned.

CRON-trigger pre-scaling is offline

CRON-trigger-aware pre-scaling (optimizer/optimizer/schedule.py) is implemented and unit-tested but not called from /optimize.

Live serving loop

Model resolution, drift monitoring and canary rollout (optimizer/optimizer/serving/) run in the live /optimize path, completing the offline pipeline above: an offline-trained model registered at shadow and promoted to production is picked up by the already-running service, with no restart, because the resolver re-reads the registry's index.json when its file mtime changes. Only dl1/dl2 resolve a model; milp, milp_hybrid and meta are unaffected.

A model must be registered with the live path's exact feature spec (optimizer.serving.live.SERVING_FEATURE_SPEC, a window plus current/min/max replicas); a mismatch, a missing optional dependency, or an unpicklable artifact leaves the engine modelless with the reason recorded in /models / /serving/{group}/{app}, and the safety shell falls back to MILP as usual.

Canary rollout

Setting canary_candidate (dl1 or dl2) routes canary_fraction of a deployment's decisions to it while milp serves the rest as incumbent. After each decision-record backfill, the candidate's realized mean cost is compared against the incumbent's; a breach of canary_cost_regression_tol, or a drift score above drift_threshold, rolls the canary back, archiving the candidate's registry version (so it cannot route again, even after a restart) and incrementing pfy_ml_canary_rollbacks_total.

Realized cost reaches the canary only from the decision-record backfill, which only runs with decision_record_recording=true. With canary_candidate set and recording left at its default false, the canary routes traffic but never rolls back on cost regression. Drift-based rollback works either way, since drift is computed from every request's observed demand window independent of recording.

canary_min_samples (default 5) is the number of realized costs required on both arms before the cost comparison is made at all, so a single unlucky decision on each side cannot archive a promoted model. A drift breach is never gated on it.

Drift monitoring

The PSI score is computed over the observed demand window (the same conversion the decision-record backfill uses), not over the forecast: the forecast horizon is only milp_startup_delay + ceil(cache_size / step_seconds) points (two, at the plugin defaults), too few to carry a distribution. At most the trailing 96 observations are scored, keeping the sample size comparable across deployments; when no observed window is available the previous reading stands. The forecast still drives the rolling-error signal: each request's forecast head is scored against the next request's observed tail.

The reference distribution is the loaded production model's manifest reference_window when a learned model serves, else the first window the deployment served (so a pure-MILP deployment still gets regime gauges). It is re-derived when a model is promoted or changes version, so a model promoted after the service started contributes its own training distribution. Each monitor keeps a bounded ring of its last 100 drift events.

Status and limits

No DL engine is the production default; the recorded decision-gate outcome for the learned engines is NO-GO (optimizer/analysis/gate_report.md). See optimizer/analysis/training_runbook.md in the repository for the offline pipeline, and Configuration for the seven config keys (model_registry_root, drift_threshold, drift_rolling_error_threshold, canary_candidate, canary_fraction, canary_cost_regression_tol, canary_min_samples), all read once at startup, so changing one needs a restart. Only promotion and rollback apply without one.

The default image is slim: without torch/lightgbm/gymnasium a promoted DL model logs once and the request falls back to MILP. The -ml image variants (built with ML_GROUP=1) install the optional [ml] group so a promoted dl1/dl2 model serves; libgomp1 is present in both.

Two optimizer replicas sharing a registry volume each run an independent ModelResolver/CanaryController in-process; canary routing and rollback are not coordinated across instances.

Operations

• Cold start is slow. The first /optimize for an application fits the SARIMAX model; on the bundled test data this takes over a minute, while warm requests take about two seconds. The plugin's 2-second timeout means it falls back until the fit is cached.
• A Prometheus outage returns 503 metrics_unavailable rather than a guess; the plugin holds the current count.
• Scale-from-zero: a scaled-to-zero task has no nomad_client_allocs_* series. With current_replicas = 0 an empty allocation result is not an error: capacity comes from the task's last-known allocation (per (group, task), cached in-process from its most recent successful query), else from fallback_cpu_alloc / fallback_mem_alloc (defaults 100 MHz / 16 MiB; set them to the task's resources stanza). With replicas running, an empty result is 503 metrics_unavailable.
• A wrong task looks different at zero than above it. The same empty result is produced by a policy whose group/task select no series that exists. Above zero replicas that is loud (503, and the plugin falls back to its own heuristic), so the optimization is lost, not silently wrong. At zero replicas it takes the fallback path instead, so a scale-from-zero decision can be sized from the fallback constants without any request failing. The pfy_optimizer_alloc_fallback_used_total counter (labelled group/task) and the accompanying WARNING distinguish the two.
• Logging: loglevel for the optimizer's loggers, api_loglevel for uvicorn. At debug every request logs the forecast, capacity/cost, solve time and the chosen schedule.
• Test suite: poetry run test. An end-to-end read-path test (tests/test_integration.py) runs only when OPTIMIZER_INTEGRATION_URL points at a stack started from docker-compose.test.yml.

See also

• Architecture
• Configuration
• Nomadscaler Plugin