Autoscaler Architecture

The Productify Autoscaler uses a two-tier architecture combining Time-Series forecasting with mathematical optimization for intelligent scaling decisions.

System Overview

┌────────────────────────────────────────────────────────┐
│              Nomad Autoscaler Framework                │
│                                                        │
│  ┌──────────────────────────────────────────────────┐  │
│  │      APM Plugin (Metrics Source)                 │  │
│  │  - Collects application metrics                  │  │
│  │  - CPU, Memory, Request count                    │  │
│  └──────────────────────────────────────────────────┘  │
│                       │                                │
│                       ▼                                │
│  ┌──────────────────────────────────────────────────┐  │
│  │  Strategy Plugin (Productify Nomadscaler)        │  │
│  │  - Evaluates scaling policy checks               │  │
│  │  - Calls optimizer for predictions               │  │
│  │  - Manages second-level cache                    │  │
│  │  - Computes the desired replica count            │  │
│  └──────────────────────────────────────────────────┘  │
│                       │                                │
│                       ▼                                │
│  ┌──────────────────────────────────────────────────┐  │
│  │      Target Plugin (Nomad)                       │  │
│  │  - Updates Nomad job counts                      │  │
│  └──────────────────────────────────────────────────┘  │
└────────────────────────────────────────────────────────┘
                       │
                       │ HTTP API
                       ▼
┌────────────────────────────────────────────────────────┐
│           Optimizer Service (Python)                   │
│                                                        │
│  ┌──────────────────────────────────────────────────┐  │
│  │      Time-Series Forecasting                     │  │
│  │  - SARIMAX model                                 │  │
│  │  - Historical data analysis                      │  │
│  │  - Future demand prediction                      │  │
│  └──────────────────────────────────────────────────┘  │
│                       │                                │
│                       ▼                                │
│  ┌──────────────────────────────────────────────────┐  │
│  │      MILP Optimization                           │  │
│  │  - Constraint-based optimization                 │  │
│  │  - Cost vs. performance balance                  │  │
│  └──────────────────────────────────────────────────┘  │
│                       │                                │
│                       ▼                                │
│  ┌──────────────────────────────────────────────────┐  │
│  │      Prediction Cache Generation                 │  │
│  │  - N seconds of predictions                      │  │
│  │  - One value per second                          │  │
│  └──────────────────────────────────────────────────┘  │
└────────────────────────────────────────────────────────┘

Components

Nomadscaler Plugin (Go)

Role: Nomad autoscaler strategy plugin

Responsibilities:

• Receive scaling policy check evaluations from the Nomad autoscaler
• Request predictions from the optimizer service
• Manage second-level prediction cache
• Select appropriate cached value based on elapsed time
• Return the desired replica count (applied to the job by the Nomad target plugin)
• Handle optimizer failures gracefully

Key Features:

• Second-level time-based cache
• Automatic cache refresh strategy
• Fallback to cached values on optimizer failure
• Constraint enforcement (min/max replicas)

Optimizer Service (Python)

Role: Time-Series forecasting and optimization engine

Responsibilities:

• Fetch historical metrics from Prometheus (application metrics and Nomad allocation metrics)
• Train SARIMAX time-series model
• Forecast demand for next N seconds
• Calculate optimal replica counts using MILP
• Provide HTTP API for predictions

Key Features:

• SARIMAX time-series forecasting
• Mixed-Integer Linear Programming
• Configurable prediction horizon
• Multi-objective optimization
• Token-based authentication

Data Flow

1. Metrics Collection

Application / Proxy / Manager → Prometheus

Metrics collected (queried by the optimizer, pfy_ prefix):

• Request rate (pfy_total_requests_total)
• Average response time (pfy_response_time_seconds histogram)
• Queue counters and backlog (queue_all, queue_processed, queue_waiting, queue_process_time_seconds)
• Users awaiting authentication (pfy_authentication_awating_users)
• Nomad allocation CPU/memory (nomad_client_allocs_*, used for per-replica capacity and cost)

2. Policy Evaluation

Nomad Autoscaler → Check Policy → Evaluate Strategy

Policy defines:

• Target metric
• Evaluation interval
• Min/max replicas
• Strategy plugin to use

3. Prediction Request

Nomadscaler Plugin → HTTP POST /optimize → Optimizer Service

Request payload:

{
  "token": "SECRET",
  "check": {
    "metric_app_name": "my-app"
  },
  "current_replicas": 3,
  "min_replicas": 1,
  "max_replicas": 10,
  "cache_size": 10
}

4. Time-Series Forecasting

Optimizer → Fetch Metrics → Train SARIMAX → Forecast Demand

SARIMAX model:

• Seasonal
• AutoRegressive
• Integrated
• Moving Average with eXogenous variables

The optimizer automatically selects the best SARIMAX order from multiple candidates:

• Candidate orders: (1,1,1), (1,0,1), (2,1,1), (1,1,0)
• Selection criteria: Minimum AIC (Akaike Information Criterion)
• Exogenous variables: avg_response_time, authentication_awating_users, queue_waiting, avg_processing_time

5. MILP Optimization

Forecasted Demand → MILP Solver → Optimal Replica Counts

Optimization objective:

Minimize: Σ (replica_cost × replicas[t] +
              penalty × sla_shortfall[t] +
              startup_cost × scale_up[t] +
              shutdown_cost × scale_down[t] +
              extra_penalty × no_replicas[t] (when demand[t] > 0))

Subject to:
  min_replicas ≤ replicas[t] ≤ max_replicas     ∀t
  capacity × replicas[t] + sla[t] ≥ demand[t]   ∀t
  scale_down[t] ≤ replicas[t-1]                  ∀t
  scale_up[t] ≤ max_scale_up                     ∀t
  scale_down[t] ≤ max_scale_down                 ∀t
  replicas[t] ∈ Integer                          ∀t

6. Response Generation

Optimizer → Generate N values → Return to Plugin

Response:

{
  "desired": [3, 3, 4, 4, 5, 5, 6, 6, 7, 7]
}

Array contains replica counts for next 10 seconds.

7. Cache Management

Plugin → Cache Response → Serve Values → Refresh

Cache behavior:

• Store all N values with timestamp
• Select the value at index seconds_elapsed (capped at the last cached value)
• After 2 cache uses, request fresh predictions
• On optimizer failure, continue using cache

8. Scaling Execution

Plugin → Update Nomad Job → Nomad Schedules Tasks

Nomad actions:

• Calculate delta (desired - current)
• Schedule new allocations (scale up)
• Stop allocations (scale down)
• Wait for health checks
• Update service catalog

Failure Modes & Handling

Optimizer Unavailable

Behavior: Plugin uses cached predictions

Impact: Continues scaling with last known good predictions

Recovery: Automatic reconnection on next refresh attempt

Optimization Failure

Behavior: If the MILP solve fails, the optimizer returns the current replica count (or 1 when there is demand and zero replicas are running)

Impact: Scaling holds steady instead of making an uninformed change

Network Partition

Behavior: Plugin keeps the last cache_size seconds of predictions cached; once exhausted, the last cached value is reused

Impact: Scaling continues on increasingly stale data until connectivity returns

Recovery: Resume normal operation when network restored

Scaling Patterns

Proactive Scale-Up

09:25 - Forecast predicts 09:30 traffic increase
09:25 - Start adding replicas
09:30 - Fully scaled when traffic arrives

Gradual Scale-Down

23:00 - Forecast predicts decreasing demand
23:00 - Begin removing replicas one by one
00:00 - Minimal replicas for overnight

Event-Driven

11:50 - Scheduled event at 12:00
11:50 - Rapidly scale to predicted capacity
12:00 - Ready for event traffic
12:30 - Gradual scale down

See Also

• Quick Start - Get started quickly
• Nomadscaler Plugin - Plugin configuration
• Optimizer Service - Optimizer setup
• Scaling Policies - Policy examples