Productify

Appearance

Productify Autoscaler — Nomadscaler ​

Nomad strategy plugin for the Productify autoscaler. This component is implemented in Go and is intended to run as a container or a Nomad job.

Overview ​

The plugin:

  • Interfaces with the Nomad Autoscaler as a strategy plugin
  • Queries the Optimizer service for predictions
  • Implements a second-level caching mechanism
  • Returns desired replica counts, which the Nomad Autoscaler applies to the job

Installation ​

Build from Source ​

bash
cd nomadscaler
./build.sh

The compiled binary will be in ./bin/productify-scaler. The script also builds a productify/scaler:local Docker image — the Nomad Autoscaler with the plugin preinstalled in /plugins.

Run Locally ​

bash
docker run --rm ghcr.io/productifyfw/autoscaler-nomadscaler:latest

Configuration ​

Configuration for nomadscaler is managed via HCL configuration. An example job that renders the config.hcl is available at:

  • nomadscaler/config/autoscaler.hcl

Strategy Configuration Parameters ​

The productify-scaler strategy plugin supports the following configuration parameters:

  • optimizer_token (required): Authentication token for the optimizer API
  • optimizer_url (required): URL of the optimizer service
  • min (required): Minimum number of replicas
  • max (required): Maximum number of replicas
  • metric_app_name (required): Name of the application for metrics
  • cache_size (optional, default: 10): Number of desired replica values to cache from optimizer response

Cache Behavior ​

The plugin uses a second-level time-based cache for accurate scaling decisions:

  • Optimizer predictions: The optimizer forecasts demand at second-level granularity and returns desired replica counts for the next N seconds (where N = cache_size)
  • Cache storage: All returned values are cached with a timestamp
  • Value selection: Each second corresponds to one cache position. The plugin selects the value matching elapsed seconds since cache update
  • Cache exhaustion: If more time has elapsed than cache size, the last cached value is reused
  • Refresh strategy: After using the cache 2 times, the plugin attempts a fresh optimizer request
  • Fallback: If the optimizer request fails, the plugin continues using cached values

Example with cache_size=10:

Time (s)  | 0  1  2  3  4  5  6  7  8  9  | 10+
Value     | 3  3  4  4  5  5  6  6  7  7  | 7 (last)
Request   | ^  -  -  F  -  -  F  -  -  -  | F
          | F=Fresh request, -=Use cache

This design ensures smooth scaling transitions while maintaining responsiveness to changing demand patterns.

Scaling Logic ​

On every policy check evaluation, the plugin:

  1. Reads the strategy configuration (optimizer_token, optimizer_url, min, max, metric_app_name, cache_size), falling back to the agent-level plugin config for the token and URL
  2. Decides whether to serve from the cache or request fresh predictions (a fresh request is made when the cache is empty or has been used at least twice)
  3. On a fresh request, sends POST /optimize to the optimizer (5-second timeout) with the check details, current/min/max replicas and the collected metric samples
  4. Caches the returned desired array and selects the value for the elapsed second; on request failure, it falls back to the cached values
  5. Compares the selected count to the current count and returns the scaling direction and desired count to the Nomad Autoscaler

Metrics Collection ​

Nomad APM ​

Use Nomad's built-in metrics:

hcl
check "cpu" {
  source = "nomad-apm"
  query  = "avg_cpu_percent"
}

check "memory" {
  source = "nomad-apm"
  query  = "avg_memory_percent"
}

Prometheus ​

Query Prometheus metrics:

hcl
check "cpu" {
  source = "prometheus"
  query  = "avg(rate(container_cpu_usage_seconds_total[5m]))"
}

Troubleshooting ​

Plugin Not Loading ​

Check:

  • Plugin binary is in correct directory
  • Binary has execute permissions
  • Nomad Autoscaler logs for errors
bash
tail -f /var/log/nomad-autoscaler.log | grep productify-scaler

Optimizer Unreachable ​

Verify:

  • Optimizer service is running
  • URL is correct and accessible
  • Network connectivity
  • Firewall rules

Test connection:

bash
curl http://{ip}:{optimizer_port}/health

No Scaling Actions ​

Debug:

  • Check if policy is enabled
  • Verify metrics are being collected
  • Review cooldown period
  • Check min/max constraints

Enable debug logging:

hcl
log_level = "DEBUG"

See Also ​