Productify

Appearance

Incident Runbook ​

First-response procedures keyed to the Productify alert rules. Each alert maps to: what it means, the first-response steps, the log-aggregation query to run, and the recovery action for data or capacity incidents.

Conventions

  • Loki/LogQL queries assume the shipped logging stack; run them in Grafana's Explore or via logcli.
  • Metrics are the Prometheus series named in each alert's expr.
  • Escalation, severities, and where alerts land are in the on-call guide.

Availability alerts ​

ProxyDown (critical) ​

All ingress is at risk — the proxy fronts the Manager, identity provider, and every app.

  1. Check the target: up{job="proxy"} in Prometheus; confirm the proxy container/allocation is running (docker compose ps proxy / nomad status).
  2. Logs: {service="proxy"} over the last 15m; look for panics, cert/ACME errors, or admin-API failures.
  3. Restart the proxy; if it will not start, check the Caddyfile and the Manager reachability (the proxy pulls vhosts from the Manager admin API).
  4. If certificates are the cause, see CertificateExpiringSoon below.

ManagerExecutorDown (critical) ​

Cron triggers will not fire while the single executor is down.

  1. Confirm up{job="manager-executor"} and the executor process state.
  2. Logs: {service="manager"} |= "executor" — DB connectivity is the usual cause.
  3. Restart the executor. Missed executions catch up on restart (trigger state is in the database), so a fast restart (<30s) bounds the impact.

OptimizerDown (warning) ​

Autoscaling falls back to reactive scaling; not customer-facing.

  1. Confirm up{job="optimizer"} and restart the optimizer service.
  2. Logs: {service="optimizer"}; check Prometheus/Nomad reachability (the optimizer queries them for its forecasts).

Error & latency alerts ​

ProxyHighErrorRate (warning) ​

Cold-start shedding — backends are not coming up fast enough (pfy_activator_timeouts_total).

  1. Identify the app label on the alert; check that app's backend health.
  2. Logs: {service="proxy"} |= "activator" and the app's own logs.
  3. If a backend is stuck scaled-to-zero or crash-looping, redeploy/scale it.

ProxyHighLatency (warning) ​

p95 of pfy_response_time_seconds > 500ms for 10m.

  1. Correlate with request volume (rate(pfy_total_requests[5m])) — is this load or a regression?
  2. If load: this is a capacity incident — see Capacity incidents.
  3. If not load: check downstream (Manager/DB latency), recent deploys, and {service="manager"} | level="error".

TriggerExecutionErrors (warning) ​

pfy_executor_trigger_execution_errors_total is rising.

  1. Logs: {service="manager"} |= "trigger" |= "error" — identify the failing trigger/app and the backend callback error.
  2. If a specific backend is unreachable, treat as an app-availability issue.

TriggerQueueStalled (critical) ​

Triggers are waiting but none are processed — the executor loop is wedged even though the process is up.

  1. Logs: {service="manager"} |= "queue"; look for a stuck transaction or a held advisory lock.
  2. Restart the executor (releases the DB-level lock); confirm rate(pfy_executor_queue_processed_total[5m]) recovers above zero.

Infrastructure alerts ​

CertificateExpiringSoon (warning) ​

probe_ssl_earliest_cert_expiry < 14 days.

  1. Confirm ACME is functioning: {service="proxy"} |= "certificate".
  2. Check DNS / port 80 reachability for the ACME challenge; the proxy renews automatically once the path is clear. Back up the cert store (caddy_data).

DiskSpaceLow (warning) ​

< 10% free — risks the database, logs, and cert storage.

  1. Identify the host/mountpoint from the alert labels.
  2. Free space: prune old Docker images, rotate/compact logs (Loki retention is tunable — see Logging), and confirm backup pruning (RETENTION_DAYS) is running.
  3. If the database volume is the culprit, expand it before it fills.

Data incidents (corruption / loss) ​

For data loss, restore from the verified backups (backup & restore drills):

  1. Identify the last good dump in BACKUP_DIR (manager-db_*.dump, pocket-id_*.dump).
  2. Prove it first: run restore-drill.sh against the candidate dump — it restores into a scratch instance and verifies tables/row-counts before you touch production.
  3. Restore into the real database per the recovery procedures in Disaster Recovery. Observe the documented RTO/RPO targets.

Capacity incidents ​

When latency/throughput degrade under load rather than from a fault:

  1. Scale the affected tier (Manager API instances, app backends) — the executor and database do not scale horizontally (see Scaling Strategy).
  2. Validate the change with the load-test harness: run the k6 scenarios against the scaled stack and confirm the thresholds hold (deployments/ops/loadtest/). A threshold breach means the scale-out did not resolve the bottleneck — look downstream (DB, a single hot backend).

See Also ​