Productify

Appearance

Incident Runbook ​

First-response procedures keyed to the Productify alert rules. Each alert maps to: what it means, the first-response steps, the log-aggregation query to run, and the recovery action for data or capacity incidents.

Conventions

  • Loki/LogQL queries assume the shipped logging stack; run them in Grafana's Explore or via logcli.
  • Metrics are the Prometheus series named in each alert's expr.
  • Escalation, severities, and where alerts land are in the on-call guide.

Availability alerts ​

ProxyDown (critical) ​

All ingress is at risk: the proxy fronts the Manager, identity provider, and every app.

  1. Check the target: up{job="proxy"} in Prometheus; confirm the proxy container/allocation is running (docker compose ps proxy / nomad status).
  2. Logs: {service="proxy"} over the last 15m; look for panics, cert/ACME errors, or admin-API failures.
  3. Restart the proxy; if it will not start, check docker compose logs proxy / the allocation log for the config error. Two common causes: the Caddyfile prelude does not load (the Manager pushes rendered vhosts onto it through the proxy's admin API, so a bad prelude on disk outlives the process that read it; see Proxy Sync), and ID_DOMAIN not resolving from inside the proxy container, because caddy-security fetches Pocket ID's OIDC metadata while the proxy starts.
  4. If certificates are the cause, see CertificateExpiringSoon below.

ManagerExecutorDown (critical) ​

Cron triggers will not fire while the single executor is down.

  1. Confirm up{job="manager-executor"} and the executor process state.
  2. Logs: {service="manager"} |= "executor"; DB connectivity is the usual cause.
  3. Restart the executor.

Missed cron occurrences are dropped, not replayed

Each tick evaluates the window since the previous tick in this process, and that marker is reset to "now" when the executor starts. Every occurrence whose fire time fell inside the outage is therefore skipped permanently; there is no catch-up. Restart speed is the only thing that bounds the loss, so treat this alert as time-critical, and re-run anything business-critical by hand afterwards (an app whose trigger fires hourly loses one fire per hour of downtime).

The same reset is why a rolling restart of the executor is not free. The per-occurrence dedup key (last_run_key) only prevents a duplicate dispatch of an occurrence that already ran; it does not resurrect one that was never evaluated.

The executor also owns the audit-chain verification and retention sweeps, credential-staleness reporting, invite and CLI-login expiry, and outbox mail delivery. Those are idempotent sweeps that do resume normally on restart; only trigger occurrences are lost.

OptimizerDown (warning) ​

The scaler plugin serves cached predictions until they are time-exhausted, then applies the fallback policy (hold the current count; scale to 1 if demand appears with no live replica); not customer-facing.

  1. Confirm up{job="optimizer"} and restart the optimizer service.
  2. Logs: {service="optimizer"}; check Prometheus/Nomad reachability (the optimizer queries them for its forecasts).

Error & latency alerts ​

ProxyHighErrorRate (warning) ​

Cold-start shedding: backends are not coming up fast enough (pfy_activator_timeouts_total).

  1. Identify the app label on the alert; check that app's backend health.
  2. Logs: {service="proxy"} |= "activator" and the app's own logs.
  3. If a backend is stuck scaled-to-zero or crash-looping, redeploy/scale it.

ProxyHighLatency (warning) ​

p95 of pfy_response_time_seconds > 500ms for 10m.

  1. Correlate with request volume (rate(pfy_total_requests[5m]); the counter has no _total suffix) to tell load from a regression.
  2. If load: this is a capacity incident. See Capacity incidents.
  3. If not load: check downstream (Manager/DB latency), recent deploys, and {service="manager"} | level="error".

TriggerExecutionErrors (warning) ​

pfy_executor_trigger_execution_errors_total is rising.

  1. Logs: {service="manager"} |= "trigger" |= "error"; identify the failing trigger/app and the backend callback error.
  2. If a specific backend is unreachable, treat as an app-availability issue.

TriggerQueueStalled (critical) ​

Triggers are waiting but none are processed: the executor loop is wedged even though the process is up.

  1. Logs: {service="manager"} |= "queue"; look for a stuck transaction or a held advisory lock.
  2. Restart the executor (releases the DB-level lock); confirm rate(pfy_executor_queue_processed_total[5m]) recovers above zero.

Infrastructure alerts ​

CertificateExpiringSoon (warning) ​

probe_ssl_earliest_cert_expiry < 14 days.

  1. Confirm ACME is functioning: {service="proxy"} |= "certificate".
  2. Check DNS / port 80 reachability for the ACME challenge; the proxy renews automatically once the path is clear. Back up the cert store (caddy_data).

DiskSpaceLow (warning) ​

< 10% free; risks the database, logs, and cert storage.

  1. Identify the host/mountpoint from the alert labels.
  2. Free space: prune old Docker images, rotate/compact logs (Loki retention is tunable, see Logging), and confirm backup pruning (RETENTION_DAYS) is running.
  3. If the database volume is the culprit, expand it before it fills.

Data incidents (corruption / loss) ​

For data loss, restore from the verified backups (backup & restore drills):

  1. Identify the last good dump in BACKUP_DIR (manager-db_*.dump, pocket-id_*.dump).
  2. Prove it first: run restore-drill.sh against the candidate dump (MANAGER_DUMP=… POCKET_ID_DUMP=… deployments/ops/backup/restore-drill.sh; by default it takes the newest dump in BACKUP_DIR). It restores into a scratch Postgres container and verifies tables and row counts before you touch production.
  3. Restore into the real database per the recovery procedures in Disaster Recovery. Observe the documented RTO/RPO targets.

Capacity incidents ​

When latency/throughput degrade under load rather than from a fault:

  1. Scale the affected tier (Manager API instances, app backends); the executor and database do not scale horizontally (see Scaling Strategy).
  2. Validate the change with the load-test harness: run the k6 scenarios against the scaled stack and confirm the thresholds hold (deployments/ops/loadtest/). A threshold breach means the scale-out did not resolve the bottleneck; look downstream (DB, a single hot backend).

See Also ​