On-Call Guide
How Productify alerts are triaged, who is paged, and how an incident escalates. It pairs with the incident runbook, which has the per-alert first-response steps.
Severity definitions
| Severity | Meaning | Response | Examples |
|---|---|---|---|
| critical | Customer-facing outage or imminent data loss | Page immediately, 24/7 | ProxyDown, ManagerExecutorDown, TriggerQueueStalled |
| warning | Degradation or a risk that will become critical if ignored | Chat notify; action within business hours (sooner if trending) | ProxyHighLatency, TriggerExecutionErrors, CertificateExpiringSoon, DiskSpaceLow, OptimizerDown |
Severity is the severity label on each rule in deployments/ops/alerting/rules.yml.
Where alerts land
Routing is defined in deployments/ops/alerting/alertmanager.yml:
- critical goes to the pager: the paging receiver (
pager-critical, PagerDuty/Opsgenie),group_wait10s,repeat_interval1h. - warning goes to chat: the chat receiver (
chat-warnings, Slack/Teams#productify-alerts),group_wait30s,repeat_interval4h. - A critical alert inhibits warning alerts that share its
instancelabel, so a single outage does not fan out into a wall of pages.
Receiver credentials (webhook URLs, routing keys) are injected from secrets at deploy time; the committed file holds placeholders only.
Escalation path
- Primary on-call acknowledges the page and works the runbook entry for the firing alert.
- If not mitigated within 30 minutes (critical), escalate to the secondary on-call.
- If the incident spans components or needs a customer comms decision, escalate to the incident lead / engineering manager and open an incident channel.
- For suspected security incidents (e.g. tenant-isolation), page the security owner and treat per the launch-blocking criteria in Launch Criteria.
During an incident
- Acknowledge the alert so it stops paging and ownership is visible.
- Keep a timeline in the incident channel (what you saw, what you changed).
- Prefer the reversible mitigation (restart, scale, fail over) before the investigative deep-dive; capture logs first if you must destroy state.
- After resolution, write a short postmortem: trigger, impact, timeline, and the follow-up that prevents recurrence.
Scope
This on-call setup stops at alert rules on existing metrics plus the shipped log aggregation and backup/restore drills. Distributed tracing (OTel across proxy, Manager, ESB and backends), Grafana dashboard suites, and a per-application SLO layer with burn-rate alerting are on the project roadmap (future directions §7) and are not part of this guide. When they land, dashboards and SLO burn-rate pages become the primary triage surface and this guide will point to them.