On-Call Guide
How Productify alerts are triaged, escalated, and where they land. Pairs with the incident runbook, which has the per-alert first-response steps.
Severity definitions
| Severity | Meaning | Response | Examples |
|---|---|---|---|
| critical | Customer-facing outage or imminent data loss | Page immediately, 24/7 | ProxyDown, ManagerExecutorDown, TriggerQueueStalled |
| warning | Degradation or a risk that will become critical if ignored | Chat notify; action within business hours (sooner if trending) | ProxyHighLatency, TriggerExecutionErrors, CertificateExpiringSoon, DiskSpaceLow, OptimizerDown |
Severity is the severity label on each rule in deployments/ops/alerting/rules.yml.
Where alerts land
Routing is defined in deployments/ops/alerting/alertmanager.yml:
- critical → pager. Routed to the paging receiver (
pager-critical, PagerDuty/Opsgenie),group_wait10s,repeat_interval1h. - warning → chat. Routed to the chat receiver (
chat-warnings, Slack/Teams#productify-alerts),repeat_interval4h. - A critical alert inhibits the noisier warning alerts for the same instance so a single outage does not fan out into a wall of pages.
Receiver credentials (webhook URLs, routing keys) are injected from secrets at deploy time — the committed file holds placeholders only.
Escalation path
- Primary on-call acknowledges the page and works the runbook entry for the firing alert.
- If not mitigated within 30 minutes (critical), escalate to the secondary on-call.
- If the incident spans components or needs a customer comms decision, escalate to the incident lead / engineering manager and open an incident channel.
- For suspected security incidents (e.g. tenant-isolation), page the security owner and treat per the launch-blocking criteria in Launch Criteria.
During an incident
- Acknowledge the alert so it stops paging and ownership is visible.
- Keep a timeline in the incident channel (what you saw, what you changed).
- Prefer the reversible mitigation (restart, scale, fail over) before the investigative deep-dive; capture logs first if you must destroy state.
- After resolution, write a short postmortem: trigger, impact, timeline, and the follow-up that prevents recurrence.
Scope — what this covers and what is deferred
This on-call setup stops at alert rules on existing metrics plus the log aggregation and backup/restore drills shipped in this wave. Deeper observability — distributed tracing (OTel proxy→manager→ESB→backends), Grafana dashboard suites, and a formal per-application SLO layer with burn-rate alerting — is intentionally deferred to the project roadmap's future-directions §7 and is not part of this guide yet. When those land, dashboards and SLO burn-rate pages become the primary triage surface and this guide will point to them.