Trigger System

The trigger system is the Manager's scheduler. A trigger is a cron expression attached to an application; when it is due, the Manager's executor sends an HTTP POST to every backend that has registered for that application. For the UI, see Triggers.

The pieces

A trigger is stored per application with name (≤100 chars), description (≤1000), cronExpression (≤50, six fields, seconds first), runKey (≤100, the identifier your backend switches on), enabled (default false), timestamps, and the users who created and last edited it.

runKey is a routing label, not a secret

It names which handler runs. Operators author it, it is freely editable, it has no entropy requirement, and nothing authenticates with it; the credential on this path is the registered backend's callback token. Do not treat it as sensitive, and do not confuse it with the Manager's internal occurrence key described below, which is also stored on the trigger as last_run_key.

A registered backend is a callback registration made by a backend through the machine-user API. It stores the callback_url, an optional version, an active flag, the last heartbeat time and a callback token the Manager sends as Authorization: Bearer on every call. A registration is scoped to one (application, environment) pair, because machine users are environment-scoped.

The executor is the part of the Manager that runs when run_mode is executor or both. It evaluates schedules, dispatches callbacks, retries undelivered runs, and exposes Prometheus metrics.

How a trigger fires

Every cron.trigger_check_interval (default 30 s) the executor:

1. Takes the executor lease (executor_lock table, id trigger_executor). With several executor processes, only the holder proceeds; the others skip the pass.

2. Loads all enabled triggers.

3. For each trigger, computes every occurrence of its schedule in the window (previous pass, now]. A schedule that fires more often than the pass interval produces several occurrences and all are dispatched, in order; more than 60 in one window are truncated to 60 and logged. Each occurrence gets an occurrence key (<trigger id> + <scheduled time>), and an occurrence already recorded as dispatched (last_run_key) is skipped, so a restart does not re-fire. This is the Manager's own idempotency key and is not the trigger's runKey; it never reaches the payload.

4. Dispatches up to 10 triggers concurrently; inside them, up to 50 backend calls run at once.

5. For each occurrence, loads the application's active registered backends (across all of its environments) and POSTs the payload to each callback_url with a 5 second timeout. A 2xx response counts as success; anything else is logged and counted as an error. The URL is re-checked against the private-address denylist at dispatch, and redirects are re-validated hop by hop.

6. If the application has no active backend (for example it is scaled to zero), the occurrence is not dropped: it is stored as a pending run and retried with backoff (starting at 15 s, up to 10 attempts) until a backend registers or the attempts run out, after which it is marked failed and pfy_executor_trigger_delivery_failed_total is incremented. Failed runs are pruned after cron.failed_trigger_run_retention (default 7 days).

   The 10 attempts are per occurrence, not per reason, and a pending run can exhaust them two different ways, which matters because both increment the same counter:

   • No backend ever appeared. Nothing was delivered. The response is to find out why the application is not running.
   • Delivery succeeded but the run could not be cleared. The occurrence was POSTed, up to 10 times, to a backend that answered 2xx each time. The bookkeeping that removes the pending row failed, so the next pass redelivered it. The response is the opposite of the first case: the application is fine, the Manager's database write is not, and the customer's scheduled job has run repeatedly.

   The log line distinguishes them; the counter alone does not. An alert on pfy_executor_trigger_delivery_failed_total should read the executor log before assuming a scaled-to-zero application.

7. Publishes each enabled trigger's next fire time as pfy_executor_next_trigger_run_seconds, which the autoscaler uses to pre-scale.

8. Releases the lease.

Schedules are evaluated in the Manager server's local time zone.

The callback contract

Request the backend receives

POST <callback_url>
Content-Type: application/json
Authorization: Bearer <callback token from registration>

{
  "trigger_id": "trigger-uuid",
  "trigger_name": "Hourly Payment Processing",
  "cron_expression": "0 0 * * * *",
  "run_key": "hourly-payment",
  "executed_at": "2025-12-04T10:00:00Z"
}

The payload carries no tenant. One delivery covers every tenant the application is enabled for; the backend lists them with GET /api/machine/application/:id/tenants (machine-user auth, response {"success": true, "tenants": [{"id": "...", "name": "..."}]}) and loops. See Backend Registration.

What the backend must do

• Verify the token: compare the Authorization header with the token from your latest registration response and answer 401 otherwise.
• Answer fast: return 2xx within 5 seconds; do the real work asynchronously. A slow or non-2xx answer is counted as an error and is not retried for an existing backend.
• Be idempotent: the same occurrence can reach you twice in edge cases (a retry after a timeout you actually served). Deduplicate on trigger_id + executed_at.
• Use run_key to decide what to do; do not switch on trigger_name, which operators may rename.

Registering a backend

Backends register with a machine user's Bearer token. The machine user is created per environment in the UI (or via createMachineUserWithCredentials); see Machine Users.

curl -X POST https://manager.example.com/api/machine/register-backend \
  -H "Authorization: Bearer <machine-user-token>" \
  -H "Content-Type: application/json" \
  -d '{
    "application_id": "550e8400-e29b-41d4-a716-446655440000",
    "callback_url": "https://backend.example.com/triggers/callback",
    "version": "1.0.0"
  }'

First registration answers 201:

{
  "success": true,
  "registered_id": "backend-uuid",
  "token": "callback-token",
  "message": "Backend registered successfully",
  "environment_id": "env-uuid",
  "application_id": "550e8400-e29b-41d4-a716-446655440000"
}

Re-registration by the same machine user for the same application answers 200 with "message": "Backend registration updated successfully", updates the callback URL and version, re-activates the row, and rotates the token; always use the token from the latest response.

Rules enforced by the handler:

• The application must belong to the same project as the machine user's environment; otherwise 404 "Application not found or does not belong to your project".
• callback_url must be a syntactically valid URL and must not resolve to a private, loopback, link-local or cloud-metadata address (400 "Invalid callback_url: ...").
• Requests to /api/machine/* are rate-limited before the token is checked.

Other machine-user routes:

Route	Purpose
GET /api/machine/check	Validates the token; returns {"authenticated": true, "machine_user_id": "...", "environment_id": "..."}
PUT /api/machine/backend/:id/heartbeat	Records a heartbeat and re-activates the registration. Send it more often than cron.backend_heartbeat_timeout (default 1 min); a registration that misses it is marked inactive and stops receiving triggers
DELETE /api/machine/backend/:id	Removes the registration. An unknown id answers 404 "Backend registration not found", which the integration SDKs treat as already gone
GET /api/machine/application/:id/tenants	Enabled tenants for the fan-out

Re-registration is the only thing that rotates a callback token. When one reaches cron.credential_rotation_max_age (default 90 days) the executor's credential-staleness sweep reports it, on the pfy_executor_stale_credentials{kind="backend_token"} gauge and in one audit row, and leaves it working; re-register to rotate it.

Managing triggers with GraphQL

Creating, updating and deleting a trigger requires the maintainer or owner role on the application's project.

Create

mutation {
  createTrigger(
    applicationId: "app-uuid"
    input: {
      name: "Daily Report Generation"
      description: "Generates daily reports at midnight"
      cronExpression: "0 0 0 * * *"
      runKey: "daily-report"
      enabled: true
    }
  ) {
    id
    name
    cronExpression
    runKey
    enabled
    createdAt
    application {
      id
      name
    }
  }
}

name, description, cronExpression and runKey are required in CreateTriggerInput; enabled defaults to false. Creator and editor are set from the caller. An unparseable cronExpression is rejected with invalid cron expression "...".

List and read

query {
  triggers(
    application: "app-uuid"
    filters: []
    order: { field: "name", direction: ASC }
    pagination: { limit: 50, offset: 0 }
  ) {
    id
    name
    description
    cronExpression
    runKey
    enabled
    createdAt
    updatedAt
  }
  trigger(application: "app-uuid", id: "trigger-uuid") {
    id
    name
  }
}

Update, enable, disable

mutation {
  updateTrigger(
    id: "trigger-uuid"
    input: { cronExpression: "0 0 2 * * *", enabled: true }
  ) {
    id
    cronExpression
    enabled
    updatedAt
  }
}

Every field of UpdateTriggerInput (name, description, cronExpression, runKey, enabled) is optional. updateTrigger(id, input: { enabled: false }) pauses a trigger without deleting it.

Delete

mutation {
  deleteTrigger(id: "trigger-uuid")
}

Returns Boolean.

Cron syntax

The parser is robfig/cron configured with a seconds field and the @ descriptors:

* * * * * *
│ │ │ │ │ └── day of week (0-6, Sunday = 0)
│ │ │ │ └──── month (1-12)
│ │ │ └────── day of month (1-31)
│ │ └──────── hour (0-23)
│ └────────── minute (0-59)
└──────────── second (0-59)

Expression	Meaning
0 0 * * * *	Every hour
0 0 0 * * *	Every day at midnight
0 0 9 * * 1	Mondays at 09:00
0 */15 * * * *	Every 15 minutes
0 0 0 1 * *	First of the month at midnight
*/10 * * * * *	Every 10 seconds
30 * * * * *	30 seconds past every minute
* * * * * *	Every second
@hourly @daily/@midnight @weekly @monthly @yearly/@annually	Descriptors

Five-field expressions are rejected. For per-second schedules set cron.trigger_check_interval: 1s; with the default 30 s interval a per-second trigger produces 30 occurrences per pass, all dispatched back-to-back.

Monitoring and metrics

The executor serves Prometheus metrics on cron.metrics_port (default 9090) at /metrics. All names carry the pfy_executor_ prefix.

Per application (label app = application id):

Metric	Type	Meaning
pfy_executor_queue_all_total	counter	Occurrences queued
pfy_executor_queue_processed_total	counter	Occurrences processed
pfy_executor_queue_waiting	gauge	Occurrences waiting, including pending (undelivered) runs
pfy_executor_queue_process_time_seconds	histogram	Time per occurrence
pfy_executor_trigger_delivery_failed_total	counter	Occurrences given up after the retry budget
pfy_executor_next_trigger_run_seconds	gauge (app, trigger)	Seconds until each enabled trigger's next fire

Global:

Metric	Type	Meaning
pfy_executor_active_triggers	gauge	Enabled triggers
pfy_executor_trigger_execution_total	counter	Dispatch attempts
pfy_executor_trigger_execution_errors_total	counter	Executor-side errors (e.g. a trigger with no application)
pfy_executor_backend_dispatch_total{status="success"|"error"}	counter	Callback outcomes
pfy_executor_backend_dispatch_duration_seconds	histogram	Callback latency
pfy_executor_trigger_check_duration_seconds	histogram	Duration of one pass
pfy_executor_proxy_sync_total{outcome}, pfy_executor_proxy_sync_duration_seconds	counter, histogram	Proxy vhost pushes (not trigger-related, same registry)
pfy_executor_audit_chain_verification_failures_total	counter	Audit-chain breaks found by the maintenance sweep

Per-run outcomes are not stored in the database, so the dashboards cannot show a last-run status; the metrics and the executor log are the record.

Configuration

run_mode: both            # api | executor | both
cron:
  trigger_check_interval: 1s          # default 30s
  backend_heartbeat_timeout: 1m       # default 1m
  metrics_port: 9090                  # default 9090
  failed_trigger_run_retention: 168h  # default 7 days
  credential_rotation_max_age: 2160h  # default 90 days; reports, does not rotate

Single process: run_mode: both. Scaled API: several run_mode: api processes plus at least one executor; the API processes accept registrations and heartbeats, the executor fires triggers. Several executors are safe; the lease guarantees one pass at a time. See Configuration.

Troubleshooting

Symptom	Check
Trigger never fires	enabled is true; a process with run_mode executor/both is running; pfy_executor_active_triggers counts it; the executor log
Backend gets nothing	Registration exists and is active (heartbeats within backend_heartbeat_timeout); callback URL reachable from the Manager and not a private/loopback address; pfy_executor_backend_dispatch_total{status="error"}; Manager log
Fires at the wrong time	Six fields, seconds first; the Manager server's time zone
Registration fails	Machine user enabled and its environment in the same project as the application; callback_url public and syntactically valid
Backend answers 401	Compare against the token in the latest registration response; a callback token changes on re-registration and at no other time
Backlog grows	pfy_executor_queue_waiting; backend latency vs the 5 s timeout; pass duration in pfy_executor_trigger_check_duration_seconds

Related

• Triggers (UI guide)
• Backend Registration: implementation examples in Node, Python and Go
• Machine Users
• Configuration