Productify

Appearance

Trigger System ​

The Productify Framework's trigger system provides event-driven automation with per-second execution scheduling and backend callback integration.

Overview ​

Triggers enable:

  • Scheduled Execution - Cron-based recurring tasks with per-second granularity
  • Backend Callbacks - HTTP POST notifications to registered backend services
  • Application Isolation - Triggers scoped to specific applications
  • Distributed Locking - Cluster-safe execution with database-level locks
  • Concurrent Execution - Parallel trigger processing with configurable limits

Core Concepts ​

Trigger ​

A trigger defines:

  • Name - Descriptive identifier for the trigger
  • Description - Purpose and details (max 1000 chars)
  • Cron Expression - Schedule using extended cron syntax with second-level precision
  • Run Key - Unique identifier for backend routing (max 100 chars)
  • Enabled - Toggle to enable/disable execution
  • Application - The application this trigger belongs to
  • Timestamps - created_at and updated_at tracking
  • Audit - Creator and last editor user references

Backend Registration ​

Backends register to receive trigger callbacks via the Machine User API:

  • Application ID - The application to receive triggers for
  • Callback URL - HTTPS endpoint to receive execution notifications
  • Machine User - Authenticated service account
  • Token - Bearer token for authenticating callbacks from Manager
  • Version - Optional backend version identifier
  • Heartbeat - Last contact timestamp for health monitoring

Execution Flow ​

When the trigger executor runs (configurable interval, default 1 second):

  1. Lock Acquisition - Acquire distributed lock (trigger_executor) to ensure single executor
  2. Query Triggers - Find all enabled triggers
  3. Schedule Evaluation - Check cron expressions against current time
  4. Queue Metrics - Increment queue counters per application
  5. Concurrent Dispatch - Execute up to 10 triggers in parallel
  6. Backend Discovery - For each trigger, find active registered backends
  7. HTTP Callbacks - POST trigger payload to each backend's callback URL
  8. Metrics Recording - Track execution time, success/error rates
  9. Lock Release - Release distributed lock for next cycle

The executor uses database-level locking (executor_lock table) to ensure only one instance processes triggers at a time across a cluster, preventing duplicate executions.

Monitoring & Metrics ​

The executor exposes Prometheus metrics on a separate HTTP port (default: 9091) at /metrics:

Per-Application Metrics (labeled with app=<application-id>):

  • pfy_executor_queue_all_total - Total triggers queued for execution
  • pfy_executor_queue_processed_total - Total triggers successfully processed
  • pfy_executor_queue_waiting - Current number of triggers waiting (gauge)
  • pfy_executor_queue_process_time_seconds - Processing time histogram

Global Metrics:

  • pfy_executor_active_triggers - Number of enabled triggers
  • pfy_executor_trigger_execution_total - Total backend dispatches
  • pfy_executor_trigger_execution_errors_total - Total execution errors
  • pfy_executor_backend_dispatch_total{status} - Backend dispatch count (success/error)
  • pfy_executor_backend_dispatch_duration_seconds - Dispatch latency histogram
  • pfy_executor_trigger_check_duration_seconds - Complete check cycle duration

See Manager Deployment Guide for metrics details.

Creating Triggers ​

Triggers are created through the GraphQL API (UI support pending):

graphql
mutation {
  createTrigger(
    input: {
      name: "Daily Report Generation"
      description: "Generates daily reports at midnight UTC"
      cronExpression: "0 0 0 * * *"
      runKey: "daily-report"
      enabled: true
      applicationID: "app-uuid"
      creatorID: "user-uuid"
      editorID: "user-uuid"
    }
  ) {
    id
    name
    description
    cronExpression
    runKey
    enabled
    createdAt
    application {
      id
      name
    }
  }
}

Required Fields:

  • name - Max 100 characters
  • cronExpression - Valid cron expression (6 or 5 fields)
  • runKey - Max 100 characters
  • applicationID - Must belong to your tenant
  • creatorID - Current user ID
  • editorID - Current user ID

Optional Fields:

  • description - Max 1000 characters
  • enabled - Defaults to false

Cron Schedule Syntax ​

Triggers support extended cron syntax with per-second precision using the robfig/cron library:

Extended Format (6 fields - per-second) ​

* * * * * *
│ │ │ │ │ └── Day of week (0-6, Sunday=0)
│ │ │ │ └──── Month (1-12)
│ │ │ └────── Day of month (1-31)
│ │ └──────── Hour (0-23)
│ └────────── Minute (0-59)
└──────────── Second (0-59)

Standard Format (5 fields) ​

* * * * *
│ │ │ │ └── Day of week (0-6, Sunday=0)
│ │ │ └──── Month (1-12)
│ │ └────── Day of month (1-31)
│ └──────── Hour (0-23)
└────────── Minute (0-59)

Examples ​

Every hour (standard format):

0 * * * *

Every day at midnight (standard format):

0 0 * * *

Every Monday at 9am (standard format):

0 9 * * 1

Every 15 minutes:

*/15 * * * *

First day of every month at midnight:

0 0 1 * *

Per-second examples (6-field format):

Every second:

* * * * * *

Every 10 seconds:

*/10 * * * * *

At 30 seconds past every minute:

30 * * * * *

Special Strings:

The cron parser also supports special strings:

  • @yearly or @annually - Run once a year (midnight Jan 1)
  • @monthly - Run once a month (midnight first day)
  • @weekly - Run once a week (midnight Sunday)
  • @daily or @midnight - Run once a day (midnight)
  • @hourly - Run once an hour (start of hour)

Backend Integration ​

Registering a Backend ​

Backends register to receive trigger callbacks using the Machine User REST API:

Endpoint: POST /api/machine/register-backend

Authentication: Bearer Token (Machine User)

Request:

bash
curl -X POST https://manager.example.com/api/machine/register-backend \
  -H "Authorization: Bearer <machine-user-token>" \
  -H "Content-Type: application/json" \
  -d '{
    "application_id": "550e8400-e29b-41d4-a716-446655440000",
    "callback_url": "https://backend.example.com/triggers/callback",
    "version": "1.0.0"
  }'

Response:

json
{
  "success": true,
  "registered_id": "backend-uuid",
  "token": "generated-bearer-token-for-callbacks",
  "message": "Backend registered successfully",
  "tenant_id": "tenant-uuid",
  "application_id": "app-uuid"
}

Important:

  • The returned token is used by the Manager to authenticate callbacks to your backend
  • Store this token securely and validate it on incoming trigger requests
  • Re-registering with the same machine user and application updates the registration and returns the same token
  • Registration is tied to the machine user's tenant - application must belong to that tenant

Receiving Callbacks ​

When a trigger executes, the Manager sends a POST request to each registered backend:

Request Headers:

http
POST /triggers/callback
Content-Type: application/json
Authorization: Bearer <backend-token-from-registration>

Request Body:

json
{
  "trigger_id": "trigger-uuid",
  "trigger_name": "Hourly Payment Processing",
  "cron_expression": "0 * * * * *",
  "run_key": "hourly-payment",
  "executed_at": "2025-12-04T10:00:00Z"
}

Note: The current implementation sends minimal payload. Application/tenant context can be inferred from the machine user registration.

Expected Response:

Your backend should respond with HTTP 2xx status:

json
{
  "status": "success",
  "message": "Processing completed"
}

Or on error (5xx status):

json
{
  "status": "error",
  "message": "Processing failed: database timeout"
}

Authentication:

  • Verify the Authorization header matches the token from registration
  • Reject requests without valid tokens (401 Unauthorized)
  • Consider implementing IP allowlisting for additional security

Managing Triggers ​

Listing Triggers ​

Query all triggers for an application:

graphql
query {
  application(id: "app-uuid") {
    triggers {
      id
      name
      description
      cronExpression
      runKey
      enabled
      createdAt
      updatedAt
      creator {
        id
        displayName
      }
      editor {
        id
        displayName
      }
    }
  }
}

Updating Triggers ​

Update schedule and enable:

graphql
mutation {
  updateTrigger(
    id: "trigger-uuid"
    input: { cronExpression: "0 2 * * *", enabled: true, editorID: "user-uuid" }
  ) {
    id
    cronExpression
    enabled
    updatedAt
  }
}

Update description and run key:

graphql
mutation {
  updateTrigger(
    id: "trigger-uuid"
    input: {
      description: "Updated description"
      runKey: "new-run-key"
      editorID: "user-uuid"
    }
  ) {
    id
    description
    runKey
  }
}

Enabling/Disabling ​

Toggle trigger execution without deleting:

graphql
mutation {
  updateTrigger(
    id: "trigger-uuid"
    input: { enabled: false, editorID: "user-uuid" }
  ) {
    id
    enabled
  }
}

Deleting Triggers ​

graphql
mutation {
  deleteTrigger(id: "trigger-uuid")
}

Best Practices ​

Scheduling ​

  • Avoid Overlap - Ensure trigger execution completes before next scheduled run
  • Off-Peak Hours - Schedule resource-intensive jobs during low-traffic periods
  • Stagger Triggers - Distribute load by offsetting similar triggers
  • Timezone Awareness - All times are UTC; adjust schedules accordingly
  • Test Expressions - Validate cron syntax before enabling (use https://crontab.guru for 5-field format)

Backend Design ​

  • Idempotency - Design handlers to safely handle duplicate calls
  • Quick Response - Return HTTP 2xx immediately, process async if needed
  • Token Validation - Always verify the Bearer token from registration
  • Error Handling - Return appropriate HTTP status codes (5xx for retryable errors)
  • Logging - Log all trigger executions with trigger_id and run_key for debugging
  • Timeout Protection - Manager uses 5-second HTTP timeout; respond within this window

Reliability ​

  • Monitor Metrics - Track pfy_executor_* metrics in Prometheus
  • Alert on Failures - Set up alerts for backend_dispatch_total{status="error"}
  • Test Callbacks - Use tools like ngrok for local development/testing
  • Health Checks - Update registration periodically (implicit heartbeat)
  • Graceful Degradation - Handle Manager downtime (missing triggers) gracefully

Security ​

  • HTTPS Only - Use TLS for callback URLs (required in production)
  • Token Security - Store backend tokens securely (environment variables, secrets manager)
  • Token Validation - Verify Authorization header on every request
  • Rate Limiting - Protect callback endpoints from abuse
  • IP Allowlisting - Consider restricting to Manager's IP addresses
  • Payload Validation - Verify request structure before processing

Troubleshooting ​

Trigger Not Executing ​

Check:

  • Trigger is enabled (enabled: true)
  • Cron expression is valid (test with https://crontab.guru or similar)
  • Trigger executor service is running (docker ps or systemctl status)
  • Executor lock is not stuck (check executor_lock table in database)
  • Check executor logs for errors
  • Verify pfy_executor_active_triggers metric shows your trigger

Backend Not Receiving Callbacks ​

Verify:

  • Backend is registered via POST /api/machine/register-backend
  • Machine user token is valid (GET /api/machine/check)
  • Callback URL is accessible from Manager (network/firewall)
  • Callback URL uses HTTPS (required in production)
  • Backend is returning 2xx status codes
  • Check Manager logs for dispatch errors
  • Monitor pfy_executor_backend_dispatch_total{status="error"} metric

Schedule Not Working as Expected ​

Common issues:

  • Timezone confusion - All times are UTC, not local time
  • Cron syntax error - Verify field count (5 or 6 fields)
  • Second-precision disabled - Use 6-field format for per-second execution
  • Overlapping executions - Previous run still processing when next scheduled
  • Test online - Use https://crontab.guru for standard format

Authentication Failures ​

Backend receives 401 Unauthorized:

  • Verify Bearer token in Authorization header matches registration token
  • Check token is not expired or revoked
  • Ensure machine user is still enabled

Registration fails:

  • Verify machine user has access to the application
  • Check application belongs to machine user's tenant
  • Ensure machine user is enabled

Performance Issues ​

Slow trigger execution:

  • Check pfy_executor_trigger_check_duration_seconds metric
  • Verify database performance (lock table queries)
  • Review concurrent execution limit (default 10)
  • Monitor pfy_executor_queue_process_time_seconds per app

High error rates:

  • Check backend response times (5s timeout)
  • Review backend error logs
  • Monitor pfy_executor_backend_dispatch_duration_seconds

Multiple Backends per Application ​

An application can have multiple backend registrations:

  • Different machine users (different services)
  • Same machine user registering multiple times (updates existing)
  • Each receives all triggers for the application
  • Useful for distributed processing or redundancy

Example:

bash
# Service A registers
curl -X POST .../register-backend \
  -H "Authorization: Bearer <service-a-token>" \
  -d '{"application_id": "app-uuid", "callback_url": "https://service-a.com/triggers"}'

# Service B registers
curl -X POST .../register-backend \
  -H "Authorization: Bearer <service-b-token>" \
  -d '{"application_id": "app-uuid", "callback_url": "https://service-b.com/triggers"}'

Both services receive trigger callbacks. Use run_key to coordinate who handles what.

High-Frequency Triggers ​

For very frequent execution (e.g., every second):

graphql
mutation {
  createTrigger(
    input: {
      name: "Monitor Queue"
      cronExpression: "* * * * * *" # Every second
      runKey: "queue-monitor"
      applicationID: "app-uuid"
      creatorID: "user-uuid"
      editorID: "user-uuid"
    }
  ) {
    id
  }
}

Considerations:

  • Backend must respond quickly (< 1s to avoid backlog)
  • Consider async processing to decouple response from work
  • Monitor pfy_executor_queue_waiting for growing queues
  • May impact database performance at scale

Configuration ​

Manager Configuration ​

Trigger executor settings in config.yml:

yaml
cron:
  # How often to check for triggers to execute (default: 1s)
  trigger_check_interval: 1s

  # Timeout for inactive backend registrations (default: 5m)
  backend_heartbeat_timeout: 5m

  # Port for Prometheus metrics endpoint (default: 9091)
  metrics_port: 9091

Deployment Architecture ​

Single Instance:

Manager (Executor Enabled) → Database → Backend Services

Multi-Instance Cluster:

Manager 1 (Executor) ─┐
                       ├→ Database ─→ Backend Services
Manager 2 (Executor) ─┘

Only one instance executes triggers at a time (distributed lock), but both can handle registration API calls.

Separate Executor:

Manager API (No Executor) ─┐
                            ├→ Database ─→ Backend Services
Dedicated Executor        ─┘

Run executor separately using environment variable:

bash
ENABLE_EXECUTOR=true ./manager

API Reference ​

GraphQL Mutations ​

Create Trigger:

graphql
mutation createTrigger($input: CreateTriggerInput!)

Update Trigger:

graphql
mutation updateTrigger($id: ID!, $input: UpdateTriggerInput!)

Delete Trigger:

graphql
mutation deleteTrigger($id: ID!): Boolean

REST Endpoints ​

Register Backend:

POST /api/machine/register-backend
Authorization: Bearer <machine-user-token>

Check Machine User:

GET /api/machine/check
Authorization: Bearer <machine-user-token>

Metrics Endpoint:

GET /metrics

(Exposed on separate port, default 9091)

For complete API documentation, see: