Productify

Appearance

Deployment Architecture ​

Complete architecture overview for deploying the Productify Framework.

System Architecture ​

One proxy instance fronts everything — the Manager, the identity provider, and every application. Its per-application vhosts are rendered by the Manager from deployment data (domains, upstreams) and pushed through Caddy's admin API; each application runs as one backend per environment serving all tenants, receiving identity via injected headers.

                    ┌─────────────────┐
                    │   Internet      │
                    └────────┬────────┘
                             │
                    ┌────────▼───────────────────┐
                    │  Productify Proxy (Caddy)  │
                    │  - TLS · OAuth2/OIDC       │◀── vhosts rendered by the
                    │  - Tenant selection        │    Manager, pushed via the
                    │  - Identity header inject  │    admin API (:2019)
                    │  - Routing / Load Balance  │
                    └───┬──────────┬─────────────┘
                        │          │ trusted headers only
                        │   ┌──────▼─────────────────────┐
                        │   │  Application Backends      │
                        │   │  one per app × environment,│
                        │   │  each serving ALL tenants  │
                        │   └──────┬─────────────────────┘
                        │          │ register · heartbeat ·
                        │          │ per-tenant config
            ┌───────────┼──────────┴──────┐
            │           │                 │
    ┌───────▼──────┐  ┌─▼───────────┐  ┌──▼───────────┐
    │   Manager    │  │  Manager    │  │   Manager    │
    │   API (1)    │  │  API (2)    │  │   API (N)    │
    │   Port 8080  │  │  Port 8080  │  │   Port 8080  │
    └───────┬──────┘  └──────┬──────┘  └─────┬────────┘
            └────────────────┼────────────────┘
                             │
                    ┌────────▼────────┐   ┌──────────────────┐
                    │   Manager       │   │ Identity Provider │
                    │   Executor      │   │ (OIDC), behind    │
                    │   (Single)      │   │ the same proxy    │
                    │   - Triggers    │   └──────────────────┘
                    └────────┬────────┘
                             │
                    ┌────────▼────────┐
                    │   PostgreSQL    │
                    │   Database      │
                    │   (HA Cluster)  │
                    └─────────────────┘

┌───────────────────────────────────────────────────────┐
│            Nomad Cluster                              │
│  ┌──────────────┐        ┌─────────────────────┐      │
│  │  Autoscaler  │◄──────►│  Optimizer Service  │      │
│  │  Plugin      │  HTTP  │  (Python/MILP)      │      │
│  └──────────────┘        └─────────────────────┘      │
│                                                       │
│  ┌─────────────────────────────────────────────┐      │
│  │  Application Workloads                      │      │
│  │  (Auto-scaled by Nomadscaler)               │      │
│  └─────────────────────────────────────────────┘      │
└───────────────────────────────────────────────────────┘

Component Overview ​

Manager ​

API Instances (Stateless)

  • Handle GraphQL and REST requests
  • Horizontally scalable
  • Load-balanced by Proxy
  • No persistent state

Executor Instance (Stateful)

  • Runs trigger execution loop
  • Single instance (no scaling)
  • Manages cron schedules
  • Sends backend callbacks

Database

  • PostgreSQL
  • Primary + standby replicas (recommended for production)

Proxy ​

Caddy Server

  • TLS termination
  • OAuth2/OIDC authentication + identity header injection
  • Tenant selection (HMAC-signed, membership-checked cookie + hosted selector — see Multi-tenancy)
  • Manager-rendered vhost configuration with admin-API reload and drift detection — see Generated Artifacts
  • Service discovery (Nomad)
  • Load balancing
  • HTTP/2 support

Autoscaler ​

Nomadscaler Plugin

  • Integrates with Nomad Autoscaler
  • Queries Optimizer for predictions
  • Second-level caching

Optimizer Service

  • SARIMAX forecasting
  • MILP optimization
  • Stateless, horizontally scalable

Network Architecture ​

┌─────────── DMZ ───────────┐
│                           │
│  ┌──────────────────────┐ │
│  │  Proxy (Public)      │ │
│  │  Ports: 80, 443      │ │
│  └──────────┬───────────┘ │
└─────────────┼─────────────┘
              │
┌─────────────▼─────────────┐
│  Application Network      │
│  (Private)                │
│                           │
│  ┌──────────────────────┐ │
│  │  Manager API         │ │
│  │  Port: 8080          │ │
│  └──────────────────────┘ │
│                           │
│  ┌──────────────────────┐ │
│  │  Manager Executor    │ │
│  │  Metrics Port: 9090  │ │
│  └──────────────────────┘ │
│                           │
│  ┌──────────────────────┐ │
│  │  Optimizer           │ │
│  │  Port: 8015          │ │
│  └──────────────────────┘ │
└───────────────────────────┘
              │
┌─────────────▼─────────────┐
│  Data Network             │
│  (Private)                │
│                           │
│  ┌──────────────────────┐ │
│  │  PostgreSQL          │ │
│  │  Port: 5432          │ │
│  └──────────────────────┘ │
└───────────────────────────┘

Deployment Models ​

Single Node (Development) ​

┌─────────────────────────────┐
│  Single Server              │
│  ┌────────────────────────┐ │
│  │ Docker Compose         │ │
│  │ - Proxy                │ │
│  │ - Manager              │ │
│  │ - PostgreSQL           │ │
│  │ - Optimizer            │ │
│  └────────────────────────┘ │
└─────────────────────────────┘

Multi-Node (Production) ​

┌─────────────┐   ┌─────────────┐   ┌─────────────┐
│  Proxy      │   │  Proxy      │   │  Proxy      │
│  (LB)       │   │  (LB)       │   │  (LB)       │
└──────┬──────┘   └──────┬──────┘   └──────┬──────┘
       └─────────────────┼─────────────────┘
                         │
       ┌─────────────────┼─────────────────┐
       │                 │                 │
┌──────▼──────┐   ┌──────▼──────┐   ┌──────▼──────┐
│  Manager    │   │  Manager    │   │  Manager    │
│  API        │   │  API        │   │  API        │
└─────────────┘   └─────────────┘   └─────────────┘

┌─────────────┐
│  Manager    │
│  Executor   │
└──────┬──────┘
       │
┌──────▼──────────────────────┐
│  PostgreSQL Cluster         │
│  Primary + Replicas         │
└─────────────────────────────┘

Nomad Cluster ​

┌───────────────────────────────────────┐
│  Nomad Cluster                        │
│                                       │
│  ┌─────────────────────────────────┐  │
│  │  System Jobs                    │  │
│  │  - Proxy (on all nodes)         │  │
│  └─────────────────────────────────┘  │
│                                       │
│  ┌──────────────────────────────────┐ │
│  │  Service Jobs                    │ │
│  │  - Manager API (3 instances)     │ │
│  │  - Manager Executor (1 instance) │ │
│  │  - Optimizer (2 instances)       │ │
│  └──────────────────────────────────┘ │
│                                       │
│  ┌──────────────────────────────────┐ │
│  │  Application Jobs                │ │
│  │  - Rendered from deployment spec │ │
│  │    (pfy nomad render|deploy)     │ │
│  │  - Auto-scaled                   │ │
│  └──────────────────────────────────┘ │
└───────────────────────────────────────┘

Application Nomad jobs render from a deployment spec on the Manager's Deployment entity (pfy nomad render|deploy); the platform-core stack renders from a local install.yaml (pfy platform render|deploy); and pfy deploy diff flags any hand-written HCL as drift. See Generated Artifacts.

Tenant topology: shared and dedicated ​

The models above describe the platform stack. Orthogonal to them is how a given tenant's traffic reaches an application backend.

Shared (default). One backend per application serves every tenant. The proxy resolves the tenant per request from the caller's signed selection cookie, and onboarding a tenant is Manager rows only — no proxy reload, no backend redeploy, no DNS.

operations.apps.example.com ──► operations:8080   (all tenants; cookie selects)

Dedicated. A named tenant gets its own deployment, its own upstream and its own hostname per application. On that hostname the tenant is pinned in the rendered vhost — the cookie is not consulted at all, and membership is still enforced per request, failing closed.

operations.apps.example.com       ──► operations:8080        (shared tenants)
operations.acme.apps.example.com  ──► operations--acme:8080  (pinned to acme)

Both coexist in one install, on the same single proxy. Dedicating a tenant adds no job, no file in deployments/, and no install.yaml field — the per-tenant vhosts are rendered by the Manager and hot-pushed over the Caddy admin API exactly as shared per-app vhosts already are.

What it does add is DNS, and this is the part that is easy to miss: a wildcard matches exactly one label, so the *.apps.example.com record a shared install runs on does not cover operations.acme.apps.example.com. Each dedicated tenant needs either its own *.<tenant>.apps.<base> wildcard (one record, covers all of that tenant's apps) or an explicit record per app. pfy tenant update <slug> --deployment-mode dedicated returns the checklist.

See Shared vs. dedicated tenants for the decision criteria, the security properties of the pin, per-tenant pilet pins, and the explicit out-of-scope status of data migration between the two modes. A worked install variant lives in deployments/examples/dedicated-tenant/.

Scaling Strategy ​

Manager API ​

Horizontal Scaling:

  • Add instances based on load
  • Stateless, can scale freely
  • Load balanced by Proxy
  • Typical: 2-10 instances

Manager Executor ​

No Scaling:

  • Single instance only
  • Stateful (runs cron loop)
  • Database-level locking prevents duplicate execution if extra instances start
  • High availability via quick restart

Optimizer ​

Horizontal Scaling:

  • Stateless service
  • Can scale freely
  • Each instance independent
  • Typical: 2-5 instances

Database ​

Vertical + Replication:

  • Scale up primary for write performance
  • Add read replicas for read scaling

High Availability ​

Manager API ​

  • Multiple instances - 3+ instances across availability zones
  • Health checks - Proxy removes unhealthy instances
  • Graceful shutdown - Connection draining

Manager Executor ​

  • Fast restart - Restart on failure (< 30s)
  • No data loss - Trigger state in database
  • Missed executions - Catch up on restart

Database ​

  • Streaming replication - Primary + 2 standby replicas
  • Automatic failover - Via Patroni or similar
  • Point-in-time recovery - WAL archiving

Proxy ​

  • Multiple instances - DNS round-robin or L4 load balancer
  • Health checks - Remove failed instances
  • Certificate replication - Shared cert storage

Deployment Strategies ​

Blue-Green Deployment ​

Zero-downtime deployment by running two identical environments:

  1. Deploy green (new version) alongside blue (current)
  2. Test green in isolated environment
  3. Switch traffic from blue to green
  4. Keep blue running for quick rollback
  5. Decommission blue after verification

Canary Deployment ​

Gradual rollout to minimize risk:

  1. Deploy canary (new version) with small traffic percentage (5-10%)
  2. Monitor metrics (errors, latency, resource usage)
  3. Gradually increase canary traffic (25% → 50% → 100%)
  4. Full rollout if metrics are acceptable
  5. Rollback immediately if issues detected

Rolling Update ​

Update instances sequentially:

  1. Update instance 1, wait for health check
  2. Update instance 2, wait for health check
  3. Continue until all instances updated
  4. Automatic rollback on health check failure

Disaster Recovery ​

Backup Strategy ​

Backups ship as executable, restore-verified scripts in deployments/ops/backup/ — not prose. backup.sh dumps both databases (manager-db + Pocket ID) daily; restore-drill.sh restores each dump into a scratch instance and verifies it (see Docker Compose - Backup & Recovery).

Objectives:

MetricTargetBasis
RPO (max data loss)≤ 24h (daily dump); ≤ 5m with WAL archivingdump cadence / continuous WAL
RTO (max downtime)≤ 1hscratch-restore drill timings + redeploy

Drill cadence: run restore-drill.sh at least weekly (and in CI on every change to the backup scripts). A backup that has never been restored is not a backup.

Database:

  • Full backup daily (backup.sh, custom-format pg_dump)
  • WAL archiving continuous (for point-in-time recovery toward the 5m RPO)
  • Retain 30 days (RETENTION_DAYS, tunable)

Configuration:

  • Version control (Git) — rendered artifacts are committed with their spec hash
  • Manager database — holds tenants, settings, and deployment specs; part of the backup set
  • Environment variables and secrets

Certificates:

  • Backup Let's Encrypt data directory
  • Export certificates for DR

Recovery Procedures ​

Database Failure:

  1. Promote standby replica
  2. Update connection strings
  3. Restore from backup if needed

Complete Failure:

  1. Restore database from backup
  2. Deploy infrastructure from IaC
  3. Restore configuration
  4. Verify services

Security ​

Network Security ​

  • Firewall rules: Restrict access to internal components
  • VPC/Private network: Isolate backend services from internet
  • TLS everywhere: Encrypt all internal communication
  • Service mesh: Optional for microservice architectures

Secrets Management ​

  • Never commit secrets: Use environment variables or secrets managers (Vault, AWS Secrets Manager)
  • Rotate credentials: Regular password/token rotation (90 days)
  • Minimal privileges: Each component gets only needed permissions
  • Encrypted at rest: Sensitive data encrypted in database

Access Control ​

  • VPN required: Access to internal services via VPN only
  • SSH keys: No password authentication
  • Audit logs: Track all administrative actions
  • RBAC: Role-based access control in Manager

Monitoring ​

Metrics ​

  • Manager: Request rate, latency, error rate
  • Database: Connections, query time, replication lag
  • Proxy: Requests, TLS handshakes, upstream health
  • Optimizer: Prediction latency, forecast accuracy

Logging ​

Centralized log aggregation ships as concrete configuration, not intent. **Loki

  • Promtail** collect every component's stdout/stderr — Manager (pkg/logger structured JSON), proxy (Caddy access log + zap), optimizer (Python logging), and registered backends — labelled by component ({service="manager"}, {service="proxy"}, {service="optimizer"}, …) so logs are queryable per service. Loki was chosen over an ELK stack for footprint parity with the existing single-node Prometheus deployment.

Both deployment paths are covered by the configs under deployments/ops/logging/:

  • Compose: overlay compose.logging.yml (merge with the install bundle's docker-compose.yml) adds Loki + Promtail; Promtail discovers the Productify containers via the Docker socket.

  • Nomad: loki.nomad runs Loki as a service (filesystem storage, or S3/GCS on a multi-node cluster) with Promtail tailing each node's container logs.

  • Structured: JSON format (Manager); other components ship raw lines.

  • Retention: 30 days by default (limits_config.retention_period: 720h), tunable up to 90 days per policy.

Alerting ​

Alert rules ship as concrete Prometheus configuration in deployments/ops/alerting/rules.yml, written against the metrics the platform already exposes — proxy (:2112), Manager executor (:9090), and optimizer (:8017) — and unit-tested with promtool test rules rules_test.yml. The rules cover the intents below:

AlertSignalSeverity
ProxyDownup{job="proxy"} == 0critical
ManagerExecutorDownup{job="manager-executor"} == 0critical
OptimizerDownup{job="optimizer"} == 0warning
ProxyHighErrorRateactivator timeout rate (shed 503s)warning
ProxyHighLatencyp95 of pfy_response_time_seconds > 500mswarning
TriggerExecutionErrorspfy_executor_trigger_execution_errors_total risingwarning
TriggerQueueStalledwaiting > 0 while nothing processedcritical
CertificateExpiringSoonprobe_ssl_earliest_cert_expiry < 14d (blackbox exporter)warning
DiskSpaceLowfilesystem < 10% free (node_exporter)warning

Routing lives in deployments/ops/alerting/alertmanager.yml (a placeholder skeleton — critical pages, warnings go to chat; receivers filled from secrets at deploy time). Database replication lag, deep tracing, and SLO burn-rate alerting are deferred to future directions §7; this plan stops at alert rules on existing metrics.

See Also ​