Monitoring and Logging

The Productify Proxy writes one structured access-log line per application request and exposes Prometheus metrics, liveness and readiness probes, and a configuration-drift endpoint on a separate operational port (:2112 by default).

Access logging

Enable it

{
	productify {
		manager http://172.17.0.1:8080
		token {env.PFY_MANAGER_TOKEN}
		enable_access_log
	}
}

The line is written by the productify route directive (and therefore by productify_protect), so only routes that carry it are logged. Routes without it (for example the Manager's own vhost) rely on Caddy's normal log directive.

What is logged

Field	Description	Example
logger	Always productify	productify
msg	Always access	access
source_ip	Remote address of the connection, including the port	192.168.1.100:54321
user	The X-Token-Subject header, or anonymous when absent	user@example.com, anonymous
user_agent	User-Agent request header	Mozilla/5.0 ...
target_system	Application ID from the directive	64e163af-2f6b-...
method	HTTP method	GET
path	Request path as the productify handler sees it: percent-decoded, without the query string, before any later uri rewrite on the route	/api/users
timestamp	Time the line was written, Unix seconds as a float	1788632170.81

ts and level are added by Caddy's logger. The query string is not logged.

Log format

Caddy emits JSON. A real line looks like this:

{
  "level": "info",
  "ts": 1788632170.8138702,
  "logger": "productify",
  "msg": "access",
  "source_ip": "127.0.0.1:41602",
  "user": "alice@example.com",
  "user_agent": "curl/8.5.0",
  "target_system": "123",
  "method": "POST",
  "path": "/api/users",
  "timestamp": 1788632170.813866
}

By default the line goes to Caddy's default log output (stderr). To write it to a file, or to change the time format, configure Caddy's global log option; see Caddyfile Reference.

Operational port

The productify app starts one HTTP server on metrics_address (default :2112, which binds every interface) with four paths:

Path	Purpose
/metrics	Prometheus exposition
/healthz	Liveness: always 200 ok while the process runs
/readyz	Readiness: 200 ready or 503 (see below)
/config-hash	{"config_hash":"<hash>"} of the configuration currently loaded

None of these paths is authenticated, and /readyz makes one outbound Manager call per hit. Bind the port to an internal address in production:

{
	productify {
		manager http://172.17.0.1:8080
		token {env.PFY_MANAGER_TOKEN}
		metrics_address {$PFY_METRICS_LISTEN:127.0.0.1:2112}
	}
}

If the address cannot be bound, the proxy logs metrics server failed to bind; continuing without metrics and keeps serving traffic.

/healthz and /readyz

• /healthz returns 200 whenever the process is up. It does not look at the Manager. Use it as the liveness probe: a failure means "restart me".
• /readyz returns 200 ready only when both hold: the productify app has started, and GET <manager>/health (the manager URL from the config, with the shared secret attached) answers a 2xx within 3 seconds. Otherwise it returns 503 with not ready: app not started or not ready: manager unreachable. The detailed error, including the Manager URL, goes to the log only.
• The Manager answers that probe from a real handler on its API port, reusing its own /readyz verdict, so not ready: manager unreachable means the Manager itself reports it cannot serve, or that it could not be reached at all. Because the proxy attaches the shared secret, its request gets the full readiness body, which names the failing subsystem; an anonymous caller to the same path gets only ok or unavailable.
• When the Manager pushes a new configuration, Caddy starts the new module instance before stopping the old one. Both probes and /config-hash consult the instance that is live now, so a reload does not flip /readyz to 503 and /config-hash never reports the configuration that was just replaced.

/config-hash and drift detection

The Manager stamps config_hash <hash> into the productify block of every configuration it renders. The proxy never computes a hash; it reports the stamped value back on /config-hash and as the metric pfy_proxy_config_info{config_hash="<hash>"} 1. A configuration without a stamp reports 200 {"config_hash":""}, which is a valid state, not an error. Exactly one pfy_proxy_config_info series exists at a time; a reload replaces the label value.

The Manager reads this endpoint when proxy_metrics_url is configured and compares it with the hash it last rendered; pfy proxy status and pfy proxy diff show the result. Without that URL the Manager can only compare its own render with its own push, which cannot detect a proxy that was changed out of band or restarted from an old file. See Proxy Vhost Sync Loop.

Prometheus metrics

Endpoint: http://<proxy>:2112/metrics.

Request metrics (enable_metrics)

pfy_total_requests (counter): requests that passed a productify directive. Labels: app, method.

pfy_response_time_seconds (histogram): time from the productify handler to the end of the response. Labels: app, method. Buckets: Prometheus defaults (0.005 to 10 s).

# Request rate per application
sum by (app) (rate(pfy_total_requests[5m]))

# 95th percentile response time per application
histogram_quantile(0.95, sum by (le, app) (rate(pfy_response_time_seconds_bucket[5m])))

Size metrics (enable_metrics and enable_size_metrics)

pfy_request_size_bytes (histogram): request Content-Length, recorded only when it is greater than zero. pfy_response_size_bytes (histogram): bytes written to the response body, recorded only when greater than zero. Labels: app, method. Buckets: 100, 1000, 10000, ... 1e9 (powers of ten).

# Bandwidth sent per application
sum by (app) (rate(pfy_response_size_bytes_sum[5m]))

Size metrics wrap the response writer, which is the only one of these options with measurable overhead. The wrapper still supports WebSocket hijacking and flushing for streamed responses.

Authentication metric (always on)

pfy_authentication_awaiting_users (gauge): users currently in the login flow. Labels: app. Incremented by productify_before_auth for every request on the /auth/* route and decremented on the authorization-code-callback. It does not depend on enable_metrics. Treat it as approximate: an abandoned login never decrements, and a restart can drive it negative. Alert on trends, not absolute values.

Scale-from-zero metrics (always registered)

Metric	Type	Labels	Meaning
pfy_activator_wakes_total	counter	app, env, result	Wake attempts; result is woken, already_up, ineligible, unknown or error
pfy_activator_wait_seconds	histogram	app, env	How long an API request was held during a cold start
pfy_activator_buffer_depth	gauge	app, env	API requests currently held
pfy_activator_timeouts_total	counter	app, env	Held requests that timed out and were shed with 503
pfy_activator_pending_requests	gauge	app, env	Pending cold-start demand, exported for the autoscaler optimizer as an interactive demand series. It keeps reporting while the deployment is at zero replicas, since the proxy stays up. It is not what wakes an app; see Known Issues. Today it tracks the same held-request count as pfy_activator_buffer_depth

Configuration metric (always registered)

pfy_proxy_config_info (gauge): always 1; label config_hash carries the hash of the running configuration (empty when unstamped). Alert with pfy_proxy_config_info{config_hash="<expected>"} != 1.

Cardinality

Labels are limited to app, method, env and the fixed result set. There is deliberately no per-user label; per-user attribution belongs in the access log.

Options

{
	productify {
		manager http://172.17.0.1:8080
		token {env.PFY_MANAGER_TOKEN}
		metrics_address 127.0.0.1:2112   # default ":2112" binds every interface
		enable_access_log                # one JSON line per request
		enable_metrics                   # pfy_total_requests, pfy_response_time_seconds
		enable_size_metrics              # pfy_request_size_bytes, pfy_response_size_bytes
	}
}

Option	Default	Description
metrics_address	:2112	Listen address for /metrics, /healthz, /readyz, /config-hash
enable_access_log	off	Structured access log
enable_metrics	off	Request count and response time
enable_size_metrics	off	Request and response sizes; requires enable_metrics

Recommended: enable_access_log and enable_metrics everywhere; enable_size_metrics where bandwidth per application matters.

Integration

Prometheus

scrape_configs:
  - job_name: "productify-proxy"
    static_configs:
      - targets: ["proxy:2112"]

Grafana panel queries

# Request rate
sum by (app) (rate(pfy_total_requests[5m]))

# 95th percentile latency
histogram_quantile(0.95, sum by (le, app) (rate(pfy_response_time_seconds_bucket[5m])))

# Bandwidth
sum by (app) (rate(pfy_response_size_bytes_sum[5m]))

# Users on the login page
pfy_authentication_awaiting_users

# Configuration drift (1 = the expected config is running)
pfy_proxy_config_info{config_hash="<hash from pfy proxy status>"}

Kubernetes or Nomad probes

Use /healthz for liveness and /readyz for readiness, both on the metrics port. Do not use /readyz for liveness: a Manager outage would restart every proxy.

Log collection

Write Caddy's log to a file with the global log option (see Caddyfile Reference) and tail it. Filter on "logger":"productify" and "msg":"access". The timestamp field is Unix seconds as a float; if your collector needs a string time, use Caddy's log { format json { time_format ... } } to change ts instead. Example for Fluentd:

<source>
  @type tail
  path /var/log/caddy/access.log
  pos_file /var/log/caddy/access.log.pos
  tag productify.access
  <parse>
    @type json
    time_key ts
    time_type float
  </parse>
</source>

<filter productify.access>
  @type grep
  <regexp>
    key msg
    pattern /^access$/
  </regexp>
</filter>

<match productify.access>
  @type elasticsearch
  host elasticsearch
  port 9200
  logstash_format true
  logstash_prefix productify
</match>

Troubleshooting

Symptom	Check
No access-log lines	enable_access_log is set; the route carries productify or productify_protect; Caddy's log level is INFO or lower (the default)
/metrics connection refused	metrics_address is what you expect; the log does not say metrics server failed to bind
pfy_total_requests missing	enable_metrics is set and at least one request has passed a productify directive
user is anonymous for logged-in users	The route runs authorize with <policy> before productify, and the policy has inject headers with claims (that is what sets X-Token-Subject)
/readyz is 503 not ready: manager unreachable	The proxy log has the full error with the Manager URL; check network reachability and the manager option

Security notes

• The access log contains user identifiers, paths and IP addresses. Apply retention and access controls.
• The operational port has no authentication. Bind it to an internal interface with metrics_address, or firewall it. /readyz triggers a Manager call per request; an exposed port is a cheap way to load the Manager.
• The configuration hash is not a secret and grants nothing, but it does reveal when the configuration changed.

Next steps

• Application Routing: what the metrics and headers mean per route
• Proxy Vhost Sync Loop: drift detection from the Manager side
• Prometheus getting started
• Grafana dashboards