Monitoring and Logging

The Productify Proxy provides comprehensive monitoring and logging capabilities through Prometheus metrics and structured logging.

Access Logging

Access logging provides detailed information about every request processed by the proxy.

Configuration

Enable access logging in your Caddyfile:

{
	productify {
		manager http://172.17.0.1:8080
		token supersecrettoken
		enable_access_log
	}
}

Logged Information

Each request generates a structured log entry containing:

Field	Description	Example
source_ip	IP address of the request origin	192.168.1.100:54321
user	Authenticated user (from X-Token-Subject header)	user@example.com or anonymous
user_agent	Browser or client information	Mozilla/5.0...
target_system	Application ID being accessed	app-123
method	HTTP method	GET, POST, etc.
path	Requested URL path	/api/users
timestamp	Request timestamp	2025-12-02T10:30:00Z

Log Format

Logs are output in structured JSON format using Zap logger:

{
  "level": "info",
  "ts": 1701512400.123456,
  "msg": "access",
  "source_ip": "192.168.1.100:54321",
  "user": "user@example.com",
  "user_agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36",
  "target_system": "app-123",
  "method": "GET",
  "path": "/api/users",
  "timestamp": "2025-12-02T10:30:00Z"
}

Prometheus Metrics

Metrics are exposed on port 2112 at the /metrics endpoint.

Endpoint

http://localhost:2112/metrics

Available Metrics

Authentication Metrics

pfy_authentication_awating_users (Gauge)

• Number of users currently on the authentication page
• Labels: app (application ID)

# Example: Current users awaiting authentication
pfy_authentication_awating_users{app="app-123"}

Request Metrics

pfy_total_requests (Counter)

• Total number of requests processed
• Labels: app (application ID), user (authenticated user), method (HTTP method)
• Requires: enable_metrics

# Example: Total requests for an application
sum(pfy_total_requests{app="app-123"})

# Example: Requests per user
sum by (user) (pfy_total_requests{app="app-123"})

pfy_response_time_seconds (Histogram)

• Response time distribution in seconds
• Labels: app, user, method
• Requires: enable_metrics

# Example: Average response time
rate(pfy_response_time_seconds_sum{app="app-123"}[5m]) /
rate(pfy_response_time_seconds_count{app="app-123"}[5m])

# Example: 95th percentile response time
histogram_quantile(0.95, sum(rate(pfy_response_time_seconds_bucket[5m])) by (le))

Size Metrics

pfy_request_size_bytes (Histogram)

• Request body size distribution in bytes
• Labels: app, user, method
• Requires: enable_size_metrics
• Buckets: 100, 1000, 10000, 100000, 1000000, 10000000, 100000000, 1000000000

# Example: Average request size
rate(pfy_request_size_bytes_sum{app="app-123"}[5m]) /
rate(pfy_request_size_bytes_count{app="app-123"}[5m])

pfy_response_size_bytes (Histogram)

• Response body size distribution in bytes
• Labels: app, user, method
• Requires: enable_size_metrics
• Buckets: 100, 1000, 10000, 100000, 1000000, 10000000, 100000000, 1000000000

# Example: Total bandwidth sent
sum(rate(pfy_response_size_bytes_sum[5m]))

Configuration Options

Complete Configuration

{
	productify {
		manager http://172.17.0.1:8080
		token supersecrettoken
		enable_access_log      # Enable detailed access logging
		enable_metrics         # Enable request count and response time metrics
		enable_size_metrics    # Enable request/response size metrics
	}
}

Option Details

Option	Description	Default
enable_access_log	Enables structured access logging	false
enable_metrics	Enables request count and response time metrics	false
enable_size_metrics	Enables request/response size metrics	false

Note: enable_size_metrics requires enable_metrics to be enabled.

Integration Examples

Prometheus Configuration

scrape_configs:
  - job_name: "productify-proxy"
    static_configs:
      - targets: ["proxy:2112"]

Grafana Dashboard

Example queries for a Grafana dashboard:

Request Rate Panel:

sum(rate(pfy_total_requests[5m])) by (app)

Response Time Panel:

histogram_quantile(0.95, sum(rate(pfy_response_time_seconds_bucket[5m])) by (le, app))

Bandwidth Panel:

sum(rate(pfy_response_size_bytes_sum[5m])) by (app)

Active Users Panel:

pfy_authentication_awating_users

Log Aggregation

Example Fluentd configuration for log collection:

<source>
  @type tail
  path /var/log/caddy/access.log
  pos_file /var/log/caddy/access.log.pos
  tag productify.access
  <parse>
    @type json
    time_key timestamp
    time_format %Y-%m-%dT%H:%M:%S%z
  </parse>
</source>

<match productify.access>
  @type elasticsearch
  host elasticsearch
  port 9200
  logstash_format true
  logstash_prefix productify
</match>

Performance Considerations

Overhead

• Access logging: Minimal overhead (~1-2% CPU)
• Request metrics: Low overhead (~2-3% CPU)
• Size metrics: Moderate overhead (~5-10% CPU) due to response buffering

Recommendations

1. Production environments: Enable enable_access_log and enable_metrics for visibility
2. High-throughput systems: Enable enable_size_metrics only when bandwidth monitoring is critical
3. Development environments: Enable all options for comprehensive debugging

Optimization

To reduce overhead while maintaining visibility:

{
	productify {
		manager http://172.17.0.1:8080
		token supersecrettoken
		enable_access_log    # Low overhead, high value
		enable_metrics       # Low overhead, essential metrics
		# enable_size_metrics  # Disable in high-throughput scenarios
	}
}

Troubleshooting

No Logs Appearing

1. Verify enable_access_log is set in Caddyfile
2. Check Caddy log level: {"logging": {"logs": {"default": {"level": "INFO"}}}}
3. Ensure the proxy is processing requests

Metrics Not Available

1. Verify metrics endpoint is accessible: curl http://localhost:2112/metrics
2. Check enable_metrics is set in Caddyfile
3. Verify Prometheus can scrape the endpoint

Missing User Information

The user field is populated from the X-Token-Subject header, which is injected by the authorization policy. Ensure:

authorization policy pocketpolicy {
	set auth url /auth/oauth2/generic
	allow roles user
	validate bearer header
	inject headers with claims  # This injects X-Token-Subject
}

Security Considerations

Log Sanitization

Access logs may contain sensitive information:

• User identifiers
• Request paths with query parameters
• IP addresses

Ensure proper log retention and access controls are in place.

Metrics Exposure

The metrics endpoint exposes application usage patterns. Consider:

• Restricting access to port 2112
• Using network policies to limit access
• Implementing authentication on the metrics endpoint

Next Steps

• Set up Prometheus
• Create Grafana dashboards
• Configure log aggregation