Managed Data Store Backup & Restore

Backing up and restoring the postgres and redis stores the platform provisions for an application (pfy data): where dumps go, what retention deletes, how to take a backup out of schedule, and how to restore one. The platform's own databases, the Manager's and Pocket ID's, are covered in Docker Compose: Backup & Recovery.

The volume is never deleted automatically

Deleting a data store, through pfy data delete or the Manager UI, removes its job and the credentials generated for it. The underlying host volume is left untouched, always. The data stays on disk, and once the record is gone nothing in the platform describes what that volume held. Reclaiming it is a deliberate act by a human on the node.

The same rule governs restores: nothing in the restore path ever deletes a dump.

Model

A data store's backup policy is three fields on the store itself:

Field	Meaning
schedule	Cron expression, evaluated in UTC. Empty means backups are off.
target	The name of a backup target declared in install.yaml.
retention	How many of this store's dumps to keep. Unset (0) means the default, 7.

Postgres stores get a daily schedule (0 3 * * *, 03:00 UTC) seeded by the Manager when they are created without one, and a postgres store whose schedule has been cleared is reported as a warning everywhere the store is listed. Redis backups are opt-in. Set or change the policy with pfy data update <slug> --backup-schedule '<cron>' --backup-target <name>.

Applying the stack (pfy nomad deploy --stack) renders one periodic batch job per store that has a schedule, named <app>--<store>--backup, alongside the store itself. The backup job is applied last, after the store and the services.

target is a name, not a location

target is a bare name matching [a-z0-9][a-z0-9_-]{0,62}. It is not a path and not a URL, and it cannot contain :, @, /, ? or #.

That is deliberate and it is a security property, not a style choice. The value is stored by the Manager and copied verbatim into the before/after snapshot of every audit row for the store; the audit trail is long-lived and widely readable. A field that accepted s3://KEY:SECRET@host/bucket was a field that eventually contained one. A name has nowhere to put a secret.

Where the dumps land, and the credentials needed to put them there, are configured by the operator, in install.yaml, and never travel through the Manager.

Configuring backup targets

Targets live in install.yaml under backup_targets:, keyed by the name a store's target field refers to.

backup_targets:
  # kind: volume: dumps are written to a Nomad host volume.
  nightly:
    kind: volume
    volume: pfy-backups     # required: the Nomad HOST VOLUME name
    path: mytask            # optional: a subdirectory inside it

  # kind: s3: dumps are shipped to an S3-compatible object store.
  offsite:
    kind: s3
    endpoint: https://s3.eu-central-1.example.net   # required
    bucket: pfy-dumps                               # required
    region: eu-central-1
    prefix: mytask                                  # optional key prefix
    credential_var: nomad/jobs/pfy-backups          # required, see below
    # client_image: amazon/aws-cli:2.15.17          # optional pin override

A kind: volume target must name a volume. A Nomad task can only write to a host volume the client declared, so a bare path would not be actionable, and a target with no volume would render a job that writes into the allocation directory, which Nomad garbage-collects. The backups would appear to succeed and then quietly disappear.

Object-store credentials

credential_var is a Nomad variable path, not a key. Seed it once, out of band, with exactly two items:

nomad var put -in=json - <<'JSON'
{
  "Path": "nomad/jobs/pfy-backups",
  "Items": {
    "access_key_id": "…",
    "secret_access_key": "…"
  }
}
JSON

The backup job resolves them at run time through a template stanza, exactly as a store resolves its own password. The keys never appear in install.yaml, never reach the Manager, never appear in a rendered jobspec, and are never passed as process arguments.

credential_var is required for a kind: s3 target and has no default. Without it the job would fall back to whatever ambient credentials the node happened to carry, which is the one failure mode where backups silently write to somebody else's bucket.

An unknown name fails the apply

If a store names a target that install.yaml does not define, pfy nomad deploy --stack fails, naming the bad value and listing the configured targets. It does not fall back to a default. The alternative, rendering a job that writes somewhere undefined, produces a database whose dashboard says "backups: nightly" and whose dumps do not exist, which is discovered during a restore.

A store with a schedule and no target at all is a warning rather than a failure (that is the normal state between creating a store and declaring the operator's targets), but no backup job is rendered and nothing is being backed up. pfy data status says so explicitly.

Retention semantics

Retention is a count of dumps kept, per store, not an age.

• Pruning runs inside the backup job, immediately after a successful write. A failed dump prunes nothing.
• The retention newest dumps are kept; everything older is deleted.
• Only files matching this store's name pattern (<app>--<store>-YYYYmmddTHHMMSSZ.dump|.rdb) are considered, so several stores can share one host volume or one bucket prefix without one store's retention deleting another's dumps.
• Ordering is by file name, which is sound because the timestamp is fixed-width UTC: lexical order and chronological order are the same string. Copying the directory does not reorder anything, which sorting by mtime would.
• retention: 0 means unset and falls back to 7. It never means "keep nothing": a job rendered that way would delete the dump it had just written, on every run, and still exit 0.

Retention deletes dumps. It never touches a store, a volume, or a database.

Checking state

pfy data status --app mytask --env prod

Reports, per store: the derived instance and its job name, the job's run state, the backup policy, the backup job's name and run state, and any server-side warnings. A store with no schedule is reported as backups OFF (no schedule) rather than being left blank; the difference between "no backups" and "could not tell" is the point of looking. --app takes the application slug or ID; --env is an environment alias from project.yaml; add --tenant <slug> for a dedicated tenant's stack.

Taking a backup now

pfy data backup-now main --app mytask --env prod

This forces the registered periodic job to run immediately, out of schedule. It is deliberately not a separate one-off job: an on-demand backup must be the same job the schedule runs, or the one that gets exercised by hand is not the one that runs at 03:00.

If the store has no schedule, there is no backup job and the command says so. If the stack has never been applied in this environment, apply it first.

Restoring

A restore is the most destructive operation pfy performs

It replaces the current contents of the store, and it takes every dependent service down while it does. The data currently in the store is not backed up by the restore command. Take a backup-now first if the current contents matter.

pfy data restore main --app mytask --env prod \
  --from mytask--main-20260101T030000Z.dump

--from is a bare file name as it appears in the store's backup target, not a path. A value containing a separator would let the restore read (postgres) or write (redis, straight onto the store's live data volume) an arbitrary path in the allocation, which is a much larger authority than "pick which of my dumps to restore".

The command prints what it is about to do, then requires the store's slug to be typed back. --yes skips the prompt; under --ci or without a TTY, --yes is required. Nothing is stopped until the confirmation has been given; a refusal that arrived after the dependents were down would be an outage for nothing.

What it does, in order

1. Stops the dependent services. These are the services that receive the store's injected connection URL. A service left running across a pg_restore --clean holds open handles to the objects being dropped, so the restore fails partway and the service serves a half-dropped schema in the meantime.

   Their jobspecs are captured from the cluster before stopping and re-submitted verbatim afterwards. They are not re-rendered: re-rendering would substitute whatever --tag and --region this invocation happened to carry, so a restore run without --tag would bring the services back on a different image, a deploy disguised as a restore.

2. Stops the store itself, redis only. A redis restore puts an RDB file onto the store's own data volume, and a running redis-server rewrites that file from memory when it shuts down; a restore performed with redis up is silently undone by its next restart. Postgres is restored over the wire and stays up.

3. Runs the restore job (<app>--<store>--restore) and waits for it to finish. nomad job run returns once the allocation is placed, not once it has finished; restarting the dependents at that point would bring them up against a half-restored database.

   • postgres: pg_restore --clean --if-exists --no-owner --single-transaction. The drop-and-recreate makes the restore a replacement rather than a merge that half-succeeds on every conflicting row.
   • redis: the RDB file is copied onto the data volume, and any append-only file is removed; an AOF takes precedence over the RDB at start-up and would make the restore a no-op that reported success.

4. Starts everything back, store first, then the dependent services in their original apply order. A consumer brought back before the store reconnects to nothing and crash-loops.

If the restore itself fails, the dependents are still started back (they are down because this command took them down) and the failure is reported. Treat the store as being in an unknown state in that case: --clean may have dropped objects before the failure.

Services that opted out

A service with inject_data_env = false never received the store's connection URL and is not stopped. If such a service reaches the store through a hand-set environment variable, stop it yourself for the duration of the restore.

Timeout

--wait-timeout (default 30m) bounds the wait on the restore job. The dependents stay stopped until it finishes, so a short timeout does not shorten the outage; it only makes pfy stop watching. If it expires, the restore job is still running: follow it with nomad job status <app>--<store>--restore and do not restart the dependents until it has finished.

Failure modes worth knowing

Symptom	Cause
Apply fails: unknown backup target "x"	The store names a target install.yaml does not declare. The message lists the configured names.
Apply warns: schedule but no target	The store was configured for backups before any target existed. No backup job is rendered.
pg_dump version error in the backup job	Should not happen: the backup job runs the store's own image, so client and server versions are equal by construction. If it does, the store's version was changed without re-applying the stack.
Redis backup succeeds but the dump is stale	Should not happen: the job waits on LASTSAVE to change before copying dump.rdb. A BGSAVE that does not complete within 300s fails the job instead.
Restore reports success, old data comes back	Redis restored while the store was running. pfy data restore stops it; a hand-run restore job does not.
Dumps stop appearing after N days	Working as configured; check retention.

Related

• Managed Data Dependencies: the model these procedures operate on: declaration, instances, the credentials model, injection.
• Anatomy of a Stack: how the backup job is applied alongside the store and its services.
• Incident Runbook: platform-level alert response.
• Configuration Reference: the rest of install.yaml.