Backup and Disaster Recovery

ObjectOS itself is stateless — the runtime can be rebuilt from the container image. Backups are about the data ObjectOS depends on, not ObjectOS itself. Plan recovery around those datasets. Everything on this page holds for Community and Enterprise alike.

What to back up

A database backup without OS_SECRET_KEY is an incomplete backup. Every stored secret — encrypted settings, secret fields, datasource credentials — is encrypted under that one key. A database restored without the original key keeps those values, unreadable, for good. Escrow the key the day you generate it, and treat key and database as one backup unit.

A container that sets OS_DATABASE_URL and OS_SECRET_KEY needs no backup of its own filesystem: its caches are rebuildable — a cached artifact is re-verified against its integrity pin on every read, so losing the cache costs a fetch, never correctness. A single host running on the defaults is different: its home directory holds the SQLite database, the uploaded files and the self-minted secret key, so include it in your file backups.

Database backups

Match the strategy to the driver:

Whatever the driver, validate that:

• backups complete within the customer's RPO;
• a fresh database can boot ObjectOS from the same artifact and serve traffic;
• restore tests are exercised end-to-end at least once a quarter.

SQLite deployments

SQLite is the default driver and is a legitimate production choice for single-node ObjectOS — desktop apps, internal tools for a small team, edge / on-prem appliances, evaluation environments. The trade-off is structural: SQLite is single-writer, so it does not fit multi-node ObjectOS, high concurrent write throughput, or shared-database deployments. For those, use PostgreSQL.

When you do run SQLite in production, the configuration that makes it safe and the backup recipe go together:

Runtime configuration

• The runtime keeps file-backed SQLite in WAL mode (readers don't block the writer; crash-safe). That means app.db-wal beside the database can hold committed writes that are not in app.db yet — never copy app.db on its own while the runtime runs.
• PRAGMA synchronous=NORMAL is the usual desktop/single-node default — safe with WAL, much faster than FULL.
• Keep the database file on local disk. Do not put it on NFS, SMB, or inside a folder synced by Dropbox / OneDrive / iCloud while the runtime is running — SQLite's locking is unreliable on those and this is the most common way SQLite deployments corrupt.

Hot snapshot (no downtime)

The runtime keeps serving traffic while you take a snapshot using SQLite's Online Backup API or VACUUM INTO:

VACUUM INTO '/var/backups/objectos/db-2026-05-27T13-00Z.sqlite';

Schedule it (cron, systemd timer, or the in-process scheduler) at a cadence matching your RPO. Tag each snapshot with the artifact version that produced it so you can roll database + artifact back together (see Artifact versioning below).

Offsite copy

A local snapshot doesn't survive disk loss. Push each snapshot to durable storage:

• Server / VM: push to S3 or any S3-compatible bucket (you likely already have one configured for the storage capability).
• Desktop app: write the snapshot to a user-controlled sync folder (OneDrive / iCloud / Google Drive) — acceptable here because the snapshot file is closed and immutable, unlike the live database.

Continuous replication (lower RPO)

For sub-minute RPO without giving up SQLite, run Litestream alongside the ObjectOS process. It streams the WAL to S3-compatible storage and supports point-in-time restore. This is the recommended path when a single-node SQLite deployment needs near-zero data loss.

Restore

Stop the runtime, replace the database file with the chosen snapshot (or run litestream restore), make sure no stale -wal or -shm file is left beside it, and start the runtime against the same artifact version that produced the snapshot.

Artifact versioning

Treat published artifacts as immutable. Tag each artifact with the release id used to compile it (e.g. objectstack-2026-05-24.json). Recovery to a known-good business state usually means:

1. Restore the database to the chosen point in time.
2. Re-point ObjectOS at the artifact version that was live at that time.
3. Restart the runtime.

If you overwrite artifacts in place, you lose the ability to roll back cleanly even when the database backup is perfect.

RPO/RTO planning

A workable starting target for customer deployments:

Tighter targets require platform changes that are outside the ObjectOS container itself (HA database, multi-AZ ingress, warm replicas).

Failure modes worth rehearsing

• Database unavailable for a long window. Since ObjectStack 17.0 this is contractual rather than something to hope for: /api/v1/ready answers 503 while a data driver is not answering, so readiness probes mark the pods unready. Rehearse the recovery — that the replicas rejoin once the database returns, and that nothing needed a restart to notice. A datasource that objects bind to and that cannot connect at startup now fails the boot, so verify your restore completes before the application replicas roll.
• Artifact regression. Roll back the artifact pointer; data is unaffected.
• Secret rotation. Rotating OS_AUTH_SECRET invalidates every session. Run during a maintenance window or stagger across replicas.
• Region outage. If the customer requires region failover, the business database, secret manager, and ingress all need to be cross-region. ObjectOS itself can run anywhere the image is available.

The restore drill

Rehearse this on a scratch host before go-live, and again after any storage change. A backup you have not restored is a hope, not a backup.

1. Provision a clean host with the same image version the backup was taken under.

2. Restore the database — the provider snapshot, or:

   pg_restore --clean --if-exists -d "$OS_DATABASE_URL" backup-2026-07-14.dump

3. Provide the original secrets. OS_SECRET_KEY and OS_AUTH_SECRET are the escrowed originals, never freshly generated values. Restore the uploaded files too, if you use local storage.

4. Boot against the same artifact version, and verify:

   curl -fsS https://app.example.com/api/v1/health
   curl -fsS https://app.example.com/api/v1/ready

   Then prove the restore end to end: sign in with a real account, open a record that existed before the backup, and read a value from an encrypted setting — that is what exercises OS_SECRET_KEY.

What to capture in the runbook

• Backup schedule, retention, and on-call contact for each dataset.
• Step-by-step restore procedure (database first, then artifact, then start ObjectOS).
• Verification queries to confirm the restored database is consistent.
• Rollback plan for both ObjectOS image and artifact version (see Upgrade and Rollback).
• Communication template for customer-visible incidents.

Enterprise adds

• Keep the previous image digest with the backup. A restore to a point in time is a restore to the digest that was running then, and on an air-gapped site the previous digest may be the only copy there is — see Air-gapped.
• OS_LICENSE_KEY belongs in the secrets baseline. A restored deployment boots with the license it is given, and a multi-organization posture refuses to start without one.
• Restore before the replicas roll. The bundled stack runs its one-shot migration before any replica starts, so the database must be complete when the stack comes up.