11 — Platform and Cloud¶

ONYX — Online Storage System — Platform. This document specifies the platform layer that sits on top of the core appliance (docs/design/01–10): how the system presents itself as a managed, cloud-connected infrastructure platform — SSO identity, edge reverse proxy with wildcard TLS, Dockerized deployment, virtualization, container management, object storage with hybrid cloud, AI-assisted administration, and the automated release pipeline.

It is the spec behind setup.sh, docker-compose.yml, scripts/npm-proxy-hosts.py, scripts/provision-authentik.sh, .github/workflows/, and the platform daemons (onyx-snapd, onyx-backupd, onyx-vmm, onyx-appd, onyx-ai, onyx-objectstore).

1. Platform identity¶

Product name: ONYX — Online Storage System — Platform
Repository slug: onyx-oss-platform
Primary domain: onyx.innotel.us

1.1 Service subdomains¶

The console is served at the primary domain itself; every other public surface is a subdomain of it. All of them are routed by Nginx Proxy Manager (NPM), behind one wildcard TLS certificate that also names the apex:

Host	Service	Container	Notes
`onyx.innotel.us`	Web console (Unity theme)	`onyx-web`	static frontend; talks to `api` + `auth`
`api.onyx.innotel.us`	HTTP gateway	`onyx-api`	`/api/v1/*`, `/healthz`; also serves `admin` routes
`auth.onyx.innotel.us`	Identity / SSO	`authentik-server`	OIDC provider for every other surface
`storage.onyx.innotel.us`	S3-compatible object storage	`onyx-objectstore`	`GET/PUT/DELETE` buckets+objects; hybrid-cloud tiering
`backup.onyx.innotel.us`	Backup service	`onyx-backupd`	JSON API: jobs, schedules, restore, intelligence
`admin.onyx.innotel.us`	Admin console	`onyx-api` (admin routes)	separate hostname, same gateway; access via Authentik

The mapping is data, not code: setup.sh writes it into the environment (NPM_SUBDOMAINS) and scripts/npm-proxy-hosts.py turns it into NPM proxy hosts.

2. Identity: Authentik¶

Authentik is the platform IdP (docs/design/08 moves to it as the primary provider; local accounts remain as a fallback for offline operation).

Deployed as containers: postgres (state), redis (cache), authentik-server, authentik-worker (see §5).
Bootstrap: AUTHENTIK_BOOTSTRAP_TOKEN + AUTHENTIK_BOOTSTRAP_EMAIL seed the first superuser; scripts/provision-authentik.sh waits for readiness, then creates the ONYX Platform OAuth2 provider and application:
authorization/redirect: https://onyx.innotel.us/*, https://admin.onyx.innotel.us/*, loopback for local dev;
client id/secret are written back into .env (AUTHENTIK_CLIENT_ID/ AUTHENTIK_CLIENT_SECRET) for downstream services.
Every host trusts auth.onyx.innotel.us; NPM protects the admin host with Authentik's forward-auth where enabled. The retired app host is a 301 to the console, so links written before the move keep working.
Passkeys: the authentication flow carries a WebAuthn stage in authentication mode, so enrolled devices confirm sign-in with a platform authenticator (docs/design/11 §10).

2.1 User sync with the Onyx console¶

Identity lives in Authentik; Onyx stores only the role and the per-share grants. The Users page presents one row per person by joining the two, so the two stores cannot drift apart:

GET /api/v1/users reads Authentik's accounts (AUTHENTIK_URL + AUTHENTIK_TOKEN, falling back to AUTHENTIK_BOOTSTRAP_TOKEN) and merges them with the local mappings. An Authentik account with no mapping gets one on first sight (role user, status from is_active) — that is the automatic mapping, and it is persisted, so the next read is stable rather than a second wave of writes. Each row reports source (onyx / authentik / authentik+onyx), in_authentik and the provider's own active.
POST /api/v1/users creates the account in Authentik first and stores the mapping only afterwards: a mapping for somebody who cannot sign in would be a lie the console tells. No password is ever set through the API — Authentik's recovery flow and passkeys are the only way one is minted.
DELETE /api/v1/users/{id} removes the Onyx mapping and disables the Authentik account, so its history and group memberships survive; ?purge=true is the deliberate erase-everywhere path.
When AUTHENTIK_URL/token are unset the page says so instead of quietly showing a local-only list (authentik.reason in the response).

3. Edge: Nginx Proxy Manager¶

NPM is the only ingress. Everything else binds loopback inside the compose network (127.0.0.1 on the host for the native install, per docs/design/07).

3.1 API provisioning (`scripts/npm-proxy-hosts.py`)¶

setup.sh integrates this script (the requirement "npm-proxy-hosts.py into setup.sh"): it is the single source of truth for the NPM state.

Login — POST /api/tokens with NPM_EMAIL/NPM_PASSWORD (env).
Wildcard certificate — POST /api/nginx/certificates requesting a Let's Encrypt certificate for ["*.onyx.innotel.us", "onyx.innotel.us"] with meta.dns_challenge = true, provider rfc2136 (RFC 2136 dynamic DNS update), carrying the TSIG key:
DNS_NAMESERVER (e.g. ns.innotel.us:53)
TSIG_KEY_NAME, TSIG_KEY_SECRET, TSIG_KEY_ALGORITHM (e.g. hmac-sha256)
Idempotent: if a cert for the wildcard already exists (matched by domain), it is reused/renewed rather than duplicated.
Proxy hosts — for each NPM_SUBDOMAINS entry, GET /api/nginx/proxy-hosts → if absent, POST /api/nginx/proxy-hosts with the forward target from the compose network (container DNS name + port); if present, update it (PUT). WebSocket-enabled for app/api/admin.
Report — prints the final URL table.

Auth to the NPM API is HTTP Basic; all requests over NPM_BASE_URL (default http://127.0.0.1:81).

3.2 Wildcard certificates¶

Issuer: Let's Encrypt, via NPM.
Challenge: DNS-01 using the rfc2136 provider — certbot performs nsupdate against the authoritative server using the TSIG key, adding a _acme-challenge TXT record, then removing it after validation. This is the same TSIG pattern used by the other innotelinc platform projects.
Because the challenge is DNS-based, no inbound port 80/443 is needed to issue the certificate — only to serve traffic.

4. Dockerized deployment¶

Every daemon has a Dockerfile under docker/ and ships in docker-compose.yml. Two runtimes coexist:

Native appliance (bare metal / OSTree image): systemd units in deploy/systemd/, installed by scripts/onyx-install — unchanged core.
Containerized platform (this document): docker compose up -d from setup.sh, all services on an internal network, ingress via NPM only.

The Go daemons use a shared multi-stage pattern (builder golang:1.27 → scratch/alpine runtime, static binaries); the Rust daemons build with rust:1-slim and copy the static binary out. Images are published to GHCR:

ghcr.io/innotelinc/onyx-oss-platform/onyx-core
ghcr.io/innotelinc/onyx-oss-platform/onyx-api
ghcr.io/innotelinc/onyx-oss-platform/onyx-shared
ghcr.io/innotelinc/onyx-oss-platform/onyx-storaged
ghcr.io/innotelinc/onyx-oss-platform/onyx-privd
ghcr.io/innotelinc/onyx-oss-platform/onyx-snapd
ghcr.io/innotelinc/onyx-oss-platform/onyx-backupd
ghcr.io/innotelinc/onyx-oss-platform/onyx-vmm
ghcr.io/innotelinc/onyx-oss-platform/onyx-appd
ghcr.io/innotelinc/onyx-oss-platform/onyx-ai
ghcr.io/innotelinc/onyx-oss-platform/onyx-objectstore

Within the compose network, gRPC over unix sockets is replaced by gRPC over the shared network (same contracts in proto/); state volumes map to /var/lib/onyx/<service> and the data pool mounts into the storaged/privd containers.

5. Stack inventory (compose)¶

Container	Image	Purpose
`onyx-privd`	ghcr.io/innotelinc/onyx-oss-platform/onyx-privd	root privilege helper (host-mapped, `--privileged`-free: only needed capabilities)
`onyx-storaged`	ghcr.io/.../onyx-storaged	Btrfs discovery, hotplug, SMART
`onyx-shared`	ghcr.io/.../onyx-shared	share config rendering
`onyx-core`	ghcr.io/.../onyx-core	control plane orchestrator
`onyx-api`	ghcr.io/.../onyx-api	HTTP gateway (exposed only to NPM)
`onyx-snapd`	ghcr.io/.../onyx-snapd	snapshots
`onyx-backupd`	ghcr.io/.../onyx-backupd	backups + intelligence
`onyx-vmm`	ghcr.io/.../onyx-vmm	virtualization
`onyx-appd`	ghcr.io/.../onyx-appd	container/app management
`onyx-ai`	ghcr.io/.../onyx-ai	AI Storage Advisor
`onyx-objectstore`	ghcr.io/.../onyx-objectstore	S3-compatible storage
`onyx-web`	ghcr.io/.../onyx-web	static SPA (v0.2)
`postgres`	postgres:16	Authentik state
`redis`	redis:7	Authentik cache/queue
`authentik-server`	ghcr.io/goauthentik/server	SSO
`authentik-worker`	ghcr.io/goauthentik/server	background jobs
`nginx-proxy-manager`	jc21/nginx-proxy-manager	ingress (ports 80/443/81)

6. Platform daemons¶

All of these are gRPC services generated from proto/onyx/v1/, following the service conventions of docs/design/04. The v0.4 "Jade" surfaces (§6.3, §6.4, §6.6) are implemented; onyx-ai (§6.5) is the v0.5 milestone, and its inputs are already assembled by onyx-api.

6.1 `onyx-snapd` — snapshots¶

Btrfs snapshot lifecycle on top of the fixed subvolume layout (@data/@apps/@backups/@snapshots, docs/design/05):

CreateSnapshot (subvolume, name, optional readonly), ListSnapshots, DeleteSnapshot, RollbackSnapshot (snapshot → subvolume, requires unmounted target).
Scheduling hooks for onyx-backupd (snapshot-before-backup).

6.2 `onyx-backupd` — backups + Backup Intelligence¶

Jobs: source (share/snapshot/pool) → target (local, NFS, remote ONYX, S3-compatible cloud via onyx-objectstore); schedules (cron); retention policies; RestoreBackup.
Backup Intelligence: GetBackupReport aggregates job history, success/failure trends, data-change velocity per source, and recovery-time/recovery-point estimates. Heuristics run locally; onyx-ai can enrich the same report with natural-language findings.

6.3 `onyx-vmm` — virtualization¶

VM inventory (ListVMs), lifecycle (CreateVM, StartVM, StopVM, DeleteVM), resources (vCPU/RAM/disk), and media attachment.
Implemented on libvirt/QEMU: inventory and install records persist in SQLite, disk images are qcow2 files on the pool (/mnt/onyx/<pool>/@apps/vms), and the domains are defined and driven through virsh — never CGo bindings, so the service runs unprivileged in the libvirt group. Creation defines a machine stopped; deletion only removes the disk image when delete_disk is set. The lifecycle rule is enforced against the hypervisor's live state, not the recorded one, so a guest that shut itself down is still startable and a still-running machine cannot be deleted.

6.4 `onyx-appd` — container management¶

App catalog (ListApps, InstallApp, UninstallApp), container lifecycle (StartContainer, StopContainer, RestartContainer), and compose manifests from the signed app store (docs/design/09).
Implemented with persistent install records (SQLite) and a container runtime seam: the Docker API drives real containers where a socket is available, and the same policy is testable without one. Installing renders the app's manifest with the per-installation config the caller supplied (ports, paths), so two installations of one app do not collide. Uninstall stops the containers; purge_data deletes the app's volumes and force is required to remove an app whose containers are still running.
Deployment: the composed stack grants onyx-appd the host's Docker socket (and only that group) and binds the pool at the same path the host uses, so a manifest's bind mount resolves to pool storage on both sides. App projects are namespaced onyx-app-<app> so an app can never take the platform stack down with a down.
The app sandbox (docs/design/09 §6) is enforced here: a catalog manifest that requests host privilege, host namespaces, an added-back capability, no resource limits, or a bind mount outside the storage root is refused at startup, and an install whose path setting points outside the pool is refused with InvalidArgument before anything is written.

6.5 `onyx-ai` — AI Storage Advisor + Backup Intelligence¶

AnalyzeStorage: pool/device telemetry (from onyx-storaged) → recommendations (free-space runway, snapshot cadence, scrub health, expansion guidance). Deterministic heuristics are computed in-process and are always the answer unless a model improves on them.
The AI plane, as ONYX consumes it (§4.5 of the build-plane convergence plan): ONYX does not own inference. The platform runs ONE OmniRoute (Group 2, on the gateway host 192.168.1.46, reached at :20128 through the SSO proxy in front of it — its own :20128 answers on that host's loopback and bridge alone) and onyx-ai gets OMNIROUTE_BASE_URL + OMNIROUTE_API_KEY — a gateway key, never an upstream provider key. The key may be a vault:// reference, resolved at startup like the S3 credentials; a reference that cannot be resolved stops the service. The documented BYO-key hook (AI_PROVIDER, AI_API_KEY, AI_MODEL — local or remote) stays as the escape hatch; the gateway wins when both are set.
AnalyzeBackups: consumes onyx-backupd's report (6.2) and produces findings + priorities.
Privacy: no telemetry leaves the box unless a model plane is configured, and then only the findings already computed — severity, code, summary, detail, action — are sent, with the local sentence as the fallback whenever the call fails (docs/design/07 §12).

6.6 `onyx-objectstore` — object storage + hybrid cloud¶

S3-compatible API surface (storage.onyx.innotel.us): ListBuckets, PutObject, GetObject, DeleteObject, DeleteBucket — served over HTTPS by NPM; static credentials from env (S3_ACCESS_KEY/S3_SECRET_KEY). Both are required: with no access key the endpoint refuses every request (403 AccessDenied) instead of serving anonymous access, because that port is published for the ingress. --s3-allow-anonymous is the explicit opt-out for a development run on loopback.
Bucket storage on the pool (Btrfs subvolume per bucket → snapshots/scrub come free).
Hybrid cloud: per-bucket lifecycle policies — LOCAL, CLOUD (primary in an external S3 provider), or TIERED (local hot tier, cloud cold tier, with sync/eviction rules). The cloud side is pluggable (AWS S3, Backblaze B2, other S3-compatible endpoints).
The transport is rclone against the shared remote catalog (/etc/rclone/rclone.conf, the same file the Shares page writes and onyx-backupd reads), so a provider is configured once for tiering, clones and backups. HYBRID_CLOUD_ENDPOINT/credentials remain the direct-endpoint form of the same thing for a deployment that does not use the catalog.
A bucket's cloud_target is a remote (b2-archive: or b2-archive:offsite), or an absolute directory on the pool for testing tiering without a cloud account; each bucket is stored under onyx-objectstore/<bucket> inside it. A bare name with no : is refused rather than read as a relative directory.
Per-tier semantics: CLOUD acknowledges a write only once the cloud holds it (and keeps no local copy); TIERED acknowledges from the hot copy and mirrors on sync. SyncBucket uploads, then runs rclone check over the whole bucket and only evicts when that verification passes, releasing local copies older than evict_after_days. A read of an evicted object is served from the cloud and refetched into the cache, so eviction never removes access.

7. Release pipeline¶

.github/workflows/ implements CI/CD:

ci.yml — on push/PR: make bootstrap (cached), make check (vet + tests), make build, plus an e2e job that builds the stack and runs the flows below against it. Validation only; nothing is published.

make check exercises the code; scripts/e2e-stack.sh (make e2e) exercises the deployment — it drives a running compose stack through the three flows that span every layer (pool creation on a real disk with POOL_DEVICE=, or on a loop device backed by a file the harness creates and removes with POOL_IMAGE=, an app install, and a tiered bucket's sync → verify → evict → refetch over the S3 API) and asserts what only a real stack can show: a pool root the unprivileged daemons can write, containers that are running rather than crash-looping, an evicted object that comes back on read and is cached again with no download left behind. A unit test cannot see a mount namespace, a file mode or a missing helper binary; this can, and both bugs it was written for — an un-clearable disk and a storage root nobody could write into — were invisible to the test suite. The e2e job is the same script, so a regression reaches CI instead of an operator's disk. - release.yml — one publish pipeline for both event kinds: - main push → build + publish :latest images; - tag v* push → build + publish :latest + :<tag> images, then create the GitHub Release with tarball artifacts; - workflow_dispatch → manual re-publish (latest).

The workflow is concurrency-guarded (group keyed by commit SHA, cancel-in-progress), so a tag pushed on a commit that just triggered the main latest build cancels the duplicate instead of racing it on the first-run compile and the GHCR push. Buildx uses the gha cache, so subsequent builds reuse compiled layers instead of compiling cold.

The release job (artifacts) depends on the image job, so the GitHub Release only appears after every image is published.

8. Operational flow (setup.sh)¶

./setup.sh
  1. validate: docker + compose present, env loaded
  2. cp .env.example .env if missing (then require editing secrets)
  3. docker compose up -d --build     (onyx stack + Authentik + NPM)
  4. wait for NPM API + Authentik readiness
  5. scripts/provision-authentik.sh    (bootstrap superuser, ONYX OIDC app, passkey stage)
  6. device trust — per DEVICE_TRUST:
       off      → nothing to do (passkeys still on)
       local    → scripts/provision-device-trust.sh (ensure the CA)
       cerulean → POST the mTLS gate request to the Cerulean API
  7. NPM provisioning — per DEVICE_TRUST/NPM_MODE:
       local edge  → scripts/npm-proxy-hosts.py (TSIG wildcard + hosts + mTLS gate)
       cerulean edge → skipped; the edge is Cerulean-managed, cert gates are
                       requested in step 6
  8. print URL table (console-at-apex/api/auth/storage/backup/admin)

Idempotent end to end: re-runs converge (cert reuse, proxy-host update, provider re-check).

9. Security posture¶

Single ingress (NPM) on 80/443; everything else loopback-only inside the compose network (extends docs/design/07 to the containerized deployment).
Wildcard cert keys live in NPM's data volume; TSIG secret only in .env (never committed — .env is git-ignored, .env.example has placeholders).
onyx-privd remains the only privilege boundary; containers run as the same unprivileged per-service users where possible.
Authentik is the SSO choke point: forward-auth on app/admin, OIDC for the API, service-to-service traffic stays on the internal network.
Device trust is opt-in (DEVICE_TRUST=off by default): when enabled, passkeys confirm identity in Authentik while mTLS client certificates gate the edge — local CA + local NPM for standalone deploys, or the Cerulean control plane (dashboard issuance, remote NPM) for managed fleets (docs/design/11 §10).

10. Device trust (passkeys + mTLS client certificates)¶

Management surfaces (app, admin) can be gated by two factors that bind a login to a device, not just a password. Device trust is optional and selected by one switch, DEVICE_TRUST=off|local|cerulean (default off):

Passkeys (WebAuthn) — Authentik is the IdP, so platform authenticators (Touch ID, Windows Hello, Android/iOS, hardware keys) are enabled at the source: the default default-authentication-identification flow gets a WebAuthn stage in authentication mode, so every login is confirmed by the device's passkey once one is registered. Users register passkeys from Authentik's settings; provision-authentik.sh flips the flow stage on automatically.
TLS client certificates (mTLS) — the edge requires a client certificate from a trusted device CA before the request even reaches Authentik's proxy. A device without a trusted certificate cannot start an OIDC flow, which removes credential-stuffing and stolen-password classes of attack for the admin surface entirely. Who runs the CA and where the gate is applied depends on DEVICE_TRUST:

Mode	Device CA	Certificate issuance	mTLS enforcement	For
`off` (default)	—	—	—	devices stay optional
`local`	ONYX host (`/etc/onyx/pki`, `provision-device-trust.sh`)	CLI helper → `.p12` for MDM import	local NPM (`ssl_verify_client`)	standalone deploys
`cerulean`	Cerulean (remote MDM/control plane)	Cerulean dashboard (approve, issue, revoke)	requested from Cerulean via its API	fleets behind Cerulean

The result: on an enrolled device, sign-in is one passkey tap. On any other device, the TLS handshake fails before a password is ever asked for.

10.1 `local` mode — device CA (`scripts/provision-device-trust.sh`)¶

A minimal internal PKI maintained under /etc/onyx/pki (host bind mount, ONYX_PKI_DIR to override; private keys are mode 0600, never leave the host):

ca.crt / ca.key          ECDSA P-256 self-signed device CA (10 years)
serial                   openssl tracking file
issued/<name>.crt/.key   per-device client certs (default 365 days)

scripts/provision-device-trust.sh (no args) creates the CA if missing — idempotent, called from setup.sh only in local mode.
scripts/provision-device-trust.sh issue laptop [days] issues (or renews) a client certificate with CN <name> plus the ONYX Device Trust / device=<name> O/OU identifiers, for distribution via your MDM (Apple Configurator / .mobileconfig, Intune, Fleet, or a manual import into the OS/browser trust store).

10.2 `local` mode — edge enforcement (`npm-proxy-hosts.py`)¶

For every subdomain listed in DEVICE_TRUST_SUBDOMAINS (default app admin), the proxy host's advanced config is extended with the Nginx mTLS directives — merged idempotently with any existing advanced config:

ssl_client_certificate /etc/letsencrypt/onyx-device-ca/ca.crt;
ssl_verify_client on;
ssl_verify_depth 2;

The CA is mounted into the NPM container from the host PKI dir (docker-compose.yml); ssl_verify_client on makes NPM return 400 with No required SSL certificate was sent for devices without an enrolled certificate — before any auth flow starts. Nginx passes the verified subject to upstreams in X-SSL-Client-Subject-DN, which Authentik forward auth can use for audit trail enrichment.

10.3 `cerulean` mode — remote control plane¶

When the platform is managed by Cerulean (the innotelinc fleet control plane; also owns the remote NPM and the open-source MDM underneath), ONYX owns policy, Cerulean owns enforcement:

Issuance lives in the Cerulean dashboard: an operator approves a device, Cerulean's MDM pushes the client certificate, and the device CA pool is managed entirely remotely — ONYX never stores CA keys in this mode (provision-device-trust.sh is not run).
ONYX requests the gate by calling Cerulean's device-trust API (CERULEAN_API_URL + CERULEAN_API_TOKEN in .env):

POST {CERULEAN_API_URL}/api/v1/fleet/{FLEET_ID}/device-trust
Authorization: Bearer ${CERULEAN_API_TOKEN}
{
  "domain": "onyx.innotel.us",
  "subdomains": ["app", "admin"],      # = DEVICE_TRUST_SUBDOMAINS
  "enforcement": "mtls",
  "client_ca": "cerulean-managed"
}

Cerulean applies the equivalent ssl_verify_client configuration on its remote NPM (or drops the mTLS requirement on revoke) and reports the enrolled device list back to the dashboard. - setup.sh skips local NPM provisioning — with a remote edge there is no local NPM to configure (§8). - Complement, not replacement: passkeys remain Authentik-side in every mode; Cerulean enforces the edge gate, Authentik enforces the identity gate, and a device needs both factors to reach app/admin.

The Cerulean contract above is an interface, not a client: this repo ships the request schema and the setup.sh hook, while the server side (and the open-source MDM it drives — e.g. Fleet, MicroMDM, or Headwind) belongs to Cerulean.

Dashboard read path: onyx-api exposes GET /api/v1/devices/trust so the ONYX dashboard can show enrolled devices without talking to Cerulean itself. Mode-aware:

`DEVICE_TRUST`	Response
`off`	`200 {"mode":"off","enabled":false,"devices":[]}`
`local`	`200` with `enabled:true` and a note that issuance is local (`scripts/provision-device-trust.sh`)
`cerulean`	proxy of `GET {CERULEAN_API_URL}/api/v1/fleet/{FLEET_ID}/devices` (query params pass through, bearer auth added server-side); upstream failures map to `502 device_trust_upstream` (retryable), missing config to `503 device_trust_not_configured`

Projection per device: name, common_name, status, issued_at, expires_at — plus fleet_id and count at the top level. Issuance and revocation stay dashboard-side; the endpoint is read-only.

10.4 Trade-offs and operations¶

Strictness: ssl_verify_client on was chosen over optional so the gate is real; enrollment is a deliberate act (local: issue cert → install on device; cerulean: approve in dashboard → MDM pushes).
Revocation: local — delete the issued cert or rotate the CA and re-provision; cerulean — revoke from the dashboard, the edge drops the device on the next sync.
Failure mode: local CA loss is recoverable by regenerating and re-enrolling the (small) fleet; cerulean mode has no local key material to lose.
S3/backup surfaces are exempt: machine clients (S3 keys, backup agents) authenticate with credentials, not device certificates; only interactive surfaces are device-gated.