Architecture

Production object storage is SeaweedFS running as a single weed mini process — master, volume server, filer, and S3 gateway in one pod. The S3 API listens on seaweedfs:8333 inside the cluster only; there is no Ingress route to it.

containers:
  - name: seaweedfs
    # Statically pinned; NOT tracked by ArgoCD Image Updater (app images only).
    image: chrislusf/seaweedfs:4.31
    command: [weed]
    args: [mini, -dir=/data]

The Django backend is the only consumer. It reads S3_ENDPOINT_URL, S3_ACCESS_KEY, and S3_SECRET_KEY from the backend-secrets Secret (mapped to the AWS_* django-storages settings in lcmd_db/config/settings/base.py). Two buckets:

Both are created idempotently by manage.py initialize_buckets (existing buckets are skipped), which also runs at the end of the migrate Job on every sync — a fresh store bootstraps itself.

File bytes are always streamed through the Django API — e.g. GET /api/v1/molecules/<id>/xyz_file/ returns a FileResponse read from the bucket. The storage endpoint itself is never exposed to clients.

For the local-development SeaweedFS (docker-compose, same weed mini setup), see Architecture → SeaweedFS storage.

Cutover runbook

The PR that carries this page is the cutover: merging it flips S3_ENDPOINT_URL in backend-secrets from MinIO to http://seaweedfs:8333. SeaweedFS starts empty — the data is re-imported from git LFS rather than copied over.

kubectl -n prod exec postgres-0 -- \
  sh -c 'pg_dump -U "$POSTGRES_USER" -Fc "$POSTGRES_DB"' \
  > pre-seaweedfs-$(date +%Y%m%d).dump

kubectl -n prod get secret backend-secrets \
  -o jsonpath='{.data.S3_ENDPOINT_URL}' | base64 -d
# → http://seaweedfs:8333

kubectl -n prod rollout restart deployment/backend
kubectl -n prod rollout status deployment/backend

kubectl -n prod exec deployment/backend -- python manage.py initialize_buckets

cd apps/backend
uv run manage.py launch_import_job --subset OSCARDHBD --reload   # canary
uv run manage.py launch_import_job --all --reload                # full reimport

kubectl -n prod exec -it deployment/backend -- python manage.py shell

from lcmd_db.apps.molecules.models import Molecule
from lcmd_db.constants.storage import BUCKETS
from lcmd_db.core.models.storage import Storage

storage = Storage(bucket=BUCKETS.molecules)
print(Molecule.objects.count())  # matches the pre-cutover count
for m in Molecule.objects.order_by("?")[:20]:
    assert storage.exists(m.xyz_file.name), m.pk

# streams the file bytes through Django
curl -fsS https://lcmd-app.epfl.ch/api/v1/molecules/<id>/xyz_file/ | head -2

# detail API still renders xyz_file as a URL string
curl -fsS https://lcmd-app.epfl.ch/api/v1/molecules/<id>/ | jq .xyz_file

Rollback

Before step 5 (no reimport yet) — revert the cutover PR; ArgoCD restores the MinIO values in backend-secrets; kubectl -n prod rollout restart deployment/backend. MinIO still holds all the data, nothing else to do.

After step 5 — revert + restart is not enough: the --reload reimport deleted and re-created the DB rows, and their files exist only in SeaweedFS. The reverted backend would point at MinIO objects keyed by rows that no longer exist. Either:

• restore the step-1 dump, which returns the database to the state matching MinIO's objects (this is what the write freeze protects — anything written after the dump is lost):

  kubectl -n prod exec -i postgres-0 -- \
    sh -c 'pg_restore -U "$POSTGRES_USER" -d "$POSTGRES_DB" --clean --if-exists' \
    < pre-seaweedfs-YYYYMMDD.dump

• or re-run the full reimport (launch_import_job --all --reload) against MinIO after the revert.

Soak checklist

Run daily for ~10 days after the cutover:

Only after a quiet soak, merge the MinIO-removal PR.

Editing the storage secrets

The prod overlay keeps one sops file per Secret in infrastructure/kubernetes/app/overlays/prod/ (backend-secrets.enc.yaml, seaweedfs-secrets.enc.yaml, ...), each listed in secrets-generator.yaml for KSOPS. Hard-won rules:

• Always edit through sops (sops infrastructure/kubernetes/app/overlays/prod/backend-secrets.enc.yaml), never as text. sops ≥ 3.13 — the version pinned in the devcontainer — hard-fails decryption when the MAC is stale. A 2026-06 incident (deleting a document from the then-multidoc secrets file as text) broke MAC verification for every secret in the repo; the per-Secret file split exists so one bad edit can never invalidate the others.
• The pre-commit hook sops-decrypt-check (scripts/check-sops-mac.sh) decrypt-verifies every changed *.enc.yaml wherever an age key is available, and skips cleanly where it isn't (e.g. CI).
• The age key lives at ~/.config/sops/age/keys.txt; in the devcontainer it persists in the lcmd-db-sops-age volume.

See Kubernetes access → Secrets management for obtaining the key and general sops usage.

Known gaps and hardening

Handover notes — read before assuming any safety net exists:

• Object storage is reproducible: every file can be regenerated from git LFS with launch_import_job --all --reload. Losing the SeaweedFS PVC costs a reimport, not data.
• PostgreSQL is not: user accounts, subset metadata, and anything created through the app exist only in the database, protected solely by manual dumps (like the cutover's step 1).
• Both stores sit on single local-path PVCs on the same node — one disk failure takes both.
• Recommended first hardening: a pg_dump CronJob with copies shipped off the node.
• Key escrow before a maintainer departs: add a second age recipient to .sops.yaml, run sops updatekeys on each *.enc.yaml, and store the private key in the lab's password manager. Until then, exactly one keys.txt (plus the in-cluster copy for ArgoCD) can decrypt prod secrets.