Kubernetes Setup

Prerequisites

kubectl installed locally
SSH access to lcmd-app.epfl.ch
Connected to EPFL network (or VPN)

kubectl Access

The recommended way to get kubectl access is now Tailscale — no SSH access, EPFL VPN, or token handling needed. The setups below remain the break-glass path.

Automated Setup

cd infrastructure/ansible
ansible-playbook playbooks/setup_kubectl.yml

Manual Setup

# 1. SSH tunnel (keep open)
ssh -L 6443:localhost:6443 root@lcmd-app.epfl.ch

# 2. Configure kubectl
kubectl config set-cluster lcmd-cluster --server=https://localhost:6443
kubectl config set-credentials lcmd-cluster --token=<node-token>
kubectl config set-context lcmd-cluster --cluster=lcmd-cluster --user=lcmd-cluster
kubectl config use-context lcmd-cluster

Get the token from the server:

ssh root@lcmd-app.epfl.ch "cat /var/lib/rancher/k3s/server/node-token"

We use SOPS + age for secret encryption. Secrets are stored encrypted in Git and decrypted by ArgoCD at deploy time. The prod overlay keeps one *.enc.yaml file per Secret (backend-secrets.enc.yaml, seaweedfs-secrets.enc.yaml, ...) — see Object storage → Editing the storage secrets for the editing rules (always via sops, never as text).

Decrypting Secrets

# Set up age key (get from 1Password)
mkdir -p ~/.config/sops/age
echo "AGE-SECRET-KEY-..." > ~/.config/sops/age/keys.txt
chmod 600 ~/.config/sops/age/keys.txt

# Decrypt and edit (one file per Secret)
cd infrastructure/kubernetes/app/overlays/prod
SOPS_AGE_KEY_FILE=~/.config/sops/age/keys.txt sops backend-secrets.enc.yaml

Adding New Secrets

cd infrastructure/kubernetes/app/overlays/prod

# 1. Add the new file name to the creation rule in .sops.yaml, then create it
#    (sops opens an editor and encrypts on save):
sops my-new-secrets.enc.yaml

# 2. Register it under `files:` in secrets-generator.yaml so KSOPS renders it

# 3. Commit — encrypted file only, no plaintext ever touches the repo
git add .sops.yaml my-new-secrets.enc.yaml secrets-generator.yaml

Never edit *.enc.yaml files as text — sops ≥ 3.13 fails decryption on a stale MAC. The sops-decrypt-check pre-commit hook catches this when an age key is present.

Recreating the Cluster

Provision the server

cd infrastructure/ansible
ansible-playbook playbooks/setup_server.yml
ansible-playbook playbooks/setup_k3s.yml

Configure storage

Point local-path provisioner at /data:

kubectl apply -k infrastructure/kubernetes/storage/
kubectl rollout restart deployment/local-path-provisioner -n kube-system

Install core components

# Sealed Secrets controller
kubectl apply -k infrastructure/kubernetes/sealed-secrets/

# ArgoCD + Image Updater + Applications
kubectl create namespace argocd
kubectl apply -k infrastructure/kubernetes/argocd/

Configure the ArgoCD SOPS key

kubectl create secret generic sops-age-key \
  --namespace argocd \
  --from-literal=keys.txt="AGE-SECRET-KEY-..."
kubectl rollout restart deployment/argocd-repo-server -n argocd

Verify

kubectl -n argocd get applications
kubectl -n prod get pods

# ArgoCD UI
kubectl port-forward svc/argocd-server -n argocd 8080:443
# → https://localhost:8080

# Admin password
kubectl -n argocd get secret argocd-initial-admin-secret \
  -o jsonpath="{.data.password}" | base64 -d

Resource Allocation

Optimized for 4-core, 8GB RAM VPS:

Component	CPU Req	CPU Limit	Mem Req	Mem Limit
Backend	200m	1000m	512Mi	1Gi
Web app	10m	250m	24Mi	128Mi
PostgreSQL	100m	500m	256Mi	1Gi
SeaweedFS	100m	1000m	512Mi	1536Mi
Docs	100m	200m	128Mi	256Mi

Troubleshooting

# Check pod status
kubectl -n prod get pods
kubectl -n prod describe pod <pod-name>
kubectl -n prod logs <pod-name>

# Check ArgoCD sync status
kubectl -n argocd get applications
kubectl -n argocd describe application lcmd-app

# Verify secrets decryption
kubectl -n prod get secrets