Tight operator reference for the single shared staging cluster. For getting one backend green see ../../.swarm/RECIPE.md; for the reboot-recovery procedure see ../runbooks/node-reboot-pv-recovery.md.

The node

Single-node k3s on a TransIP VPS, staging only. IP 136.144.174.219, ssh sarkoutmahmoud@, key ~/.ssh/id_ed25519, Debian 13 (trixie). k3s v1.34.x.
~4 GB RAM, and CPU is the real bottleneck during churn, not memory. All datastore + backend pods restarting at once spikes CPU and starves the scheduler/kubelet.
Storage: local-path only (no distributed/networked SV). No Prometheus CRD (no ServiceMonitor). k3s enforces NetworkPolicies.

Bring stacks up STAGGERED — one at a time

Two outages on 2026-05-22 were caused by simultaneous cold-start storms. When many stacks (or the whole fleet after a reboot) start together, the combined cold-start CPU burst exceeds what the 4 GB node can serve, the kubelet goes unresponsive, and pods cascade into Pending/CrashLoop.

After any reboot or fleet restart, quiesce everything to 0, then scale up ONE project at a time, waiting for Running + 1/1 + /actuator/health = UP before moving on (procedure in the reboot runbook).

Between each, watch the node:

kubectl top node
kubectl describe node | grep -A6 'Allocated resources'

If it won't all fit, leave lower-priority stacks at 0 and report NEED-CAPACITY — never evict another project to make room.

Staging memory / CPU budget

Keep lean resource requests in every values-staging.yaml (replicaCount: 1, autoscaling off). With lean requests all ~6 stacks fit at roughly ~88% node memory — so there is little headroom; a fat request or an unneeded sidecar can push the node over.
Drop bitnami metrics.enabled exporter sidecars (no Prometheus CRD here) — each pg/redis exporter wastes ~128Mi for nothing.

Storage: local-path PV nodeAffinity gotcha + node-name pin

Every local-path PV is provisioned with a nodeAffinity pinned to the node name that existed at provisioning time. nodeAffinity on a bound PV is immutable.
On VPS reboot the k3s node can re-register under a different name (seen live: the old cloud.example.com node went NotReady and a fresh cloud node took over). Every PV is now stranded on a node that no longer exists → all PVC-backed pods (Postgres, Redis, anything with a volume) become unschedulable with volume node affinity conflict.
Recovery (staging data is throwaway): delete + recreate the stranded PVCs so local-path reprovisions them on the live node — full steps in ../runbooks/node-reboot-pv-recovery.md.
Durable fix: pin the k3s node name so a reboot can't rename it — tracked in mahmoudholding/project-template#2 (OPEN). Until it ships, re-run the runbook after every reboot.

Postgres (bitnami pg18): initdb runs as NON-superuser → runtime CREATEROLE

The two-user pattern (app DML user + Flyway/admin user) needs the Flyway user to hold CREATEROLE so its migration can create the app role.
⚠️ On bitnami postgresql 18.x the .sql initdb scripts in primary.initdb.scripts run as the NON-superuser app user, so ALTER ROLE … WITH CREATEROLE there silently fails at init (confirmed on claimio, mahmoud, auditPic — the older ".sql runs as superuser" claim is FALSE for pg18).
Reliable fix = runtime superuser grant, once Postgres is up:
```
kubectl exec -n <ns> <pg-pod> -- bash -c \
  'PGPASSWORD="$(cat $POSTGRES_POSTGRES_PASSWORD_FILE)" \
   psql -U postgres -d <db> -c "ALTER ROLE \"<flyway_user>\" WITH CREATEROLE;"'
```
then restart the backend so Flyway retries. The grant persists in the PVC (survives restarts) — but is lost on a PVC wipe / fresh init, so re-run it after any local-path reprovision (e.g. the reboot recovery above).
Durable alternative: pre-provision the app role, or use a .sh initdb that connects as postgres over the local socket (no password var).

TLS / cert-manager

Single ClusterIssuer/letsencrypt-prod issues certs for the *.staging.glorylabs.nl hosts. Not a single wildcard cert — it's one cert per host, per namespace (e.g. audit-pic-staging-tls, claimio-staging-tls, europe-login-staging-tls, recruitment-platform-staging-tls, valideerleeftijd-staging-tls, plus developer-glorylabs-nl-tls); all currently READY=True.
This supersedes the old "no DNS/TLS for *.staging.glorylabs.nl yet" note — staging hosts are now HTTPS-reachable. Inspect with:
```
kubectl get certificate,clusterissuer -A
```

Reacties

Nog geen reacties

Athena — ops/cluster.md

k3s cluster / ops reference (staging)