Cluster: single k3s node, TransIP VPS 136.144.174.219 (ssh sarkoutmahmoud@, key ~/.ssh/id_ed25519), staging only, memory-tight (~4 GB), storage local-path only.

Durable fix tracked in project-template#2 (pin the k3s node name so this can't recur). Until that ships, this runbook is the recovery.

SYMPTOM

After the VPS reboots, the k3s node re-registers under a different name. Every local-path PV was provisioned with a nodeAffinity pinned to the old node name, so all PVs are now stranded on a node that no longer exists. Result: every PVC-backed pod (all Postgres, all Redis, anything with a volume) is unschedulable — 0/1 nodes are available: … node(s) had volume node affinity conflict. Backends that depend on those datastores CrashLoop or stay Pending.

DIAGNOSIS

Get the live node name:
```
kubectl get nodes -o name
```

Get the node name baked into each PV's affinity:

kubectl get pv -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.nodeAffinity.required.nodeSelectorTerms[0].matchExpressions[0].values[0]}{"\n"}{end}'

If the PV column ≠ the live node name, the PVs are stranded → this runbook applies. Confirm pods are stuck:

kubectl get pods -A | grep -E 'Pending|CrashLoop'
kubectl describe pod -n <ns> <pod> | grep -i 'volume node affinity'

nodeAffinity on a bound PV is immutable, so you cannot edit the name in place — the PVC/PV must be recreated.

RECOVERY USED

Staging data is throwaway, so we recreate the stranded volumes. The node is memory-tight, so bring stacks up one at a time, never all at once.

Quiesce everything — scale every *-staging workload to 0 so nothing fights for the node or the PVs during recovery:

for ns in $(kubectl get ns -o name | grep -- '-staging' | cut -d/ -f2); do
  kubectl scale deploy,statefulset -n "$ns" --all --replicas=0
done

Per namespace: delete + recreate the stranded PVCs. Their stranded PVs are released; local-path reprovisions fresh ones pinned to the live node. Do this only with data you're willing to lose (staging).
```
kubectl delete pvc -n <ns> --all      # datastore re-inits empty on next start
```
Staggered, ONE-AT-A-TIME bring-up, watching node CPU/memory between each:
- Scale up the datastore(s) for one project, wait for Running + a fresh PVC bound to the live node.
- Because the Postgres PVC was wiped, the two-user CREATEROLE init must be re-run (the runtime superuser grant lives in the PVC and was lost). After Postgres is Running:
```
kubectl exec -n <ns> <pg-pod> -- bash -c \
  'PGPASSWORD="$(cat $POSTGRES_POSTGRES_PASSWORD_FILE)" psql -U postgres -d <db> -c "ALTER ROLE \"<flyway_user>\" WITH CREATEROLE;"'
```
- Scale the backend to 1 so Flyway re-runs against the empty schema and creates the app role from ${appPassword}. Wait for 1/1 and /actuator/health = UP.
- Only then move to the next project. Watch the node so you don't OOM it:
```
kubectl top node ; kubectl describe node | grep -A6 'Allocated resources'
```
- If the node won't fit everything, leave lower-priority stacks at 0 and report NEED-CAPACITY rather than evicting another project.

DURABLE FIX

Pin the k3s node name so a reboot can't re-register it under a new name and strand the PVs. Tracked in project-template#2. Until merged, re-run this runbook after any VPS reboot.

Pre-flight checklist (next time)

[ ] kubectl get nodes — note the live name.
[ ] PV affinity vs live name (diagnosis step 2) — confirm the mismatch before deleting anything.
[ ] Confirm the affected data is staging/throwaway before delete pvc.
[ ] Quiesce all *-staging to 0 first.
[ ] Bring up one stack at a time; re-run the CREATEROLE grant per Postgres; watch node capacity.

Reacties

Nog geen reacties

Athena — runbooks/node-reboot-pv-recovery.md

Runbook: node reboot → local-path PV recovery