Status: Draft proposal. Nothing in this document has been applied.
Author: swarm worker adhoc-guardrail2 · Date: 2026-05-24
Goal (owner directive): the VPS staging node must never exceed ~90 % utilization
— keep ≥10 % memory + CPU headroom at all times — so the 2026-05-24 saturation
outage cannot recur.
⚠️ This is analysis + design only. Parts (2) and (3) need owner approval before any implementation. Part (1) is low-risk and can be staged via normal PRs once approved.
Single staging node (cloud), TransIP VPS 136.144.174.219, k3s v1.34.5.
| Resource | Allocatable | Requests (reserved) | Actual (top) | |---|---|---|---| | CPU | 2000m (2 vCPU) | 1890m — 94 % | 390m — 19 % | | Memory | 3884Mi (~3.8Gi) | 3564Mi — 91 % | 3174Mi — 81 % |
Source: kubectl describe node (Allocated resources) + kubectl top nodes/pods -A.
CPU is wildly over-requested, barely used. Requests reserve 94 % of CPU but the node actually burns 19 %. CPU is a compressible resource — over-requesting it costs nothing at runtime but blocks the scheduler from admitting new pods and makes the "94 %" figure that triggered panic during recovery essentially fictional. Right-sizing CPU requests is pure upside, near-zero risk.
Memory is the real constraint and it is genuinely tight. Requests are 91 % but actual usage is 81 % — and that 81 % is real RAM consumed mostly by 5 Spring Boot JVMs (each 267–390Mi). Right-sizing memory requests restores scheduler headroom but does not lower actual RAM — so the true defence for the 90 % memory ceiling is (a) trimming the over-reserved DB/redis/frontend requests, and (b) capping JVM heaps so actual memory can't creep. The node is small (3.8Gi) for 5 JVM stacks; memory is the binding resource, not CPU.
| Namespace | Pod | CPU req | CPU act | Mem req | Mem act | |---|---|---|---|---|---| | audit-pic-staging | backend | 100m | 8m | 256Mi | 267Mi | | audit-pic-staging | marketing | 10m | 1m | 64Mi | 3Mi | | audit-pic-staging | minio | 100m | 2m | 128Mi | 125Mi | | audit-pic-staging | postgres | 100m | 11m | 192Mi | 38Mi | | audit-pic-staging | webapp | 10m | 1m | 64Mi | 2Mi | | claimio-staging | backend | 250m | 4m | 320Mi | 335Mi | | claimio-staging | postgres | 100m | 10m | 192Mi | 35Mi | | claimio-staging | redis | 50m | 30m | 64Mi | 11Mi | | europe-login-staging | backend | 100m | 3m | 384Mi | 307Mi | | europe-login-staging | frontend | 25m | 1m | 64Mi | 3Mi | | europe-login-staging | postgresql | 100m | 11m | 256Mi | 35Mi | | europe-login-staging | redis | 50m | 27m | 64Mi | 9Mi | | recruitment-staging | backend | 100m | 4m | 384Mi | 390Mi | | recruitment-staging | frontend | 25m | 1m | 64Mi | 4Mi | | recruitment-staging | postgresql | 100m | 10m | 192Mi | 33Mi | | recruitment-staging | redis | 50m | 25m | 64Mi | 8Mi | | valideerleeftijd-staging | backend | 100m | 4m | 256Mi | 168Mi | | valideerleeftijd-staging | frontend | 50m | 1m | 64Mi | 3Mi | | valideerleeftijd-staging | redis | 100m | 30m | 64Mi | 8Mi |
System/shared (cert-manager, ingress-nginx, coredns, metrics-server, developer-portal, local-path, sealed-secrets) ≈ 450m / 370Mi requests, ~25m/420Mi actual.
Note:
kubectl top nodescurrently shows two rows (cloud+cloud.example.com) with identical 390m/3174Mi — an artefact of the node re-registering across the recovery reboot (two boot IDs indescribe). It is one physical node; figures above are that node.
Set each request to roughly actual × 1.5–2 headroom (room for spikes/restarts), never below observed peak. CPU requests collapse (DBs/redis/frontends idle); memory requests drop on the DB/redis/frontend/sidecar pods, while JVM backend memory requests stay ~as-is (they really do use 256–390Mi). This frees genuine scheduler headroom and makes the 90 % gate in Lever (3) meaningful.
Per-stack: current → recommended requests (apply in each chart's values-staging.yaml):
| Stack | Component | CPU req → | Mem req → | Chart values file |
|---|---|---|---|---|
| auditPic | backend | 100m → 50m | 256Mi → 320Mi¹ | auditPic/helm/audit-pic/values-staging.yaml |
| | minio | 100m → 25m | 128Mi → 160Mi | auditPic/helm/audit-pic/values-staging.yaml |
| | postgres | 100m → 25m | 192Mi → 96Mi | auditPic/helm/audit-pic/values-staging.yaml |
| | marketing/webapp | 10m → 10m | 64Mi → 24Mi | auditPic/helm/audit-pic-frontend/values-staging.yaml |
| claimio | backend | 250m → 50m | 320Mi → 400Mi¹ | claimio/k8s/values-staging.yaml |
| | postgres | 100m → 25m | 192Mi → 96Mi | claimio/k8s/values-staging.yaml |
| | redis | 50m → 25m | 64Mi → 32Mi | claimio/k8s/values-staging.yaml |
| europeLogin | backend | 100m → 50m | 384Mi → 384Mi | europeLogin/helm/europe-login/values-staging.yaml |
| | frontend | 25m → 10m | 64Mi → 24Mi | europeLogin/helm/europe-login/values-staging.yaml |
| | postgresql | 100m → 25m | 256Mi → 96Mi | europeLogin/helm/europe-login/values-staging.yaml |
| | redis | 50m → 25m | 64Mi → 32Mi | europeLogin/helm/europe-login/values-staging.yaml |
| recruitment | backend | 100m → 50m | 384Mi → 448Mi¹ | mahmoud-consultancy/helm/recruitment-platform/values-staging.yaml |
| | frontend | 25m → 10m | 64Mi → 24Mi | same |
| | postgresql | 100m → 25m | 192Mi → 96Mi | same |
| | redis | 50m → 25m | 64Mi → 32Mi | same |
| valideerleeftijd | backend | 100m → 50m | 256Mi → 256Mi | valideerleeftijd/k8s/values-staging.yaml |
| | frontend | 50m → 10m | 64Mi → 24Mi | same |
| | redis | 100m → 25m | 64Mi → 32Mi | same |
¹ Memory request raised slightly to track real usage (backend actual ≈ request or above — under-requesting a JVM invites OOM). The savings come from CPU + DB/redis/frontend, not from squeezing the JVMs.
Projected node requests after Lever (1):
| Resource | Before | After (est.) | Headroom | |---|---|---|---| | CPU requests | 1890m / 94 % | ~870m / 44 % | ✅ ~56 % free | | Memory requests | 3564Mi / 91 % | ~3050Mi / 79 % | ✅ ~21 % free |
CPU stops being a phantom blocker; memory requests drop under 80 %, giving the scheduler real room and keeping a buffer below the 90 % ceiling.
Pair with a JVM heap cap (companion to Lever 1, controls actual memory): set
JAVA_TOOL_OPTIONS=-XX:MaxRAMPercentage=70.0 (or explicit -Xmx) on each backend so a JVM
can't grow past its container memory limit and push actual node memory toward 90 %. This is
the only part of Lever (1) that touches running behaviour — stage one backend first, watch for
OOMKills, then roll out.
Risk: Low. Requests-only changes are scheduler hints; they don't restart workloads except via the normal rolling redeploy. Worst case if set too low: a pod could be scheduled onto a tighter node and compete — mitigated by keeping requests ≥ observed peak and by Lever (3)'s gate.
Make the kubelet itself refuse to over-commit by carving out a reserved slice for the OS/k3s and setting eviction thresholds that fire at ~10 % headroom, so the node sheds load before the kernel OOM-killer does it unpredictably.
Proposed kubelet config (k3s --kubelet-arg, or /etc/rancher/k3s/kubelet.config):
# PROPOSAL ONLY — DO NOT APPLY without owner sign-off + a maintenance window
systemReserved: { cpu: "200m", memory: "300Mi" } # k3s + containerd + sshd + OS
kubeReserved: { cpu: "100m", memory: "200Mi" } # kubelet itself
evictionHard: { memory.available: "400Mi", nodefs.available: "10%" } # ~10% of 3.8Gi
evictionSoft: { memory.available: "600Mi" }
evictionSoftGracePeriod: { memory.available: "1m30s" }
Why RISKY: a mis-set evictionHard evicts live, healthy pods the moment actual memory
crosses the line — and since staging memory actual already sits at 81 %, a too-generous
threshold could start evicting immediately on apply. Eviction is disruptive (pods killed,
rescheduled, possible cascading restarts). Reserved slices also reduce allocatable, so they
must be introduced after Lever (1) frees headroom, never before.
Conditions before this is safe: (a) Lever (1) merged and actual memory comfortably <75 %;
(b) applied in a maintenance window; (c) values validated with kubectl top slack to spare;
(d) a rollback plan (revert kubelet args + restart k3s). Do not apply standalone.
A CI/CD step, run before helm upgrade, that computes what node requests would be after
the deploy and aborts if it would exceed 90 % of allocatable CPU or memory. This is the
enforcement that keeps the guardrail from eroding as stacks are added.
Sketch (scripts/node-capacity-precheck.sh, called in each repo's CD workflow pre-helm upgrade):
#!/usr/bin/env bash
# Pre-deploy guardrail: abort if projected node requests exceed 90% of allocatable.
set -euo pipefail
CEIL=90
ALLOC_CPU=$(kubectl get node -o jsonpath='{.items[0].status.allocatable.cpu}') # e.g. "2"
ALLOC_MEM=$(kubectl get node -o jsonpath='{.items[0].status.allocatable.memory}') # e.g. "3977664Ki"
# sum current requests across all namespaces (millicores / Mi)
CUR_CPU=$(kubectl describe node | awk '/cpu *[0-9]+m \([0-9]+%\)/{print}' ) # parse Allocated block
# ... compute current % from `kubectl describe node` "Allocated resources" block,
# add THIS release's requests (helm template | parse resources.requests),
# fail if (current + delta) / allocatable * 100 > CEIL.
echo "Projected CPU ${PCT_CPU}% / MEM ${PCT_MEM}% (ceiling ${CEIL}%)"
if (( PCT_CPU > CEIL || PCT_MEM > CEIL )); then
echo "::error::Deploy aborted — would exceed ${CEIL}% node allocation. Right-size requests or free capacity."
exit 1
fi
Implementation notes:
describe node "Allocated resources" + helm template <release> | yq of the
incoming chart's resources.requests; subtracts the release's current requests so a
redeploy of an existing stack isn't double-counted.developer-portal or a small ops repo) and is
curl-sourced by each CD workflow, OR copied per-repo. Cross-repo wiring NEEDS-HUMAN.kubectl get/describe); cannot itself harm anything.
The only "risk" is a false-positive blocking a legit deploy — tunable via CEIL and by
fixing requests (Lever 1).| Phase | Action | Lever | Approval | Reversible? |
|---|---|---|---|---|
| 0 | Adopt this doc; agree the 90 % ceiling as policy | — | owner ack | n/a |
| 1a | Right-size CPU requests (all stacks) via values-staging.yaml PRs | 1 | per-repo PR | ✅ git revert |
| 1b | Right-size memory requests on DB/redis/frontend/sidecars (not JVMs) | 1 | per-repo PR | ✅ git revert |
| 1c | Add MaxRAMPercentage/-Xmx to one backend, observe 24h, then roll out | 1 | per-repo PR | ✅ git revert |
| 2 | Wire the pre-check gate into CD (warn-only first, then hard-fail) | 3 | owner (cross-repo) | ✅ remove step |
| 3 | Apply kube/system-reserved + eviction thresholds in a maintenance window | 2 | owner + window | ⚠️ revert kubelet args + restart k3s |
Safe-now (after owner ack): Phase 1a–1c (requests right-sizing + heap caps) and Phase 2 as
warn-only. These are ordinary values-staging.yaml / CD-script PRs, fully git-revertible,
and free real headroom immediately.
Needs owner approval (do not self-apply): Phase 2 hard-fail (cross-repo CD change) and Phase 3 (kubelet eviction — can evict live pods; most dangerous part of the whole plan).
Never apply Lever (2) before Lever (1) has measurably dropped actual memory headroom below ~75 % — otherwise the eviction threshold may fire on apply. Lever (1) → Lever (3 warn) → Lever (3 hard-fail) → Lever (2).
The node is 2 vCPU / 3.8Gi carrying 5 Spring Boot JVM stacks + 5 databases + 5 redis. CPU is a non-issue (19 % real). Memory is the structural limit — 5 JVMs at ~300Mi each is ~1.5Gi of irreducible floor before DBs. The guardrail above keeps the existing footprint safely under 90 %, but adding a 6th JVM stack to staging will breach the memory ceiling regardless of right-sizing. Two strategic options to raise later: (a) bump the staging VPS to 4 vCPU / 8Gi, or (b) keep the local-first policy and run only a rotating subset of stacks on the shared VPS. Out of scope for this guardrail, flagged for planning.
Reacties