Athena — ops/node-capacity-guardrail-proposal.md

Node Capacity Guardrail — Proposal (PROPOSE-ONLY)

Status: Draft proposal. Nothing in this document has been applied. Author: swarm worker adhoc-guardrail2 · Date: 2026-05-24 Goal (owner directive): the VPS staging node must never exceed ~90 % utilization — keep ≥10 % memory + CPU headroom at all times — so the 2026-05-24 saturation outage cannot recur.

⚠️ This is analysis + design only. Parts (2) and (3) need owner approval before any implementation. Part (1) is low-risk and can be staged via normal PRs once approved.


1. Measured state (read-only, today)

Single staging node (cloud), TransIP VPS 136.144.174.219, k3s v1.34.5.

| Resource | Allocatable | Requests (reserved) | Actual (top) | |---|---|---|---| | CPU | 2000m (2 vCPU) | 1890m — 94 % | 390m — 19 % | | Memory | 3884Mi (~3.8Gi) | 3564Mi — 91 % | 3174Mi — 81 % |

Source: kubectl describe node (Allocated resources) + kubectl top nodes/pods -A.

The core finding — two different problems

  1. CPU is wildly over-requested, barely used. Requests reserve 94 % of CPU but the node actually burns 19 %. CPU is a compressible resource — over-requesting it costs nothing at runtime but blocks the scheduler from admitting new pods and makes the "94 %" figure that triggered panic during recovery essentially fictional. Right-sizing CPU requests is pure upside, near-zero risk.

  2. Memory is the real constraint and it is genuinely tight. Requests are 91 % but actual usage is 81 % — and that 81 % is real RAM consumed mostly by 5 Spring Boot JVMs (each 267–390Mi). Right-sizing memory requests restores scheduler headroom but does not lower actual RAM — so the true defence for the 90 % memory ceiling is (a) trimming the over-reserved DB/redis/frontend requests, and (b) capping JVM heaps so actual memory can't creep. The node is small (3.8Gi) for 5 JVM stacks; memory is the binding resource, not CPU.

Per-pod: current request vs. actual

| Namespace | Pod | CPU req | CPU act | Mem req | Mem act | |---|---|---|---|---|---| | audit-pic-staging | backend | 100m | 8m | 256Mi | 267Mi | | audit-pic-staging | marketing | 10m | 1m | 64Mi | 3Mi | | audit-pic-staging | minio | 100m | 2m | 128Mi | 125Mi | | audit-pic-staging | postgres | 100m | 11m | 192Mi | 38Mi | | audit-pic-staging | webapp | 10m | 1m | 64Mi | 2Mi | | claimio-staging | backend | 250m | 4m | 320Mi | 335Mi | | claimio-staging | postgres | 100m | 10m | 192Mi | 35Mi | | claimio-staging | redis | 50m | 30m | 64Mi | 11Mi | | europe-login-staging | backend | 100m | 3m | 384Mi | 307Mi | | europe-login-staging | frontend | 25m | 1m | 64Mi | 3Mi | | europe-login-staging | postgresql | 100m | 11m | 256Mi | 35Mi | | europe-login-staging | redis | 50m | 27m | 64Mi | 9Mi | | recruitment-staging | backend | 100m | 4m | 384Mi | 390Mi | | recruitment-staging | frontend | 25m | 1m | 64Mi | 4Mi | | recruitment-staging | postgresql | 100m | 10m | 192Mi | 33Mi | | recruitment-staging | redis | 50m | 25m | 64Mi | 8Mi | | valideerleeftijd-staging | backend | 100m | 4m | 256Mi | 168Mi | | valideerleeftijd-staging | frontend | 50m | 1m | 64Mi | 3Mi | | valideerleeftijd-staging | redis | 100m | 30m | 64Mi | 8Mi |

System/shared (cert-manager, ingress-nginx, coredns, metrics-server, developer-portal, local-path, sealed-secrets) ≈ 450m / 370Mi requests, ~25m/420Mi actual.

Note: kubectl top nodes currently shows two rows (cloud + cloud.example.com) with identical 390m/3174Mi — an artefact of the node re-registering across the recovery reboot (two boot IDs in describe). It is one physical node; figures above are that node.


2. The three levers

Lever (1) — PRIMARY: right-size pod resource requests to reflect actual (SAFE-NOW once approved)

Set each request to roughly actual × 1.5–2 headroom (room for spikes/restarts), never below observed peak. CPU requests collapse (DBs/redis/frontends idle); memory requests drop on the DB/redis/frontend/sidecar pods, while JVM backend memory requests stay ~as-is (they really do use 256–390Mi). This frees genuine scheduler headroom and makes the 90 % gate in Lever (3) meaningful.

Per-stack: current → recommended requests (apply in each chart's values-staging.yaml):

| Stack | Component | CPU req → | Mem req → | Chart values file | |---|---|---|---|---| | auditPic | backend | 100m → 50m | 256Mi → 320Mi¹ | auditPic/helm/audit-pic/values-staging.yaml | | | minio | 100m → 25m | 128Mi → 160Mi | auditPic/helm/audit-pic/values-staging.yaml | | | postgres | 100m → 25m | 192Mi → 96Mi | auditPic/helm/audit-pic/values-staging.yaml | | | marketing/webapp | 10m → 10m | 64Mi → 24Mi | auditPic/helm/audit-pic-frontend/values-staging.yaml | | claimio | backend | 250m → 50m | 320Mi → 400Mi¹ | claimio/k8s/values-staging.yaml | | | postgres | 100m → 25m | 192Mi → 96Mi | claimio/k8s/values-staging.yaml | | | redis | 50m → 25m | 64Mi → 32Mi | claimio/k8s/values-staging.yaml | | europeLogin | backend | 100m → 50m | 384Mi → 384Mi | europeLogin/helm/europe-login/values-staging.yaml | | | frontend | 25m → 10m | 64Mi → 24Mi | europeLogin/helm/europe-login/values-staging.yaml | | | postgresql | 100m → 25m | 256Mi → 96Mi | europeLogin/helm/europe-login/values-staging.yaml | | | redis | 50m → 25m | 64Mi → 32Mi | europeLogin/helm/europe-login/values-staging.yaml | | recruitment | backend | 100m → 50m | 384Mi → 448Mi¹ | mahmoud-consultancy/helm/recruitment-platform/values-staging.yaml | | | frontend | 25m → 10m | 64Mi → 24Mi | same | | | postgresql | 100m → 25m | 192Mi → 96Mi | same | | | redis | 50m → 25m | 64Mi → 32Mi | same | | valideerleeftijd | backend | 100m → 50m | 256Mi → 256Mi | valideerleeftijd/k8s/values-staging.yaml | | | frontend | 50m → 10m | 64Mi → 24Mi | same | | | redis | 100m → 25m | 64Mi → 32Mi | same |

¹ Memory request raised slightly to track real usage (backend actual ≈ request or above — under-requesting a JVM invites OOM). The savings come from CPU + DB/redis/frontend, not from squeezing the JVMs.

Projected node requests after Lever (1):

| Resource | Before | After (est.) | Headroom | |---|---|---|---| | CPU requests | 1890m / 94 % | ~870m / 44 % | ✅ ~56 % free | | Memory requests | 3564Mi / 91 % | ~3050Mi / 79 % | ✅ ~21 % free |

CPU stops being a phantom blocker; memory requests drop under 80 %, giving the scheduler real room and keeping a buffer below the 90 % ceiling.

Pair with a JVM heap cap (companion to Lever 1, controls actual memory): set JAVA_TOOL_OPTIONS=-XX:MaxRAMPercentage=70.0 (or explicit -Xmx) on each backend so a JVM can't grow past its container memory limit and push actual node memory toward 90 %. This is the only part of Lever (1) that touches running behaviour — stage one backend first, watch for OOMKills, then roll out.

Risk: Low. Requests-only changes are scheduler hints; they don't restart workloads except via the normal rolling redeploy. Worst case if set too low: a pod could be scheduled onto a tighter node and compete — mitigated by keeping requests ≥ observed peak and by Lever (3)'s gate.


Lever (2) — kube-reserved / system-reserved + HARD eviction thresholds (RISKY — NEEDS-HUMAN)

Make the kubelet itself refuse to over-commit by carving out a reserved slice for the OS/k3s and setting eviction thresholds that fire at ~10 % headroom, so the node sheds load before the kernel OOM-killer does it unpredictably.

Proposed kubelet config (k3s --kubelet-arg, or /etc/rancher/k3s/kubelet.config):

# PROPOSAL ONLY — DO NOT APPLY without owner sign-off + a maintenance window
systemReserved:   { cpu: "200m", memory: "300Mi" }   # k3s + containerd + sshd + OS
kubeReserved:     { cpu: "100m", memory: "200Mi" }   # kubelet itself
evictionHard:     { memory.available: "400Mi", nodefs.available: "10%" }   # ~10% of 3.8Gi
evictionSoft:     { memory.available: "600Mi" }
evictionSoftGracePeriod: { memory.available: "1m30s" }

Why RISKY: a mis-set evictionHard evicts live, healthy pods the moment actual memory crosses the line — and since staging memory actual already sits at 81 %, a too-generous threshold could start evicting immediately on apply. Eviction is disruptive (pods killed, rescheduled, possible cascading restarts). Reserved slices also reduce allocatable, so they must be introduced after Lever (1) frees headroom, never before.

Conditions before this is safe: (a) Lever (1) merged and actual memory comfortably <75 %; (b) applied in a maintenance window; (c) values validated with kubectl top slack to spare; (d) a rollback plan (revert kubelet args + restart k3s). Do not apply standalone.


Lever (3) — deploy-time node-allocation PRE-CHECK gate (SAFE design — NEEDS-HUMAN to wire into CD)

A CI/CD step, run before helm upgrade, that computes what node requests would be after the deploy and aborts if it would exceed 90 % of allocatable CPU or memory. This is the enforcement that keeps the guardrail from eroding as stacks are added.

Sketch (scripts/node-capacity-precheck.sh, called in each repo's CD workflow pre-helm upgrade):

#!/usr/bin/env bash
# Pre-deploy guardrail: abort if projected node requests exceed 90% of allocatable.
set -euo pipefail
CEIL=90
ALLOC_CPU=$(kubectl get node -o jsonpath='{.items[0].status.allocatable.cpu}')      # e.g. "2"
ALLOC_MEM=$(kubectl get node -o jsonpath='{.items[0].status.allocatable.memory}')   # e.g. "3977664Ki"
# sum current requests across all namespaces (millicores / Mi)
CUR_CPU=$(kubectl describe node | awk '/cpu  *[0-9]+m \([0-9]+%\)/{print}' )  # parse Allocated block
# ... compute current % from `kubectl describe node` "Allocated resources" block,
#     add THIS release's requests (helm template | parse resources.requests),
#     fail if (current + delta) / allocatable * 100 > CEIL.
echo "Projected CPU ${PCT_CPU}% / MEM ${PCT_MEM}% (ceiling ${CEIL}%)"
if (( PCT_CPU > CEIL || PCT_MEM > CEIL )); then
  echo "::error::Deploy aborted — would exceed ${CEIL}% node allocation. Right-size requests or free capacity."
  exit 1
fi

Implementation notes:

  • Reads live describe node "Allocated resources" + helm template <release> | yq of the incoming chart's resources.requests; subtracts the release's current requests so a redeploy of an existing stack isn't double-counted.
  • Lives once in a shared location (e.g. developer-portal or a small ops repo) and is curl-sourced by each CD workflow, OR copied per-repo. Cross-repo wiring NEEDS-HUMAN.
  • Read-only against the cluster (only kubectl get/describe); cannot itself harm anything. The only "risk" is a false-positive blocking a legit deploy — tunable via CEIL and by fixing requests (Lever 1).

3. Phased rollout

| Phase | Action | Lever | Approval | Reversible? | |---|---|---|---|---| | 0 | Adopt this doc; agree the 90 % ceiling as policy | — | owner ack | n/a | | 1a | Right-size CPU requests (all stacks) via values-staging.yaml PRs | 1 | per-repo PR | ✅ git revert | | 1b | Right-size memory requests on DB/redis/frontend/sidecars (not JVMs) | 1 | per-repo PR | ✅ git revert | | 1c | Add MaxRAMPercentage/-Xmx to one backend, observe 24h, then roll out | 1 | per-repo PR | ✅ git revert | | 2 | Wire the pre-check gate into CD (warn-only first, then hard-fail) | 3 | owner (cross-repo) | ✅ remove step | | 3 | Apply kube/system-reserved + eviction thresholds in a maintenance window | 2 | owner + window | ⚠️ revert kubelet args + restart k3s |

Safe-now (after owner ack): Phase 1a–1c (requests right-sizing + heap caps) and Phase 2 as warn-only. These are ordinary values-staging.yaml / CD-script PRs, fully git-revertible, and free real headroom immediately.

Needs owner approval (do not self-apply): Phase 2 hard-fail (cross-repo CD change) and Phase 3 (kubelet eviction — can evict live pods; most dangerous part of the whole plan).

Sequencing rule

Never apply Lever (2) before Lever (1) has measurably dropped actual memory headroom below ~75 % — otherwise the eviction threshold may fire on apply. Lever (1) → Lever (3 warn) → Lever (3 hard-fail) → Lever (2).


4. Root-cause note (for the owner)

The node is 2 vCPU / 3.8Gi carrying 5 Spring Boot JVM stacks + 5 databases + 5 redis. CPU is a non-issue (19 % real). Memory is the structural limit — 5 JVMs at ~300Mi each is ~1.5Gi of irreducible floor before DBs. The guardrail above keeps the existing footprint safely under 90 %, but adding a 6th JVM stack to staging will breach the memory ceiling regardless of right-sizing. Two strategic options to raise later: (a) bump the staging VPS to 4 vCPU / 8Gi, or (b) keep the local-first policy and run only a rotating subset of stacks on the shared VPS. Out of scope for this guardrail, flagged for planning.

Reacties

Nog geen reacties