This is the canonical CI/CD shape every project in this workspace converges to. Drift from this document is tracked as a dod-exception GitHub issue per affected repo.
Sister docs:
definition-of-done.md,claude-github-app-setup.md, rootCLAUDE.md.
Every backend-bearing project (mahmoud-consultancy, europeLogin, auditPic, claimio, valideerleeftijd, athena) must have these workflow files under .github/workflows/:
| Filename | Trigger | Purpose |
|----------|---------|---------|
| ci-backend.yml | push, PR | Build, unit + integration tests, JaCoCo coverage gate, SpotBugs, PMD, Dependency-Check. Builds and pushes the Docker image on main/develop. Name must be "CI — Backend" (CD workflow keys off this). |
| ci-frontend.yml | push, PR | Build, lint, Karma unit tests, Cucumber/Playwright E2E. Builds and pushes the frontend image on main/develop. (Skip on Flutter-mobile-only projects — use ci-flutter.yml instead.) |
| ci-flutter.yml | push, PR | flutter analyze + flutter test --coverage. (Only for mobile-bearing projects: auditPic, claimio, eventually roomy-mobile.) |
| maestro-ci.yml | push to main/develop | Maestro Cloud run for mobile flows. (Mobile only.) |
| pr-validation.yml | PR opened/updated | Conventional Commit title check, file size warnings, large-diff warnings. |
| deploy-backend.yml | workflow_run from CI on main/develop | Helm deploy to k3s on the shared VPS. Filename and shape are canonical — claimio's cd-backend.yml is non-conforming, see issue. |
| deploy-frontend.yml | workflow_run from CI on main/develop | Same as backend, for the frontend image. (Skip on mobile-only projects.) |
| rotate-secrets.yml (or split) | scheduled + manual | Quarterly rotation of JWT, DB password, Redis password, HMAC keys, MinIO creds, etc. auditPic and mahmoud-consultancy currently use multiple files; new projects should consolidate (see issue #39 on auditPic, #159 on mahmoud-consultancy). |
| zap-scan.yml | scheduled (weekly) + manual | OWASP ZAP baseline scan against the staging deploy. Only claimio has this today — other projects need it. |
| claude.yml | issue / PR comment containing @claude | Claude GitHub App integration. Added to all 6 projects by the 2026-05-18 audit. See claude-github-app-setup.md. |
| backup-verify.yml | scheduled | Verify nightly DB backups are restorable. Only claimio has this today. |
deploy-backend.yml shapeEvery project's deploy workflow must follow this skeleton. Tracking issues below the snippet list per-project drift.
name: CD — Backend
on:
workflow_run:
workflows: ["CI — Backend"]
types: [completed]
branches: [main, develop]
workflow_dispatch:
inputs:
image_tag: { description: 'Image tag (sha-...)', required: true }
environment:
description: 'Target environment'
type: choice
options: [production, staging]
default: production
env:
IMAGE: ghcr.io/mahmoudholding/<repo>/<service> # all-lowercase, canonical org = mahmoudholding
VPS_HOST: 136.144.174.219 # shared VPS — same for all projects
VPS_USER: sarkoutmahmoud
jobs:
deploy:
runs-on: ubuntu-latest
if: >
(github.event_name == 'workflow_run' && github.event.workflow_run.conclusion == 'success') ||
github.event_name == 'workflow_dispatch'
environment:
name: ${{ github.event.inputs.environment || (github.event.workflow_run.head_branch == 'main' && 'production' || 'staging') }}
steps:
- uses: actions/checkout@v4
- name: Resolve deployment config # → environment (staging|production), image_tag, namespace
- name: Setup SSH # uses secrets.VPS_SSH_KEY
- name: Ensure namespace and GHCR pull secret
- name: Apply SealedSecret for this env # applies k8s/sealed-secrets/backend-secrets-${env}.yaml DIRECTLY (no sed-rename)
- name: Copy Helm chart to VPS # ssh "rm -rf <dst> && mkdir -p <dst>", then scp -r helm/<chart>/ → <dst>
- name: Helm upgrade # heredoc starts with `set -euo pipefail`; cd into the nested chart subdir; helm upgrade --install --atomic --timeout 5m
- name: Summary # GITHUB_STEP_SUMMARY block
Apply SealedSecret for this env applies one file per env. Pre-2026-05-19, every project sed-renamed the production sealed manifest's namespace to a staging namespace and applied that. SealedSecrets are namespace-scoped by default — the controller cannot decrypt a re-targeted manifest, so the workflow silently produced no key could decrypt secret and the pods then timed out under --atomic. The fix is: seal one file per env (backend-secrets-staging.yaml, backend-secrets-production.yaml), pick the right one based on steps.config.outputs.environment. Per-project scripts/seal-secrets.sh now takes a staging|production arg and reads from <project>-secrets-${env} in the matching 1P vault.Copy Helm chart to VPS must mkdir -p the destination AND cd one level deeper. OpenSSH 9 switched scp to SFTP by default; SFTP doesn't auto-create the destination directory, and scp -r src/ dest/ copies src AS A CHILD of dest (so the chart ends up at /tmp/<chart>/<chart>/, not /tmp/<chart>/). The fix is ssh "rm -rf X && mkdir -p X" + cd /tmp/<chart>/<src-basename> in the next step.Helm upgrade heredoc opens with set -euo pipefail. Without it, helm upgrade errors get swallowed and a subsequent kubectl rollout status on a stale-but-existing deployment succeeds, falsely greening the workflow. developer-portal CD was silently no-op for an unknown period because of this.helm dependency update has no flags. --quiet was previously passed but isn't a valid flag on that subcommand and exited non-zero.| Project | Production NS | Staging NS | Helm release name | Helm chart path |
|---------|---------------|------------|-------------------|-----------------|
| mahmoud-consultancy | recruitment | recruitment-staging | recruitment-platform | helm/recruitment-platform/ |
| europeLogin | europe-login | europe-login-staging | europe-login | helm/europe-login/ |
| auditPic | audit-pic | audit-pic-staging | audit-pic | helm/audit-pic/ |
| claimio | claimio | claimio-staging | claimio | k8s/ ⚠️ non-canonical |
| valideerleeftijd | valideerleeftijd | valideerleeftijd-staging | valideerleeftijd | k8s/ ⚠️ non-canonical |
| developer-portal | developer-portal | (no staging) | developer-portal | helm/developer-portal/ |
| athena | athena | athena-staging | athena | ops/helm/athena/ |
These items are open as chore(ci): converge to canonical CI/CD issues per affected repo. The "safe approach" rule means we do not auto-apply destructive changes without review.
| # | Drift | Affected projects | Severity | Status |
|---|-------|-------------------|----------|--------|
| 1 | IMAGE uses non-canonical org glorylabs | europeLogin | High | ⏳ in flight — PR mahmoudholding/europeLogin#76 flips workflows + helm chart; Java package + prod domain stay (scoped out) |
| 2 | Root CLAUDE.md says ghcr.io/theroomyapp/... but real org is mahmoudholding | workspace | Medium | ✅ fixed in 2026-05-18 audit (root CLAUDE.md patched) |
| 3 | VPS_HOST/VPS_USER in env: (hardcoded) vs secrets: (templated) | mixed: auditPic, mahmoud-consultancy hardcode; europeLogin, valideerleeftijd, claimio use secrets | Low — hardcoded VPS IP is leak surface in public repos | still open |
| 4 | Helm chart lives under k8s/ instead of helm/<chart>/ | claimio, valideerleeftijd | Medium | still open — helm renders correctly today but layout is non-canonical |
| 5 | CD workflow filename is cd-backend.yml instead of deploy-backend.yml | claimio | Low | still open — purely cosmetic |
| 6 | 1Password loaded at deploy time (instead of Sealed Secrets only) | claimio | Medium | ⚠️ partial — workflow updated 2026-05-19 to load env-specific item, but pattern still diverges from other projects |
| 7 | FORCE_JAVASCRIPT_ACTIONS_TO_NODE24 env hack | europeLogin, valideerleeftijd | Low | still open |
| 8 | Multiple rotation workflow files instead of one consolidated rotate-secrets.yml | auditPic (3), mahmoud-consultancy (4) | Low | issues #39, #159 — still open |
| 9 | Missing zap-scan.yml | mahmoud-consultancy, europeLogin, auditPic, valideerleeftijd | High — DoD requires ZAP baseline | still open |
| 10 | Missing backup-verify.yml | all except claimio | Medium | still open |
| 11 | Java 17 → 21 not yet bumped | mahmoud-consultancy, europeLogin, auditPic, valideerleeftijd | Medium | Java 17 EOL Sept 2026 — issues per repo |
| 12 | OpenSSH 9 SFTP scp bugs (no mkdir, nested cd path, missing set -e, invalid helm dependency update --quiet) | all 5 backend projects | High — CD silently no-op for an unknown period | ✅ fixed 2026-05-19 across all projects |
| 13 | SealedSecret namespace-scope — sed-renaming prod manifest to staging produces no key could decrypt secret | all 5 backend projects | High — staging deploys never worked | ✅ fixed 2026-05-19: env-aware seal-secrets.sh + per-env sealed manifest applied directly |
| 14 | Helm chart values.yaml missing optional defaults (postgresql.operator.enabled, minio.enabled, cvService.enabled, localDev.enabled) | mahmoud-consultancy | High | ✅ fixed 2026-05-19 (#169/#170/#171) |
| 15 | CI workflow path filters skip CD-triggering — ci-backend.yml has paths: ['backend/**'] so workflow-only PRs don't fire CI, which then doesn't fire CD via workflow_run | all 5 backend projects | Medium — surprises future contributors | still open — by design but undocumented |
All projects deploy to the same k3s cluster at 136.144.174.219. Convergence means:
<project>-staging) so an oncall engineer can kubectl -n europe-login-staging get pods without checking docsscp -r helm/<chart>/ always works the same wayingress-nginxEvery backend-bearing repo must have these secrets configured (org-level recommended where possible):
| Secret | Used by | Source |
|--------|---------|--------|
| VPS_SSH_KEY | deploy-* workflows | ~/.ssh/id_ed25519 private key |
| VPS_HOST | deploy-* workflows | 136.144.174.219 (TransIP VPS, shared) — added as repo secret 2026-05-18 across the projects that use secrets.VPS_* |
| VPS_USER | deploy-* workflows | sarkoutmahmoud |
| GHCR_TOKEN | deploy-* workflows | GitHub PAT, read:packages scope. Org-level secret scoped to all 6 deploying repos (extended 2026-05-19 from 1 → 6 via the org secret access list). |
| OP_SERVICE_ACCOUNT_TOKEN | claimio CD (1Password load-secrets-action), and local seal-secrets.sh runs | 1Password service account token — value is in Employee/Service Account Auth Token: github (see onepassword_layout memory). Not yet org-wide as of 2026-05-19. |
| ANTHROPIC_API_KEY | claude.yml | https://console.anthropic.com/settings/keys |
Per-product vaults in mahmoudholdingbv.1password.com, with one item per env:
| Vault | Items |
|-------|-------|
| AuditPic | audit-pic-secrets-{production,staging} |
| InterimPlaza | recruitment-platform-secrets-{production,staging} |
| Valideerleeftijd | valideerleeftijd-secrets-{production,staging} + rabobank-sandbox-credentials-{production,staging} |
| Claimio | claimio-secrets-{production,staging} |
| EuropeLogin | europe-login-secrets-{production,staging} |
| Athena | athena-secrets-{production,staging} (placeholder items, empty — no codebase feature needs secrets yet as of 2026-09-01) |
Each project's scripts/seal-secrets.sh takes a staging|production arg and reads from <project>-secrets-${env} in the matching vault, writing to k8s/sealed-secrets/backend-secrets-${env}.yaml (europeLogin: helm/europe-login/sealed-secrets/). The CD workflow applies the env-specific file directly via scp + kubectl apply — no rewriting.
Load-bearing rule: never rewrite a SealedSecret manifest's namespace via sed. SealedSecrets are namespace-scoped — the controller can't decrypt a re-targeted manifest. See root CLAUDE.md → "1Password Vault Layout".
Per root CLAUDE.md:
main → production deploy (manual approval gate via GitHub Environment protection)develop → staging deploy (auto)feature/*, fix/*, chore/* → CI only, no deployhotfix/* → cut from main, merged into both main AND develop via two PRsCLAUDE.md — tech stack, port registry, branchingdefinition-of-done.md — what "merge-ready" meansclaude-github-app-setup.md — how to collaborate with Claude via GitHubCLAUDE.md — project-specific overrides and history
Reacties