Athena — ci-cd-reference.md

CI/CD Reference — Canonical Pipeline for All Projects

This is the canonical CI/CD shape every project in this workspace converges to. Drift from this document is tracked as a dod-exception GitHub issue per affected repo.

Sister docs: definition-of-done.md, claude-github-app-setup.md, root CLAUDE.md.


1. Required workflows (per backend-bearing project)

Every backend-bearing project (mahmoud-consultancy, europeLogin, auditPic, claimio, valideerleeftijd, athena) must have these workflow files under .github/workflows/:

| Filename | Trigger | Purpose | |----------|---------|---------| | ci-backend.yml | push, PR | Build, unit + integration tests, JaCoCo coverage gate, SpotBugs, PMD, Dependency-Check. Builds and pushes the Docker image on main/develop. Name must be "CI — Backend" (CD workflow keys off this). | | ci-frontend.yml | push, PR | Build, lint, Karma unit tests, Cucumber/Playwright E2E. Builds and pushes the frontend image on main/develop. (Skip on Flutter-mobile-only projects — use ci-flutter.yml instead.) | | ci-flutter.yml | push, PR | flutter analyze + flutter test --coverage. (Only for mobile-bearing projects: auditPic, claimio, eventually roomy-mobile.) | | maestro-ci.yml | push to main/develop | Maestro Cloud run for mobile flows. (Mobile only.) | | pr-validation.yml | PR opened/updated | Conventional Commit title check, file size warnings, large-diff warnings. | | deploy-backend.yml | workflow_run from CI on main/develop | Helm deploy to k3s on the shared VPS. Filename and shape are canonical — claimio's cd-backend.yml is non-conforming, see issue. | | deploy-frontend.yml | workflow_run from CI on main/develop | Same as backend, for the frontend image. (Skip on mobile-only projects.) | | rotate-secrets.yml (or split) | scheduled + manual | Quarterly rotation of JWT, DB password, Redis password, HMAC keys, MinIO creds, etc. auditPic and mahmoud-consultancy currently use multiple files; new projects should consolidate (see issue #39 on auditPic, #159 on mahmoud-consultancy). | | zap-scan.yml | scheduled (weekly) + manual | OWASP ZAP baseline scan against the staging deploy. Only claimio has this today — other projects need it. | | claude.yml | issue / PR comment containing @claude | Claude GitHub App integration. Added to all 6 projects by the 2026-05-18 audit. See claude-github-app-setup.md. | | backup-verify.yml | scheduled | Verify nightly DB backups are restorable. Only claimio has this today. |

2. Canonical deploy-backend.yml shape

Every project's deploy workflow must follow this skeleton. Tracking issues below the snippet list per-project drift.

name: CD — Backend

on:
  workflow_run:
    workflows: ["CI — Backend"]
    types: [completed]
    branches: [main, develop]
  workflow_dispatch:
    inputs:
      image_tag: { description: 'Image tag (sha-...)', required: true }
      environment:
        description: 'Target environment'
        type: choice
        options: [production, staging]
        default: production

env:
  IMAGE: ghcr.io/mahmoudholding/<repo>/<service>      # all-lowercase, canonical org = mahmoudholding
  VPS_HOST: 136.144.174.219                            # shared VPS — same for all projects
  VPS_USER: sarkoutmahmoud

jobs:
  deploy:
    runs-on: ubuntu-latest
    if: >
      (github.event_name == 'workflow_run' && github.event.workflow_run.conclusion == 'success') ||
      github.event_name == 'workflow_dispatch'
    environment:
      name: ${{ github.event.inputs.environment || (github.event.workflow_run.head_branch == 'main' && 'production' || 'staging') }}
    steps:
      - uses: actions/checkout@v4
      - name: Resolve deployment config        # → environment (staging|production), image_tag, namespace
      - name: Setup SSH                         # uses secrets.VPS_SSH_KEY
      - name: Ensure namespace and GHCR pull secret
      - name: Apply SealedSecret for this env   # applies k8s/sealed-secrets/backend-secrets-${env}.yaml DIRECTLY (no sed-rename)
      - name: Copy Helm chart to VPS            # ssh "rm -rf <dst> && mkdir -p <dst>", then scp -r helm/<chart>/ → <dst>
      - name: Helm upgrade                      # heredoc starts with `set -euo pipefail`; cd into the nested chart subdir; helm upgrade --install --atomic --timeout 5m
      - name: Summary                           # GITHUB_STEP_SUMMARY block

Why every step looks the way it does (2026-05-19 root-cause notes)

  • Apply SealedSecret for this env applies one file per env. Pre-2026-05-19, every project sed-renamed the production sealed manifest's namespace to a staging namespace and applied that. SealedSecrets are namespace-scoped by default — the controller cannot decrypt a re-targeted manifest, so the workflow silently produced no key could decrypt secret and the pods then timed out under --atomic. The fix is: seal one file per env (backend-secrets-staging.yaml, backend-secrets-production.yaml), pick the right one based on steps.config.outputs.environment. Per-project scripts/seal-secrets.sh now takes a staging|production arg and reads from <project>-secrets-${env} in the matching 1P vault.
  • Copy Helm chart to VPS must mkdir -p the destination AND cd one level deeper. OpenSSH 9 switched scp to SFTP by default; SFTP doesn't auto-create the destination directory, and scp -r src/ dest/ copies src AS A CHILD of dest (so the chart ends up at /tmp/<chart>/<chart>/, not /tmp/<chart>/). The fix is ssh "rm -rf X && mkdir -p X" + cd /tmp/<chart>/<src-basename> in the next step.
  • Helm upgrade heredoc opens with set -euo pipefail. Without it, helm upgrade errors get swallowed and a subsequent kubectl rollout status on a stale-but-existing deployment succeeds, falsely greening the workflow. developer-portal CD was silently no-op for an unknown period because of this.
  • helm dependency update has no flags. --quiet was previously passed but isn't a valid flag on that subcommand and exited non-zero.

Project namespace map (must not change without updating root CLAUDE.md ingress registry)

| Project | Production NS | Staging NS | Helm release name | Helm chart path | |---------|---------------|------------|-------------------|-----------------| | mahmoud-consultancy | recruitment | recruitment-staging | recruitment-platform | helm/recruitment-platform/ | | europeLogin | europe-login | europe-login-staging | europe-login | helm/europe-login/ | | auditPic | audit-pic | audit-pic-staging | audit-pic | helm/audit-pic/ | | claimio | claimio | claimio-staging | claimio | k8s/ ⚠️ non-canonical | | valideerleeftijd | valideerleeftijd | valideerleeftijd-staging | valideerleeftijd | k8s/ ⚠️ non-canonical | | developer-portal | developer-portal | (no staging) | developer-portal | helm/developer-portal/ | | athena | athena | athena-staging | athena | ops/helm/athena/ |

3. Drift inventory (refreshed 2026-05-19 — items struck through are now fixed)

These items are open as chore(ci): converge to canonical CI/CD issues per affected repo. The "safe approach" rule means we do not auto-apply destructive changes without review.

| # | Drift | Affected projects | Severity | Status | |---|-------|-------------------|----------|--------| | 1 | IMAGE uses non-canonical org glorylabs | europeLogin | High | ⏳ in flight — PR mahmoudholding/europeLogin#76 flips workflows + helm chart; Java package + prod domain stay (scoped out) | | 2 | Root CLAUDE.md says ghcr.io/theroomyapp/... but real org is mahmoudholding | workspace | Medium | ✅ fixed in 2026-05-18 audit (root CLAUDE.md patched) | | 3 | VPS_HOST/VPS_USER in env: (hardcoded) vs secrets: (templated) | mixed: auditPic, mahmoud-consultancy hardcode; europeLogin, valideerleeftijd, claimio use secrets | Low — hardcoded VPS IP is leak surface in public repos | still open | | 4 | Helm chart lives under k8s/ instead of helm/<chart>/ | claimio, valideerleeftijd | Medium | still open — helm renders correctly today but layout is non-canonical | | 5 | CD workflow filename is cd-backend.yml instead of deploy-backend.yml | claimio | Low | still open — purely cosmetic | | 6 | 1Password loaded at deploy time (instead of Sealed Secrets only) | claimio | Medium | ⚠️ partial — workflow updated 2026-05-19 to load env-specific item, but pattern still diverges from other projects | | 7 | FORCE_JAVASCRIPT_ACTIONS_TO_NODE24 env hack | europeLogin, valideerleeftijd | Low | still open | | 8 | Multiple rotation workflow files instead of one consolidated rotate-secrets.yml | auditPic (3), mahmoud-consultancy (4) | Low | issues #39, #159 — still open | | 9 | Missing zap-scan.yml | mahmoud-consultancy, europeLogin, auditPic, valideerleeftijd | High — DoD requires ZAP baseline | still open | | 10 | Missing backup-verify.yml | all except claimio | Medium | still open | | 11 | Java 17 → 21 not yet bumped | mahmoud-consultancy, europeLogin, auditPic, valideerleeftijd | Medium | Java 17 EOL Sept 2026 — issues per repo | | 12 | OpenSSH 9 SFTP scp bugs (no mkdir, nested cd path, missing set -e, invalid helm dependency update --quiet) | all 5 backend projects | High — CD silently no-op for an unknown period | ✅ fixed 2026-05-19 across all projects | | 13 | SealedSecret namespace-scope — sed-renaming prod manifest to staging produces no key could decrypt secret | all 5 backend projects | High — staging deploys never worked | ✅ fixed 2026-05-19: env-aware seal-secrets.sh + per-env sealed manifest applied directly | | 14 | Helm chart values.yaml missing optional defaults (postgresql.operator.enabled, minio.enabled, cvService.enabled, localDev.enabled) | mahmoud-consultancy | High | ✅ fixed 2026-05-19 (#169/#170/#171) | | 15 | CI workflow path filters skip CD-triggeringci-backend.yml has paths: ['backend/**'] so workflow-only PRs don't fire CI, which then doesn't fire CD via workflow_run | all 5 backend projects | Medium — surprises future contributors | still open — by design but undocumented |

4. Why this matters for the shared VPS staging environment

All projects deploy to the same k3s cluster at 136.144.174.219. Convergence means:

  • One reusable SSH known-hosts entry, one VPS user, one set of GHCR pull-secret semantics
  • Predictable namespace naming (<project>-staging) so an oncall engineer can kubectl -n europe-login-staging get pods without checking docs
  • One Helm chart layout so the CD step scp -r helm/<chart>/ always works the same way
  • Ingress hostnames declared once in root CLAUDE.md, owned per project but routed by the shared ingress-nginx

5. Required GitHub repo secrets

Every backend-bearing repo must have these secrets configured (org-level recommended where possible):

| Secret | Used by | Source | |--------|---------|--------| | VPS_SSH_KEY | deploy-* workflows | ~/.ssh/id_ed25519 private key | | VPS_HOST | deploy-* workflows | 136.144.174.219 (TransIP VPS, shared) — added as repo secret 2026-05-18 across the projects that use secrets.VPS_* | | VPS_USER | deploy-* workflows | sarkoutmahmoud | | GHCR_TOKEN | deploy-* workflows | GitHub PAT, read:packages scope. Org-level secret scoped to all 6 deploying repos (extended 2026-05-19 from 1 → 6 via the org secret access list). | | OP_SERVICE_ACCOUNT_TOKEN | claimio CD (1Password load-secrets-action), and local seal-secrets.sh runs | 1Password service account token — value is in Employee/Service Account Auth Token: github (see onepassword_layout memory). Not yet org-wide as of 2026-05-19. | | ANTHROPIC_API_KEY | claude.yml | https://console.anthropic.com/settings/keys |

5a. 1Password vault layout (added 2026-05-19)

Per-product vaults in mahmoudholdingbv.1password.com, with one item per env:

| Vault | Items | |-------|-------| | AuditPic | audit-pic-secrets-{production,staging} | | InterimPlaza | recruitment-platform-secrets-{production,staging} | | Valideerleeftijd | valideerleeftijd-secrets-{production,staging} + rabobank-sandbox-credentials-{production,staging} | | Claimio | claimio-secrets-{production,staging} | | EuropeLogin | europe-login-secrets-{production,staging} | | Athena | athena-secrets-{production,staging} (placeholder items, empty — no codebase feature needs secrets yet as of 2026-09-01) |

Each project's scripts/seal-secrets.sh takes a staging|production arg and reads from <project>-secrets-${env} in the matching vault, writing to k8s/sealed-secrets/backend-secrets-${env}.yaml (europeLogin: helm/europe-login/sealed-secrets/). The CD workflow applies the env-specific file directly via scp + kubectl apply — no rewriting.

Load-bearing rule: never rewrite a SealedSecret manifest's namespace via sed. SealedSecrets are namespace-scoped — the controller can't decrypt a re-targeted manifest. See root CLAUDE.md → "1Password Vault Layout".

6. Reference branching

Per root CLAUDE.md:

  • main → production deploy (manual approval gate via GitHub Environment protection)
  • develop → staging deploy (auto)
  • feature/*, fix/*, chore/* → CI only, no deploy
  • hotfix/* → cut from main, merged into both main AND develop via two PRs

7. Useful reading order for a new contributor

  1. Root CLAUDE.md — tech stack, port registry, branching
  2. This doc — pipeline shape
  3. definition-of-done.md — what "merge-ready" means
  4. claude-github-app-setup.md — how to collaborate with Claude via GitHub
  5. Project's own CLAUDE.md — project-specific overrides and history

Reacties

Nog geen reacties