This DR plan covers the InterimPlaza recruitment platform running on k3s (TransIP VPS, 136.144.174.219). It defines recovery procedures for the following failure scenarios:

| Component | Recovery Target | |-----------|----------------| | Backend API (Spring Boot) | RTO: 5 min · RPO: 0 (stateless) | | Frontend (Angular, nginx) | RTO: 5 min · RPO: 0 (stateless) | | PostgreSQL database | RTO: 15 min · RPO: 24 h (daily backups) | | Redis cache | RTO: 5 min · RPO: 0 (cache is ephemeral) | | k3s cluster (VPS) | RTO: 60 min · RPO: 24 h | | DNS / SSL | RTO: 10 min · RPO: 0 |

RTO = Recovery Time Objective (max acceptable downtime)
RPO = Recovery Point Objective (max acceptable data loss)

2. On-Call Contact

| Role | Contact | |------|---------| | Primary engineer | Mahmoud (GloryLabs) | | VPS provider | TransIP support: https://www.transip.nl/support/ | | 1Password | support.1password.com |

3. Backup Strategy

3.1 Database Backups

A pg_dump CronJob runs daily at 02:00 UTC. Backups are stored inside the cluster at /backups/ (PVC) and should be copied off-site.

Check backup status:

ssh -i ~/.ssh/id_ed25519 sarkoutmahmoud@136.144.174.219
KUBECONFIG=~/.kube/config kubectl get cronjob -n recruitment
KUBECONFIG=~/.kube/config kubectl get pods -n recruitment -l job-name --sort-by=.metadata.creationTimestamp | tail -5

List backups on VPS:

KUBECONFIG=~/.kube/config kubectl exec -n recruitment deployment/recruitment-platform-postgres \
  -- ls -lh /backups/

Off-site backup (manual, run weekly):

# Copy latest backup from VPS to local machine
scp -i ~/.ssh/id_ed25519 \
  sarkoutmahmoud@136.144.174.219:/tmp/latest-backup.sql.gz \
  ~/backups/interimplaza-$(date +%Y%m%d).sql.gz

3.2 Application State

Deployments / Helm charts: Version-controlled in k8s/ directory
Sealed Secrets: Version-controlled in k8s/sealed-secrets/
Plaintext secrets: Stored in 1Password (vault: InterimPlaza, item: recruitment-platform-secrets)

4. Incident Response

Step 1: Detect

Monitor signals:

Sentry: https://sentry.io (project: interimplaza) — real-time error alerts
k3s pod status: kubectl get pods -n recruitment
Grafana/Loki: Port-forward to check logs (see monitoring setup)

Step 2: Classify

| Severity | Examples | Response Time | |----------|---------|---------------| | SEV-1 (Critical) | Full outage, data breach | Immediate (< 15 min) | | SEV-2 (High) | Partial outage, login broken | Within 1 hour | | SEV-3 (Medium) | Feature degraded, slow response | Within 4 hours | | SEV-4 (Low) | UI glitch, warning in logs | Within 24 hours |

Step 3: Communicate

For SEV-1/SEV-2 incidents:

Notify client (Mahmoud Consultancy) immediately
Create a GitHub issue: gh issue create --title "SEV-1: <summary>" --label "incident"
Keep the issue updated with timeline and actions taken

Step 4: Contain → Resolve → Post-Mortem

Document timeline, root cause, and remediation in the GitHub issue. For SEV-1, schedule a post-mortem within 48 hours.

5. Recovery Procedures

5.1 Restart a Crashed Pod

ssh -i ~/.ssh/id_ed25519 sarkoutmahmoud@136.144.174.219
export KUBECONFIG=~/.kube/config

# Check what's failing
kubectl get pods -n recruitment
kubectl describe pod <pod-name> -n recruitment
kubectl logs <pod-name> -n recruitment --previous

# Force restart
kubectl rollout restart deployment/recruitment-platform-backend -n recruitment
kubectl rollout restart deployment/recruitment-platform-frontend -n recruitment

5.2 Rollback to Previous Version

# Check rollout history
kubectl rollout history deployment/recruitment-platform-backend -n recruitment

# Rollback one version
kubectl rollout undo deployment/recruitment-platform-backend -n recruitment

# Rollback to specific revision
kubectl rollout undo deployment/recruitment-platform-backend -n recruitment --to-revision=3

Or trigger a rollback from GitHub Actions (manually select a previous image tag):

GitHub → Actions → CD — Backend → Run workflow → image_tag: sha-<short-sha>

5.3 Restore Database from Backup

ssh -i ~/.ssh/id_ed25519 sarkoutmahmoud@136.144.174.219
export KUBECONFIG=~/.kube/config

# Scale down backend to prevent writes during restore
kubectl scale deployment/recruitment-platform-backend --replicas=0 -n recruitment

# Copy backup file to PostgreSQL pod
kubectl cp /backups/interimplaza-<date>.sql.gz \
  recruitment/<postgres-pod>:/tmp/restore.sql.gz

# Restore
kubectl exec -n recruitment <postgres-pod> -- bash -c \
  "gunzip -c /tmp/restore.sql.gz | psql -U postgres recruitment"

# Scale backend back up
kubectl scale deployment/recruitment-platform-backend --replicas=1 -n recruitment
kubectl rollout status deployment/recruitment-platform-backend -n recruitment

5.4 Full VPS Recovery (Total Loss)

If the VPS is destroyed or unrecoverable:

Provision new VPS at TransIP (same spec: 4 vCPU, 8GB RAM, Debian 13)

Run VPS setup:

ssh-copy-id -i ~/.ssh/id_ed25519 sarkoutmahmoud@<new-ip>
VPS_IP=<new-ip> ./scripts/setup-vps.sh

Restore sealed secrets:

# Fetch new cluster's Sealed Secrets cert
ssh -i ~/.ssh/id_ed25519 sarkoutmahmoud@<new-ip> \
  'KUBECONFIG=~/.kube/config kubeseal --fetch-cert \
  --controller-name=sealed-secrets-controller \
  --controller-namespace=kube-system' > k8s/sealed-secrets-cert.pem
# Re-seal with new cert
OP_SERVICE_ACCOUNT_TOKEN=<token> ./scripts/seal-secrets.sh

Deploy application:

# Trigger CD manually from GitHub Actions with latest image tag
gh workflow run deploy-backend.yml -f image_tag=main
gh workflow run deploy-frontend.yml -f image_tag=main

Restore database from most recent backup (see §5.3)
Update DNS at TransIP to new IP address

Renew SSL:

ssh -i ~/.ssh/id_ed25519 sarkoutmahmoud@<new-ip>
sudo certbot --nginx -d glorylabs.nl -d www.glorylabs.nl

5.5 Secret Compromise

If a secret (JWT, DB password, Redis password, Firecrawl API key) is suspected to be compromised:

Immediately rotate using the appropriate GitHub Actions workflow:
- JWT secret: Actions → Rotate JWT Secret → Run workflow
- DB password: Actions → Rotate Database Password → Run workflow
- Redis password: Actions → Rotate Redis Password → Run workflow
- Firecrawl API key: rotate at app.firecrawl.dev → update 1Password → re-seal
Audit logs in Grafana/Loki for suspicious activity
Check Sentry for unexpected errors
File a GDPR breach report within 72 hours if personal data was accessed (Art. 33 GDPR)

6. Backup Restore Test

Test the full backup/restore cycle quarterly:

# 1. Create a test database
kubectl exec -n recruitment <postgres-pod> -- \
  psql -U postgres -c "CREATE DATABASE recruitment_restore_test;"

# 2. Restore latest backup into test DB
kubectl exec -n recruitment <postgres-pod> -- bash -c \
  "gunzip -c /backups/latest.sql.gz | psql -U postgres recruitment_restore_test"

# 3. Spot-check row counts match production
kubectl exec -n recruitment <postgres-pod> -- \
  psql -U postgres recruitment_restore_test -c "SELECT COUNT(*) FROM users;"
kubectl exec -n recruitment <postgres-pod> -- \
  psql -U postgres recruitment -c "SELECT COUNT(*) FROM users;"

# 4. Drop test DB
kubectl exec -n recruitment <postgres-pod> -- \
  psql -U postgres -c "DROP DATABASE recruitment_restore_test;"

Log the result (pass/fail + row counts) in a GitHub issue tagged dr-test.

7. Key Resources

| Resource | Location | |----------|----------| | VPS SSH | ssh -i ~/.ssh/id_ed25519 sarkoutmahmoud@136.144.174.219 | | 1Password secrets | Vault: InterimPlaza. Items: recruitment-platform-secrets-production, recruitment-platform-secrets-staging (per-env, established 2026-05-19) | | GitHub Actions (CD) | https://github.com/mahmoudholding/mahmoud-consultancy/actions | | TransIP VPS panel | https://www.transip.nl/cp/vps/prm/396466/ | | Sentry | https://sentry.io (project: interimplaza) | | Helm chart | k8s/ directory in repo | | Sealed Secrets cert | k8s/sealed-secrets-cert.pem |

Reacties

Nog geen reacties

Athena — mahmoud-consultancy/disaster-recovery.md

Disaster Recovery Plan — InterimPlaza Recruitment Platform

1. Scope