| Metric | Target | |--------|--------| | Recovery Time Objective (RTO) | 4 hours | | Recovery Point Objective (RPO) | 24 hours (nightly backup) | | Maximum Tolerable Downtime | 8 hours |
This plan covers the claimio production environment:
| Component | Location |
|-----------|----------|
| Application (k3s) | TransIP VPS — 136.144.174.219 |
| PostgreSQL | In-cluster, persistent volume on VPS |
| Redis | In-cluster, ephemeral (cache only) |
| Container images | GHCR — ghcr.io/mahmoudholding/claimio/backend |
| Database backups | TransIP Object Store — claimio-backups (S3-compatible) |
| Source code | GitHub — mahmoudholding/claimio |
| Property | Value |
|----------|-------|
| Tool | pg_dump (plain SQL, gzip-compressed) |
| Schedule | Daily at 03:00 UTC |
| Destination | s3://claimio-backups/backups/ (TransIP Object Store) |
| Retention | 90 days |
| Immutability | S3 Object Lock — COMPLIANCE mode, 90-day retention period |
| Automation | k8s CronJob claimio-db-backup (Helm chart) |
| Verification | Weekly automated restore test (backup-verify.yml workflow) |
All infrastructure is declarative and stored in git:
k8s/k8s/sealed-secrets/.github/workflows/| Risk | Likelihood | Impact | Mitigation |
|------|-----------|--------|------------|
| VPS hardware failure | Low | High | Restore from backup to new VPS; IaC allows rapid re-provision |
| Database corruption | Very Low | High | Daily backups with 90-day retention; immutable Object Lock |
| Data breach / ransomware | Low | Very High | PII encrypted at rest (AES-256-GCM); off-site immutable backups unreachable from app |
| DDoS attack | Medium | Medium | Rate limiting (Bucket4j/Redis); TransIP DDoS mitigation |
| Accidental data deletion | Low | High | Immutable backups; anonymized_at soft-delete for GDPR retention |
| GitHub/GHCR outage | Low | Medium | Images cached on VPS; rollback via helm rollback |
| 1Password / secrets outage | Very Low | Medium | Sealed Secrets in git are encrypted and self-contained |
Estimated time: 2–3 hours
curl -sfL https://get.k3s.io | sh -
kubectl apply -f https://github.com/bitnami-labs/sealed-secrets/releases/latest/download/controller.yaml
Note: The Sealed Secrets controller private key must be restored from backup before applying sealed secrets. Back up the controller key:
kubectl get secret -n kube-system sealed-secrets-key -o yaml > sealed-secrets-key-backup.yaml
kubectl create secret docker-registry ghcr-pull-secret \
--docker-server=ghcr.io \
--docker-username=<actor> \
--docker-password=<GHCR_TOKEN> \
-n claimio
# Deploy postgres separately or use in-cluster via Helm
kubectl apply -f k8s/postgres/
helm upgrade --install claimio k8s/ \
--namespace claimio \
--set image.tag=<sha-tag>
curl https://api.claimio.nl/actuator/healthEstimated time: 30–60 minutes
# List available backups
aws s3 ls s3://claimio-backups/backups/ \
--endpoint-url https://storage.eu2.transip.net | sort | tail -5
# Download and restore latest backup
BACKUP=claimio_20260330T030001Z.sql.gz
aws s3 cp s3://claimio-backups/backups/${BACKUP} - \
--endpoint-url https://storage.eu2.transip.net \
| gunzip \
| psql postgresql://claimio@<host>:5432/claimiodb
# Verify
psql postgresql://claimio@<host>:5432/claimiodb \
-c "SELECT COUNT(*) FROM claims;"
# List Helm history
helm history claimio -n claimio
# Roll back to previous release
helm rollback claimio -n claimio
# Or deploy a specific image tag
helm upgrade claimio k8s/ -n claimio \
--set image.tag=sha-<previous-sha>
If credentials are suspected compromised:
openssl rand -base64 32OP_SERVICE_ACCOUNT_TOKEN=<token> ./scripts/seal-secrets.shgit add k8s/sealed-secrets/ && git commit -m "chore: rotate secrets"JWT_PREVIOUS_SECRET = old secret during rotation window (see rotate-secrets workflow)| Stakeholder | Contact | When | |-------------|---------|------| | Technical lead | [TBD — fill in] | Immediately on P1/P2 | | Tenants (insurers) | Status page + email | Within 1 hour of confirmed outage | | DPO | [TBD] | Immediately if personal data is at risk | | AP (Autoriteit Persoonsgegevens) | www.autoriteitpersoonsgegevens.nl | Within 72h if data breach confirmed |
| Test | Frequency | Owner | Last Tested |
|------|-----------|-------|-------------|
| Backup restore verification | Weekly (automated) | backup-verify.yml workflow | — |
| Full DR runbook walkthrough | Annually | Tech lead | — |
| Failover to new VPS | Bi-annually | DevOps | — |
This document must be reviewed:
| Version | Date | Author | Changes | |---------|------|--------|---------| | 1.0 | 2026-03-30 | GloryLabs | Initial version |
Reacties