Athena — claimio/compliance/bcdr.md


title: Business Continuity and Disaster Recovery Plan project: claimio version: 1.0 date: 2026-03-30 owner: GloryLabs classification: Internal

Business Continuity and Disaster Recovery Plan — claimio

1. Objectives

| Metric | Target | |--------|--------| | Recovery Time Objective (RTO) | 4 hours | | Recovery Point Objective (RPO) | 24 hours (nightly backup) | | Maximum Tolerable Downtime | 8 hours |

2. Scope

This plan covers the claimio production environment:

| Component | Location | |-----------|----------| | Application (k3s) | TransIP VPS — 136.144.174.219 | | PostgreSQL | In-cluster, persistent volume on VPS | | Redis | In-cluster, ephemeral (cache only) | | Container images | GHCR — ghcr.io/mahmoudholding/claimio/backend | | Database backups | TransIP Object Store — claimio-backups (S3-compatible) | | Source code | GitHub — mahmoudholding/claimio |

3. Backup Strategy

3.1 Database Backups

| Property | Value | |----------|-------| | Tool | pg_dump (plain SQL, gzip-compressed) | | Schedule | Daily at 03:00 UTC | | Destination | s3://claimio-backups/backups/ (TransIP Object Store) | | Retention | 90 days | | Immutability | S3 Object Lock — COMPLIANCE mode, 90-day retention period | | Automation | k8s CronJob claimio-db-backup (Helm chart) | | Verification | Weekly automated restore test (backup-verify.yml workflow) |

3.2 Application State

  • All application state is in PostgreSQL (stateless application tier)
  • Redis holds only transient rate-limit counters (expendable)
  • Container images are immutable and tagged with commit SHA; redeployment is idempotent

3.3 Infrastructure as Code

All infrastructure is declarative and stored in git:

  • Helm chart: k8s/
  • Sealed Secrets: k8s/sealed-secrets/
  • CI/CD: .github/workflows/

4. Risk Register

| Risk | Likelihood | Impact | Mitigation | |------|-----------|--------|------------| | VPS hardware failure | Low | High | Restore from backup to new VPS; IaC allows rapid re-provision | | Database corruption | Very Low | High | Daily backups with 90-day retention; immutable Object Lock | | Data breach / ransomware | Low | Very High | PII encrypted at rest (AES-256-GCM); off-site immutable backups unreachable from app | | DDoS attack | Medium | Medium | Rate limiting (Bucket4j/Redis); TransIP DDoS mitigation | | Accidental data deletion | Low | High | Immutable backups; anonymized_at soft-delete for GDPR retention | | GitHub/GHCR outage | Low | Medium | Images cached on VPS; rollback via helm rollback | | 1Password / secrets outage | Very Low | Medium | Sealed Secrets in git are encrypted and self-contained |

5. Recovery Procedures

5.1 VPS Total Failure

Estimated time: 2–3 hours

  1. Provision new TransIP VPS (Debian 13)
  2. Install k3s:
    curl -sfL https://get.k3s.io | sh -
    
  3. Install Sealed Secrets controller:
    kubectl apply -f https://github.com/bitnami-labs/sealed-secrets/releases/latest/download/controller.yaml
    

    Note: The Sealed Secrets controller private key must be restored from backup before applying sealed secrets. Back up the controller key: kubectl get secret -n kube-system sealed-secrets-key -o yaml > sealed-secrets-key-backup.yaml

  4. Configure GHCR pull secret:
    kubectl create secret docker-registry ghcr-pull-secret \
      --docker-server=ghcr.io \
      --docker-username=<actor> \
      --docker-password=<GHCR_TOKEN> \
      -n claimio
    
  5. Deploy database (PostgreSQL + persistent volume):
    # Deploy postgres separately or use in-cluster via Helm
    kubectl apply -f k8s/postgres/
    
  6. Restore database from latest backup (see §5.2)
  7. Deploy application via Helm:
    helm upgrade --install claimio k8s/ \
      --namespace claimio \
      --set image.tag=<sha-tag>
    
  8. Verify health: curl https://api.claimio.nl/actuator/health

5.2 Database Restore from Backup

Estimated time: 30–60 minutes

# List available backups
aws s3 ls s3://claimio-backups/backups/ \
  --endpoint-url https://storage.eu2.transip.net | sort | tail -5

# Download and restore latest backup
BACKUP=claimio_20260330T030001Z.sql.gz
aws s3 cp s3://claimio-backups/backups/${BACKUP} - \
  --endpoint-url https://storage.eu2.transip.net \
  | gunzip \
  | psql postgresql://claimio@<host>:5432/claimiodb

# Verify
psql postgresql://claimio@<host>:5432/claimiodb \
  -c "SELECT COUNT(*) FROM claims;"

5.3 Application Rollback

# List Helm history
helm history claimio -n claimio

# Roll back to previous release
helm rollback claimio -n claimio

# Or deploy a specific image tag
helm upgrade claimio k8s/ -n claimio \
  --set image.tag=sha-<previous-sha>

5.4 Secret Rotation After Breach

If credentials are suspected compromised:

  1. Generate new secrets: openssl rand -base64 32
  2. Update 1Password vault (op://Claimio/claimio-secrets/)
  3. Re-seal: OP_SERVICE_ACCOUNT_TOKEN=<token> ./scripts/seal-secrets.sh
  4. Commit sealed secrets: git add k8s/sealed-secrets/ && git commit -m "chore: rotate secrets"
  5. Push and deploy: CI/CD picks up automatically
  6. For JWT: set JWT_PREVIOUS_SECRET = old secret during rotation window (see rotate-secrets workflow)

6. Communication Plan

| Stakeholder | Contact | When | |-------------|---------|------| | Technical lead | [TBD — fill in] | Immediately on P1/P2 | | Tenants (insurers) | Status page + email | Within 1 hour of confirmed outage | | DPO | [TBD] | Immediately if personal data is at risk | | AP (Autoriteit Persoonsgegevens) | www.autoriteitpersoonsgegevens.nl | Within 72h if data breach confirmed |

7. Testing

| Test | Frequency | Owner | Last Tested | |------|-----------|-------|-------------| | Backup restore verification | Weekly (automated) | backup-verify.yml workflow | — | | Full DR runbook walkthrough | Annually | Tech lead | — | | Failover to new VPS | Bi-annually | DevOps | — |

8. Document Maintenance

This document must be reviewed:

  • Annually, or
  • After any significant infrastructure change, or
  • After any incident that triggers this plan

| Version | Date | Author | Changes | |---------|------|--------|---------| | 1.0 | 2026-03-30 | GloryLabs | Initial version |

Reacties

Nog geen reacties