Athena — mahmoud-consultancy/archive/sessions/SESSION_2025-11-02_K8S_DEPLOYMENT.md

Kubernetes Local Deployment Session - November 2, 2025

Session Overview

Duration: ~3 hours Goal: Deploy GloryLabs Recruitment Platform to local Kubernetes cluster Status: Partial Success - Infrastructure setup complete, troubleshooting deployment issues


✅ Accomplishments

1. Docker Image Building

Successfully built production-ready Docker images for both frontend and backend:

Backend Image:

  • Image: recruitment-backend:local
  • Size: 381MB
  • Fixes Applied:
    • Updated base image from alpine to full JRE (ARM64 compatibility)
    • Fixed user/group creation for non-root execution
    • Included checkstyle configuration files in build

Frontend Image:

  • Image: recruitment-frontend:local
  • Size: 97.6MB
  • Fixes Applied:
    • Changed npm ci to npm install (no package-lock.json)
    • Increased Angular budget limits to allow build to succeed
    • Fixed user permissions for nginx

Build Commands:

# Backend
cd backend
docker build -t recruitment-backend:local .

# Frontend
cd frontend/recruitment-portal
docker build --target production -t recruitment-frontend:local .

2. Kubernetes Cluster Setup

  • ✅ Enabled Kubernetes in Docker Desktop
  • ✅ Verified cluster accessibility
  • ✅ Switched to docker-desktop context
  • ✅ Created glorylabs-local namespace

3. Helm Chart Improvements

Made significant improvements to the Helm chart for production readiness:

Chart.yaml Updates:

  • Updated PostgreSQL dependency: 13.2.2418.1.3
  • Updated Redis dependency: 18.6.123.2.2

New Templates Created:

  • serviceaccount.yaml - Required for pod creation
  • Added nginx ConfigMap with Kubernetes service name resolution

Frontend Deployment Enhancements:

  • Added ConfigMap volume mount for nginx.conf
  • Fixed nginx upstream to use K8s service name: glorylabs-recruitment-platform-backend
  • Changed health check endpoints from / to /health

Values Configuration (values-local.yaml):

image:
  backend:
    repository: recruitment-backend
    pullPolicy: Never
    tag: "local"
  frontend:
    repository: recruitment-frontend
    pullPolicy: Never
    tag: "local"

postgresql:
  enabled: true
  operator:
    enabled: false
  image:
    tag: "18.0.0-debian-12-r6"
  auth:
    username: recruitment
    password: "localpass123"
    database: recruitment_local

redis:
  enabled: true
  image:
    tag: "8.2.2-debian-12-r0"
  auth:
    password: "localredis123"

minio:
  enabled: false

cvService:
  enabled: false

4. Configuration Files Updated

Modified Files:

  1. /backend/Dockerfile - ARM64 compatibility fixes
  2. /frontend/recruitment-portal/Dockerfile - Build and permission fixes
  3. /frontend/recruitment-portal/angular.json - Increased style budgets
  4. /helm/recruitment-platform/Chart.yaml - Updated dependencies
  5. /helm/recruitment-platform/values-local.yaml - Local deployment config
  6. /helm/recruitment-platform/templates/frontend-configmap.yaml - Added nginx.conf
  7. /helm/recruitment-platform/templates/frontend-deployment.yaml - Added volume mounts
  8. /helm/recruitment-platform/templates/serviceaccount.yaml - Created new
  9. /helm/recruitment-platform/templates/cv-service-deployment.yaml - Added conditional

⚠️ Current Issues

Issue 1: PostgreSQL/Redis Image Overrides Not Working

Problem: Bitnami subchart dependencies are not respecting image tag overrides in values-local.yaml

Current Behavior:

  • PostgreSQL still trying to pull: bitnami/postgresql:16.1.0-debian-11-r15 (doesn't exist)
  • Redis still trying to pull: bitnami/redis:7.2.3-debian-11-r2 (doesn't exist)

Error:

Error: ImagePullBackOff
Failed to pull image "docker.io/bitnami/postgresql:16.1.0-debian-11-r15":
manifest for bitnami/postgresql:16.1.0-debian-11-r15 not found

Root Cause: The Bitnami Helm subcharts have their own values.yaml with hardcoded image tags that are overriding our configuration.

Potential Solutions:

  1. Update Chart.yaml dependencies to even newer versions
  2. Use --set flags during Helm install to force override
  3. Fork and modify the Bitnami charts
  4. Use standalone PostgreSQL/Redis deployments instead of subcharts

Issue 2: Backend Pod Not Created

Problem: Backend deployment exists but no pods are being created

Status: RESOLVED (ServiceAccount created) Previous Error: serviceaccount "glorylabs-recruitment-platform" not found Fix Applied: Created serviceaccount.yaml template

Note: Still need to verify backend pod starts after database issues are resolved

Issue 3: Frontend Crash Loop

Problem: Frontend pods crash because they can't connect to backend

Error:

nginx: [emerg] host not found in upstream "glorylabs-recruitment-platform-backend"

Status: Configuration is correct, waiting for backend to start


📊 Deployment Status

| Component | Image Status | Config Status | Pod Status | Notes | |-----------|--------------|---------------|------------|-------| | Backend | ✅ Built | ✅ Ready | ❌ Not Running | Waiting for DB | | Frontend | ✅ Built | ✅ Ready | ❌ Crash Loop | Waiting for backend | | PostgreSQL | ❌ Image Issue | ⚠️ Override fails | ❌ ImagePullBackOff | Bitnami subchart issue | | Redis | ❌ Image Issue | ⚠️ Override fails | ❌ ImagePullBackOff | Bitnami subchart issue | | Nginx Ingress | N/A | N/A | Skipped | Using port-forward instead |


🔧 Troubleshooting Commands

# Check cluster status
kubectl cluster-info
kubectl get nodes

# Check all resources
kubectl get all -n glorylabs-local

# Check pod logs
kubectl logs <pod-name> -n glorylabs-local

# Describe pod for events
kubectl describe pod <pod-name> -n glorylabs-local

# Check Helm release
helm list -n glorylabs-local
helm status glorylabs -n glorylabs-local

# Force recreate statefulsets
kubectl delete statefulset glorylabs-postgresql glorylabs-redis-master -n glorylabs-local

# Upgrade Helm release
helm upgrade glorylabs ./helm/recruitment-platform -f ./helm/recruitment-platform/values-local.yaml -n glorylabs-local

# Uninstall and reinstall
helm uninstall glorylabs -n glorylabs-local
kubectl delete namespace glorylabs-local
helm install glorylabs ./helm/recruitment-platform -f ./helm/recruitment-platform/values-local.yaml -n glorylabs-local --create-namespace

🎯 Next Steps

Priority 1: Fix Database Image Issues

Try one of these approaches:

Option A: Force Image Override with --set

helm upgrade glorylabs ./helm/recruitment-platform \\
  -f ./helm/recruitment-platform/values-local.yaml \\
  -n glorylabs-local \\
  --set postgresql.image.tag=18.0.0-debian-12-r6 \\
  --set redis.image.tag=8.2.2-debian-12-r0

Option B: Update to Latest Bitnami Chart Versions Edit Chart.yaml dependencies to use the absolute latest versions that have ARM64 support

Option C: Use Simple K8s Manifests Instead Create simple Deployment/StatefulSet YAML files for PostgreSQL and Redis instead of using Helm subcharts

Priority 2: Verify Backend Starts

Once databases are running:

  1. Check backend pod logs: kubectl logs <backend-pod> -n glorylabs-local
  2. Verify database connection
  3. Check application startup

Priority 3: Test Application

Once all pods are running:

  1. Port-forward frontend: kubectl port-forward svc/glorylabs-recruitment-platform-frontend 4200:80 -n glorylabs-local
  2. Port-forward backend: kubectl port-forward svc/glorylabs-recruitment-platform-backend 8090:8080 -n glorylabs-local
  3. Test application at http://localhost:4200

Priority 4: Create Quickstart Guide

Document the working deployment process in K8S_LOCAL_QUICKSTART.md


📝 Lessons Learned

  1. Helm Subcharts are Complex: Bitnami charts have their own configurations that can be difficult to override
  2. ARM64 Compatibility: Many Docker images need specific tags for Apple Silicon
  3. Nginx in K8s: Requires proper ConfigMap mounting and correct service name resolution
  4. StatefulSets: Cannot be easily recreated; must delete and let Helm recreate
  5. ServiceAccounts: Required for pods to be created in most K8s configurations

🔗 Related Files

  • Dockerfiles: /backend/Dockerfile, /frontend/recruitment-portal/Dockerfile
  • Helm Chart: /helm/recruitment-platform/
  • Values: /helm/recruitment-platform/values-local.yaml
  • Task Tracking: /CURRENT_TASK.md

⏱️ Time Spent

  • Docker Image Building: ~30 min
  • Kubernetes Setup: ~15 min
  • Helm Chart Configuration: ~1 hour
  • Troubleshooting: ~1.5 hours
  • Total: ~3 hours

🎓 Recommendation

Given the complexity of the Helm chart with Bitnami subcharts, consider one of these paths forward:

  1. Short-term: Use local services (already running in background) for development and testing
  2. Medium-term: Create simplified K8s manifests without Helm for local deployment
  3. Long-term: Fix the Helm chart for production deployment (cloud environments)

The current work is valuable and will be needed for production deployment. The Helm chart improvements made today will be useful when deploying to a production Kubernetes cluster.


Session End: November 2, 2025 Next Session: Continue with database image resolution or switch to simplified K8s approach

Reacties

Nog geen reacties