Status: PROPOSE-ONLY (research, read-only). No code or config was changed. Date: 2026-05-25 Author: adhoc-ci-research (swarm) Audience: owner + manager

TL;DR

The block is a spending-limit overage, not an account suspension. Evidence below. It is therefore bypassable with a self-hosted runner.
Fastest unblock that costs nothing: register one self-hosted Actions runner (run it as a Linux container on the laptop — see §3). Self-hosted runner minutes are not metered, so they sidestep the spending limit. Caveat: this is not a zero-edit change — every job's runs-on: ubuntu-latest must become runs-on: self-hosted (108 occurrences across the repos). Everything else in the workflow YAML stays identical.
Fastest unblock if you'll pay: raise the Actions spending limit (or fix the card) in org billing settings — one click, ~$20/mo of current overage. No code change at all.
Recommended path: do both — flip on a self-hosted runner now to unblock the swarm for free, and raise the spending limit as a safety net. Treat a full migration off GitHub (§4) as a slower, optional Plan B; it is not needed to unblock.

1. Which block is it? (must establish first)

GitHub shows one combined message for two very different states:

"recent account payments have failed or your spending limit needs to be increased"

| State | What it means | Self-hosted runner bypass? | |---|---|---| | Spending-limit reached | Free minutes used up, Actions spend hit the configured limit | ✅ YES — self-hosted minutes aren't metered | | Failed payment / account suspended | Card declined on the subscription itself; Actions disabled org-wide | ❌ NO — Actions are turned off entirely, no runner helps |

What the evidence shows: spending-limit (bypassable)

Verified read-only via gh api on 2026-05-25:

The org subscription is alive and paid. GET /orgs/mahmoudholding returns plan.name = "team", filled_seats = 3, seats = 3. A failed-payment suspension downgrades/locks the plan; this is an active, paying Team org.
Actions usage crossed the free tier and is accruing a small billable amount. The enhanced billing endpoint GET /organizations/<id>/settings/billing/usage returns freely (not locked) and shows:

| Month | Actions Linux minutes | Gross | Discount (free tier) | Net billable | |---|---|---|---|---| | Jan 2026 | 483 | $2.90 | $2.90 | $0.00 | | Feb 2026 | 94 | $0.56 | $0.56 | $0.00 | | Mar 2026 | 1,584 | $9.50 | $9.50 | $0.00 | | Apr 2026 | 723 | $4.34 | $4.34 | $0.00 | | May 2026 | 6,358 | $38.15 | $18.15 (≈3,000 free min) | $20.00 |

May blew past the 3,000 free Team-plan minutes and ran up $20.00 of metered overage. That tiny, recent overage with an otherwise-healthy plan is the textbook signature of a spending-limit hit, not a declined subscription invoice.
The failed jobs die in ~1 second "waiting for a hosted runner." A sample failed run (europeLogin run 26379430326) logs: Job is waiting for a hosted runner to come online → fails in ~2s with zero steps executed. That is GitHub refusing to allocate a metered runner — exactly what a spending block does. A code/test failure would run steps first.

Conclusion: spending-limit block → bypassable by a self-hosted runner.

How the owner can 100%-confirm in 30 seconds

Open github.com → Organizations → mahmoudholding → Settings → Billing & licensing:

If you see "Spending limit reached" / a spending-limit slider at its cap → spending-limit (bypassable; this is what the API points to).
If you see a red "We couldn't charge your payment method" / "payment failed" banner on the plan → failed payment (NOT bypassable; fix the card first, a self-hosted runner won't help because Actions is disabled org-wide).

2. What our pipeline actually needs from a runner

Knowing this scopes every option below. The CD path (e.g. europeLogin/.github/workflows/deploy-backend.yml) is:

runs-on: ubuntu-latest
  → docker build + push to ghcr.io/mahmoudholding/<project>/<service>
  → ssh into TransIP VPS (136.144.174.219) with VPS_SSH_KEY
  → scp Helm chart + sealed-secrets, then helm upgrade --install on the VPS
  → kubectl rollout status

So a runner needs: Linux x64, Docker (build/push), ssh/scp/kubectl clients, and outbound network to GHCR + the VPS. It does not need to be the cluster — it drives the VPS over SSH. Notable inputs across the workspace:

108 jobs all use runs-on: ubuntu-latest (counts: mahmoud-consultancy 14 workflows, europeLogin/claimio/auditPic 11 each, valideerleeftijd 7, developer-portal 3). One macOS line is commented out.
claude.yml (Claude GitHub App) is the only workflow tightly bound to GitHub itself.

3. Fastest unblock: self-hosted runner

Why it works: self-hosted runner minutes are never metered, so they don't count against the spending limit. Storage/data-transfer can still bill trivially, but compute — the blocked resource — is free.

3.1 Where to run it — run it on the laptop, NOT the VPS

| Host | Verdict | |---|---| | TransIP VPS | ❌ Avoid for build jobs. The node is ~4 GB RAM at ~81–91 % memory after the 2026-05-24 saturation outage (see docs/ops/node-capacity-guardrail-proposal.md). A docker build runner there would re-trigger saturation. Local-first policy already says the VPS is final-promotion only. | | Laptop (this machine) | ✅ Preferred. More RAM than the VPS, already holds the VPS SSH key + docker + kubectl, and matches the swarm's local-first model. |

Important nuance: the laptop is macOS (darwin), but the workflows assume Ubuntu (apt-get, Linux Docker). So don't register a bare-metal macOS runner — run the runner inside a Linux container on the laptop (Docker Desktop's Linux VM). That gives a Linux x64 environment the existing steps expect.

3.2 Concrete steps (no-pay unblock)

These are the steps for the owner/manager to run — this task made no changes.

a) Get an org registration token (read/admin:org scope is present on the active CLI):

gh api -X POST /orgs/mahmoudholding/actions/runners/registration-token --jq .token

b) Run an ephemeral Linux runner container on the laptop (Docker-out-of-Docker via the mounted socket, so build/push works; example uses the community image):

docker run -d --restart=always --name gha-runner \
  -e REPO_URL="https://github.com/mahmoudholding" \
  -e RUNNER_SCOPE="org" \
  -e ORG_NAME="mahmoudholding" \
  -e LABELS="self-hosted,linux,x64" \
  -e ACCESS_TOKEN="<a PAT, or use RUNNER_TOKEN=<token from step a>>" \
  -v /var/run/docker.sock:/var/run/docker.sock \
  -v "$HOME/.ssh:/home/runner/.ssh:ro" \
  myoung34/github-runner:latest

(Equivalent with the official tarball: download actions/runner, then ./config.sh --url https://github.com/mahmoudholding --token <token> --labels self-hosted,linux,x64 and ./run.sh. The container form is easier to keep "always-on" and Linux-clean.)

Mounting ~/.ssh read-only lets the CD SSH/scp steps reach the VPS the same way the hosted runner did.

c) Point jobs at the self-hosted runner — the one unavoidable edit. Per job:

# from
runs-on: ubuntu-latest
# to
runs-on: [self-hosted, linux, x64]

108 occurrences. Quickest as a scripted sweep + one PR per repo, e.g.:

# illustrative — run per repo on a feature branch, review the diff, PR it
grep -rl 'runs-on: ubuntu-latest' .github/workflows \
  | xargs sed -i '' 's/runs-on: ubuntu-latest/runs-on: [self-hosted, linux, x64]/'

⚠️ The task brief hoped the workflows could stay fully unchanged. They can't: GitHub has no way to make the magic ubuntu-latest label resolve to a self-hosted runner — self-hosted runners are matched by their own labels. So one line per job must change. Everything else (steps, secrets, GHCR, SSH, Helm) is byte-for-byte identical. If you want to avoid even that edit, the only alternative is paying (see §3.3).

d) Verify: push a trivial commit, watch a job pick up on the self-hosted runner and go green. GHCR pushes and VPS deploys work unchanged because the runner has Docker + the SSH key.

3.3 Even-faster, zero-edit, but costs money

If a ~$20/mo overage is acceptable, the absolute fastest unblock is no code at all:

Org → Settings → Billing → raise the Actions spending limit above the current spend (or set it to a capped value like $50), and confirm the payment method is valid.
Jobs resume on GitHub-hosted runners immediately; nothing else changes.

This is worth doing in addition to the runner as a safety net, even if you adopt self-hosted as the primary path.

4. Migration options (if leaving GitHub Actions / GitHub)

These are heavier and not required to unblock. Ordered by fit for our GHCR → Helm → k3s pipeline. Effort is relative to our 6-repo, 108-job footprint.

GHCR / registry impact (applies to all "leave GitHub" options)

GHCR is tied to the mahmoudholding GitHub org, not to Actions. You can keep pushing/pulling GHCR from any CI using a PAT — leaving Actions does not force leaving GHCR.
Leaving GitHub entirely (moving repos to Gitea/Forgejo/GitLab) means you either (a) keep GHCR as an external registry via a token, or (b) self-host a registry (Forgejo/Gitea and GitLab both ship one; or run registry:2 / Harbor). On the 4 GB VPS a registry is feasible but adds memory pressure — prefer keeping GHCR.
claude.yml (Claude GitHub App) and GitHub-native PR checks would need rework on any non-GitHub host.

Option comparison

| Option | Workflow-YAML reuse | Self-host on our infra | Migration effort | Cost | Fit notes | |---|---|---|---|---|---| | Gitea/Forgejo Actions | ⭐ High — runs the same Actions YAML via act_runner | Yes, but the runner should live on the laptop (build memory), Forgejo server tiny | Medium if also moving repos; Low if only adding runners | Free (self-host) | Closest drop-in. runs-on labels differ; most marketplace actions work. Keep GHCR via token. | | Self-hosted GH runner (§3) | ⭐⭐ Highest — identical YAML except runs-on | Laptop runner | Lowest | Free (compute) | Not a migration — stays on GitHub. The recommended unblock. | | Woodpecker CI | Low — rewrite to Woodpecker pipeline YAML | Yes, lightweight (good for 4 GB) | Medium–High (rewrite all 108 jobs) | Free | Lean, Docker-native. Pairs with Gitea/Forgejo. Full rewrite cost. | | Drone | Low — rewrite to .drone.yml | Yes | High | Free (OSS) | Mature but Woodpecker (its fork) is more actively maintained; prefer Woodpecker. | | GitLab CI | None — rewrite to .gitlab-ci.yml + move repos to GitLab | Yes (self-managed GitLab is heavy, won't fit 4 GB VPS comfortably; needs its own box) | Highest (repo move + full rewrite) | Free OSS tier / paid SaaS | Best built-in registry + k8s integration, but heaviest lift and infra. Overkill here. | | Jenkins | None — rewrite to Jenkinsfiles | Yes (JVM, memory-hungry) | High | Free | Most flexible, most maintenance. JVM footprint fights the tight node. Not recommended for a 1-person/agency setup. |

Reading of the table

The only low-effort, high-reuse migration is Gitea/Forgejo Actions — and even that only pays off if you also want to leave GitHub the host (cost control, self-sovereignty). If you just want to dodge the billing block, the self-hosted GitHub runner gets you there with strictly less work.
Woodpecker/Drone/GitLab/Jenkins all require rewriting 108 jobs and (GitLab) moving repos — large, risky, and unnecessary for the immediate problem.

5. Recommendation

Now (free, ~15 min): confirm the block type on the billing page (expected: spending-limit). Then register one self-hosted Linux-container runner on the laptop and flip runs-on to [self-hosted, linux, x64] (one PR per repo). Swarm CI is unblocked at $0 metered compute.
Now (safety net): raise the Actions spending limit / verify the card so an accidental fallback to hosted runners doesn't re-block, and so claude.yml and any hosted-only job still function.
Later (optional Plan B): if the goal becomes leaving GitHub for cost or control, migrate repos to Forgejo + Forgejo Actions (highest YAML reuse) with runners on the laptop and GHCR retained via PAT. Do not take on Woodpecker/Drone/GitLab/ Jenkins unless a concrete requirement demands it — the rewrite cost is not justified by the billing block alone.

Do not put a build runner on the 4 GB TransIP VPS — it would re-trigger the 2026-05-24 saturation. The VPS stays final-promotion only (local-first policy).

Read-only research. No workflows, secrets, or infra were modified.

Reacties

Nog geen reacties

Athena — ops/ci-alternatives-proposal.md

CI Alternatives — Getting the Swarm Off the GitHub Actions Billing Block