Some of my CI can't run on GitHub-hosted runners. Pushing images to my Docker registry, deploying updated Nomad jobs, and applying Terragrunt, which covers everything from cloud instances and DNS to Vault policies and ACLs, all need to reach the cluster, so they have to run inside it. The usual answer is self-hosted runners sitting in Nomad waiting for work, and on a personal account that gets ugly fast. Outside an org, a self-hosted runner can only be registered to one repo. CI for six repos means six runners, each registered by hand with its own token, and all of them tying up a real slice of a homelab cluster to do nothing most of the day.
I was already running Temporal on the cluster for nightly backups, snapshots, and general maintenance, so I got to thinking I could set up a polling workflow to watch my repos and dispatch a parameterized Nomad job with the right registration token, repo, labels, and image plugged in.
The shape
Temporal Schedule (every 30s) | v PollAndDispatch - load per-repo config from Consul KV - list queued self-hosted jobs per repo - count runners already pending/running - start one HandleRunner child per missing runner | v HandleRunner (one per runner) - mint a registration token - nomad job dispatch ci-runner (token passed as meta) - wait for the allocation to finish - stop the dispatched job The runner itself is a Nomad parameterized batch job. Every dispatch is one ephemeral runner: it registers, takes exactly one job, deregisters, and exits. Restarts and rescheduling are off, because a finished runner should stay finished.
Here's the runner job:
# CI Runner — on-demand ephemeral GitHub Actions runner (parameterized) # # Each dispatch spawns one ephemeral self-hosted runner that takes a single job, # then deregisters and exits. Dispatched by the Temporal poller, which mints the # registration token and passes it as meta. job "ci-runner" { region = "global" datacenters = ["munchbox"] type = "batch" node_pool = "default" # Dispatched per CI run; meta carries the target repo + minted token parameterized { meta_required = ["repo_url", "runner_token"] meta_optional = ["labels"] } # Default labels when a dispatch omits them meta { labels = "self-hosted" } group "runner" { count = 1 # amd64-only image; keep it off arm64 nodes (Pi5s) where it can't run constraint { attribute = "${attr.cpu.arch}" value = "amd64" } network { mode = "host" } # One-shot: an ephemeral runner runs a single job then exits; never # restart or reschedule a finished/failed runner restart { attempts = 0 mode = "fail" } reschedule { attempts = 0 unlimited = false } task "runner" { driver = "docker" # WI exchanged for a Vault token so the template below can read the # scoped Nomad ACL token. Role + policy live in terragrunt vault-config. vault { role = "ci-runner" change_mode = "noop" } # The default WI (identity.env) carries no Nomad policy; the scoped # NOMAD_TOKEN templated in below overrides it. identity { env = true file = true aud = ["vault.io"] } config { image = "registry.munchbox.cc/ci-runner:latest" # force_pull so a rebuilt :latest (e.g. a tool version bump) is picked up # on the next dispatch instead of a stale cached image. force_pull = true image_pull_timeout = "10m" volumes = [ # pki_int signs the Nomad server cert; this CA backs NOMAD_CACERT. "/opt/nomad/tls/vault-intermediate-ca.pem:/etc/ssl/certs/munchbox-ca.pem:ro", ] } # Scoped Nomad ACL token (submit-job) for `nomad job validate`/`plan`. # Minted by terragrunt nomad-acls -> secret/ci-runner-nomad. template { data = <<-EOF {{ with secret "secret/data/ci-runner-nomad" }} NOMAD_TOKEN={{ .Data.data.nomad_token }} {{ end }} EOF destination = "secrets/nomad.env" env = true } env { RUNNER_SCOPE = "repo" REPO_URL = "${NOMAD_META_repo_url}" RUNNER_TOKEN = "${NOMAD_META_runner_token}" LABELS = "${NOMAD_META_labels}" EPHEMERAL = "true" DISABLE_AUTO_UPDATE = "true" RUN_AS_ROOT = "false" RUNNER_NAME = "ci-runner-${NOMAD_ALLOC_ID}" RUNNER_WORKDIR = "/tmp/runner-work" # Nomad API for `nomad job validate`/`plan`; NOMAD_TOKEN from template NOMAD_ADDR = "https://192.168.68.61:4646" NOMAD_TLS_SERVER_NAME = "server.global.nomad" NOMAD_CACERT = "/etc/ssl/certs/munchbox-ca.pem" } # Ephemeral one-shot, so a flat reservation has no idle cost (no need # for memory_max/oversubscription) resources { cpu = 2000 memory = 2048 } } } } Talking to Nomad
Everything the scaler does to Nomad goes through a small wrapper around the official Go client, github.com/hashicorp/nomad/api. Dispatching a runner is a single Jobs().Dispatch call that returns the ID of the concrete dispatched job, and reaping one is a Deregister with purge, so finished dispatch jobs don't pile up in Nomad's state. The two calls worth showing are the ones the reconcile and the wait depend on.
Counting the runners already in flight doesn't need any state of its own. Every dispatched runner is a child job named <parent>/dispatch-…, and its meta still carries the repo and labels it was dispatched with, so Nomad already knows everything the reconcile needs:
// ActiveRunnerSlots returns the dispatch identity of every active (pending or // running) dispatched child of parentJobID. It lists allocations, keeps those // whose job is a dispatched child still occupying a slot, and reads each child // job's repo_url + labels meta. A child whose job info can't be fetched (already // garbage-collected) is skipped -- it no longer occupies a slot. Ephemeral // runners aren't bound to a specific job, so the caller reconciles by counting // these against queued jobs rather than tracking a runner per job_id. func (n *Nomad) ActiveRunnerSlots(ctx context.Context, parentJobID string) ([]RunnerSlot, error) { allocs, _, err := n.client.Allocations().List((&api.QueryOptions{}).WithContext(ctx)) if err != nil { return nil, fmt.Errorf("list allocations: %w", err) } prefix := parentJobID + "/dispatch-" active := make(map[string]struct{}) for _, al := range allocs { if !strings.HasPrefix(al.JobID, prefix) { continue } if al.ClientStatus == api.AllocClientStatusPending || al.ClientStatus == api.AllocClientStatusRunning { active[al.JobID] = struct{}{} } } slots := make([]RunnerSlot, 0, len(active)) for jobID := range active { job, _, err := n.client.Jobs().Info(jobID, (&api.QueryOptions{}).WithContext(ctx)) if err != nil { continue } slots = append(slots, RunnerSlot{ RepoURL: metaString(job.Meta, "repo_url"), Labels: metaString(job.Meta, "labels"), }) } return slots, nil } Knowing when a runner is done means checking its allocations. A runner that hasn't been scheduled yet has none, and that counts as not done, so a pending runner is never reaped before it runs:
// RunnerTerminal reports whether a dispatched runner job has finished: every // allocation is in a terminal client status, or the job is already gone. A job // with no allocations yet (dispatched but not scheduled) is not terminal, so a // caller polling this waits for the runner to actually run before reaping -- // never reaping one still pending or mid-job. func (n *Nomad) RunnerTerminal(ctx context.Context, jobID string) (bool, error) { allocs, _, err := n.client.Jobs().Allocations(jobID, false, (&api.QueryOptions{}).WithContext(ctx)) if err != nil { if IsJobNotFound(err) { return true, nil } return false, err } if len(allocs) == 0 { return false, nil } for _, al := range allocs { if al.ClientStatus == api.AllocClientStatusPending || al.ClientStatus == api.AllocClientStatusRunning { return false, nil } } return true, nil } Finding queued jobs
GitHub has no "list queued jobs" endpoint, so the poller has to build that list itself. It walks the repo's workflow runs that are queued or in_progress, because a run with several jobs can be in progress while one of its jobs is still waiting, and keeps the jobs that are queued and ask for self-hosted. The same job can show up under both run states, so it dedupes by job ID at the end:
func listQueuedSelfHostedJobs(ctx context.Context, cli *github.Client, owner, repo string) ([]QueuedJob, error) { var all []QueuedJob for _, status := range []string{"queued", "in_progress"} { runIDs, err := listWorkflowRunIDs(ctx, cli, owner, repo, status) if err != nil { return nil, err } for _, runID := range runIDs { jobs, err := queuedSelfHostedJobsForRun(ctx, cli, owner, repo, runID) if err != nil { return nil, err } all = append(all, jobs...) } } return dedupByID(all), nil } // queuedSelfHostedJobsForRun returns runID's jobs that are still queued and ask // for a self-hosted runner. func queuedSelfHostedJobsForRun(ctx context.Context, cli *github.Client, owner, repo string, runID int64) ([]QueuedJob, error) { opts := &github.ListWorkflowJobsOptions{Filter: "latest", PerPage: 100} var jobs []QueuedJob for job, err := range cli.Actions.ListWorkflowJobsIter(ctx, owner, repo, runID, opts) { if err != nil { return nil, fmt.Errorf("list jobs for run %d in %s/%s: %w", runID, owner, repo, err) } if job.GetStatus() != "queued" || !slices.Contains(job.Labels, selfHostedLabel) { continue } jobs = append(jobs, QueuedJob{ ID: job.GetID(), RunID: job.GetRunID(), Name: job.GetName(), Labels: job.Labels, }) } return jobs, nil } Forgejo does have an endpoint for a repo's runner jobs, but it has its own catch. A job reports waiting from the moment it's created until a runner claims it, and for a moment after the claim the status hasn't caught up yet. Counting that job as queued would dispatch a second runner for work that's already being done, so the Forgejo client also checks whether a task has been assigned:
for _, j := range jobs { if !strings.EqualFold(j.Status, statusWaiting) || j.TaskID != 0 { continue } out = append(out, git.QueuedJob{ ID: j.ID, Name: j.Name, Labels: j.RunsOn, }) } Registering each runner to the right repo
Repos in a personal account can't share self-hosted runners the way repos in a GitHub org can. Each repo needs its own runner registration, so a runner has to be registered to whichever repo queued the job. The scaler does that at dispatch time: it mints a registration token for that repo through a GitHub App installed on all my repos, and passes it to the runner as dispatch meta. Nothing is stored and nothing is set up per repo. Adding a repo to CI is installing the App on it and adding one line to the config below.
The App client itself only ever holds a JWT signed with the App's private key. Every call that touches a repo first mints an installation token scoped to that one repo with the one permission the call needs: administration: write to create a registration token, actions: read to look at the queue. From the shared GitHub client:
// installationClient mints an installation token scoped to repo with perms and // returns a token-authenticated client. This is the one place the per-call token // dance lives (SetRepoSecret and the runner methods all share it); the App // client itself only ever holds the JWT. func (g *GitHub) installationClient(ctx context.Context, repo string, perms *github.InstallationPermissions) (*github.Client, error) { tok, _, err := g.app.Apps.CreateInstallationToken(ctx, g.instID, &github.InstallationTokenOptions{ Repositories: []string{repo}, Permissions: perms, }) if err != nil { return nil, fmt.Errorf("mint installation token for %s: %w", repo, err) } opts := []github.ClientOptionsFunc{ github.WithTransport(shared.OTelTransport("github", nil)), github.WithAuthToken(tok.GetToken()), } if g.baseURL != "" { opts = append(opts, github.WithEnterpriseURLs(g.baseURL, g.baseURL)) } cli, err := github.NewClient(opts...) if err != nil { return nil, fmt.Errorf("github token client: %w", err) } return cli, nil } // CreateRunnerRegistrationToken mints a short-lived registration token a runner // uses to join owner/repo, returning the token and its expiry. Requires the App // installation to grant administration:write. func (g *GitHub) CreateRunnerRegistrationToken(ctx context.Context, owner, repo string) (string, time.Time, error) { cli, err := g.installationClient(ctx, repo, &github.InstallationPermissions{Administration: new("write")}) if err != nil { return "", time.Time{}, err } tok, _, err := cli.Actions.CreateRegistrationToken(ctx, owner, repo) if err != nil { return "", time.Time{}, fmt.Errorf("create runner registration token for %s/%s: %w", owner, repo, err) } return tok.GetToken(), tok.GetExpiresAt().Time, nil } The scaler calls this inside the dispatch activity and hands the result straight to Nomad as the runner_token meta. Only the dispatched job ID comes back out of the activity, which keeps the token out of Temporal's workflow history (kept for the namespace's retention period and visible in the UI).
Not every repo can use the App. For a repo I'm only a collaborator on, the App can't be installed, so the tokens have to be set up by hand once. I create a low-privilege personal access token for polling, which only needs Actions: read. Registering a runner needs admin on the repo, so that token has to come from the owner. Both go into Vault, where the scaler reads the polling token and the dispatched job reads the admin token and registers itself. The higher-privilege token never goes to the scaler. The same code also polls my Forgejo mirror, which mints registration tokens per repo the same way the App does.
The scaler reads its config from one JSON document at runners/config in Consul KV. Here is a portion of my Consul KV config:
{ "afreidah/munchbox": { "maxConcurrent": 3, "mode": "app" }, "afreidah/nomad-temporal-jobs": { "maxConcurrent": 3, "mode": "app" }, "alex/s3-orchestrator": { "forgejoUrl": "http://forgejo.service.consul:30028", "maxConcurrent": 1, "mode": "forgejo", "profiles": [ { "job": "forgejo-build-runner", "label": "build", "maxConcurrent": 1 } ], "vaultPath": "forgejo/scaler" }, "ev-the-dev/moat": { "mode": "vault", "profiles": [ { "job": "github-runner-moat-vm", "label": "vm", "maxConcurrent": 1 }, { "job": "go-ci-runner", "label": "go", "maxConcurrent": 3 }, { "job": "github-runner-moat", "label": "moat", "maxConcurrent": 2 } ], "registerVaultPath": "github/moat-runner", "vaultPath": "github/moat-poll" } } Profiles route jobs to different runner pools by their runs-on labels, first match wins. A plain self-hosted job falls through to the default ci-runner. Caps keep a pool from eating the cluster; anything over the cap just waits in GitHub's queue until a slot frees.
Learning how GitHub assigns jobs to self-hosted runners
My first version tied each runner to a specific queued job. It started one workflow per job and used the job's ID as the workflow's ID, so the same job couldn't be handled twice.
That isn't how GitHub works. You can't control which job a runner picks up. When a repo has several jobs queued with the same labels, GitHub hands a new runner any one of them, not necessarily the one it was started for. So the runner started for the first job could end up running the second. The first job was still waiting, but as far as the scaler knew it had already been handled, and the duplicate check stopped it from starting another runner. That job sat in the queue forever.
The fix was to stop thinking in jobs and reconcile by queue depth. Each tick, the poller groups queued jobs by (repo, labels), counts the runners already pending or running for each group from their Nomad dispatch meta, and starts children only for the difference. If a job gets skipped, it's still queued on the next tick, the count comes up short, and another runner is dispatched. There's no dedup state to get wrong.
Here's that working in the scaler's logs (trimmed to the relevant fields). A job was queued on munchbox, a runner was dispatched for it, and on the next tick the job was still queued because the runner hadn't claimed it yet. The runner already counted as active, so nothing new started:
21:01:01.826 Poll complete queued=1 active=0 started=1 21:01:02.341 Dispatched ephemeral runner repo=afreidah/munchbox job=ci-runner labels=[self-hosted] minted=true 21:01:31.853 Poll complete queued=1 active=1 started=0 21:02:02.026 Poll complete queued=0 active=0 started=0 21:02:07.465 Runner finished job=ci-runner/dispatch-1789592462-171d8bee 21:02:07.572 Reaped ephemeral runner job=ci-runner/dispatch-1789592462-171d8bee The reconcile itself, from PollAndDispatch. A bucket key is the repo plus its sorted labels...
