Day 79: Polling and a Registry

Day 79 of DevOps, Day 29 of Azure. Both tasks today turned out to run somewhere other than where the configuration implied.

A Jenkins job set up to build on an agent polls the repository from the controller, which is the machine the last four days have been about getting work off. And a container image meant to be built by the registry got built on the lab host instead, because the permission for the server-side build was missing.

One Jenkins task, one Azure task. Make a job trigger itself on a commit, then create a container registry and push an image to it. The tasks come from the KodeKloud Engineer platform.

Task 1: A job that triggers itself

The task: a job named datacenter-app-deployment, running on the stapp01 agent, triggered when a commit lands on master, updating /var/www/html.

Step 1: Check the ownership first

Check that /var/www/html is owned by sarah before relying on it.

Small step, worth naming, because this batch answers one question three different ways. Day 77 assumed the agent's user could write to the document root. This day verifies it. Day 80 gives up and uses sudo with a password in the build step. The cheapest of the three is the one that just looks.

Step 2: Poll SCM, described accurately

My own note's explanation is correct and names the limitation, which is more than a lot of tutorials manage. Jenkins "periodically checks the source code repository (SCM = Source Code Management) for changes. It does not continuously listen for pushes like a webhook."

The documentation agrees: the trigger "Accepts a cron-style string to define a regular interval at which Jenkins should check for new source changes."

Two minutes, in this case. And it worked, verified the right way, by pushing a real commit and watching the build fire on its own:

[sarah@stapp01 web]$ git push origin master To http://gitea:3000/sarah/web.git 4d0f926..6ff653e master -> master 

The schedule should be H/2 * * * * rather than */2 * * * *, for Day 74's reason: H "should be used wherever possible" and is "a hash of the job name, not a random function", so it spreads load without becoming unpredictable. With a six-minute log copy from Day 74 and a ten-minute backup from Day 76 already on round intervals, all three will eventually fire together.

Jenkins also warns about short cycles specifically: "short cycles such as */3 or H/3 will not work consistently near the end of most months", because "Hashes are always chosen in the 1-28 range". Two minutes is shorter still.

Step 3: Notice where the polling happens

This is the part my notes do not mention, and it quietly undoes the point of the last four days.

The git plugin: "By default, the plugin polls by executing a polling process or thread on the Jenkins controller."

So the build runs on stapp01. The thing that decides every two minutes whether to run it runs on the Jenkins machine. Days 75 through 78 were all about moving work off the controller, for the reason Jenkins gives directly: "running tasks on the built-in node is discouraged for security, performance, and scalability reasons." Adding an agent moves the build. It does not move the trigger.

One thing the configuration gets right by accident. The plugin "polls remotely using ls-remote when configured with a single branch (no wildcards!)", which is a network call rather than a workspace checkout. Pointing the job at master specifically is what keeps it on that cheap path, and some branch-exclusion options forfeit it: "Using this behavior will preclude the faster remote polling mechanism."

What the docs recommend

There is a configuration that gets exactly what this task wants without the two-minute heartbeat, and it is not obvious from the UI.

For a push notification to trigger a build, the git plugin requires that a job "Have Build Triggers > Poll SCM enabled. No polling schedule is required."

That last sentence is the whole trick. Tick Poll SCM, leave the schedule box empty. Nothing polls on a timer, and the repository drives the build instead. Then, "To minimise the delay between a push and a build", you "configure the remote repository to use a Webhook to notify Jenkins of changes", or for a repository without one, "a post-receive hook in the remote repository to notify Jenkins".

Better on both axes this task cares about. The build starts when the commit lands rather than up to two minutes later, and nothing runs on the controller in between. Gitea does webhooks, so the lab could have done it.

The case for keeping a polling schedule is when the repository cannot reach Jenkins, which is a firewall problem rather than this one.

One more thing I would change. The agent's remote root here is /var/www/html, the same directory being deployed to. Day 77's walkthrough kept them separate, with the agent in /home/sarah/jenkins_agent. The remote root is where Jenkins creates its workspace, its jar cache and its own bookkeeping, so pointing it at a document root puts Jenkins' files inside what Apache serves, and makes a workspace wipe a document-root wipe. The extra cd in Day 77's build step buys both of those back.

Task 2: A registry and an image

The task: a container registry on the Basic tier, with an image built from a local Dockerfile and pushed as latest.

Step 1: The name is a hostname

"LoginServer": "devopsacr652387499.azurecr.io" 

Which is why the task hands you a generated alphanumeric name instead of letting you pick devops-acr. Same reason as storage accounts on Day 16: the name becomes a DNS label, so hyphens and underscores are out.

My notes also gave the length range as 5 to 50 characters. I could not source that from the registry documentation this time, so I am recording it as unverified rather than repeating it. The DNS consequence is evidenced by the output above.

The task then asks for the image to be named after the registry, so the full reference repeats itself:

devopsacr652387499.azurecr.io/devopsacr652387499:latest └──── registry (DNS) ───────┘ └── repository ───┘ └tag┘ 

Nothing creates that repository. It comes into existence when the first image is pushed under that path, which is ECR's model from Day 38 except that ECR makes you run create-repository first. The registry is the ARM resource; repositories inside it are not, which is why az resource list never shows them.

Step 2: The build that was supposed to happen in the cloud

The documented way to build an image for ACR needs no Docker at all:

az acr build -r $ACR -t $ACR:latest /root/pyapp 

(AuthorizationFailed) The client ... does not have authorisation to perform action 'Microsoft.ContainerRegistry/registries/scheduleRun/action' 

ACR Tasks needs scheduleRun on the registry, and the lab identity does not have it. So the fallback I had ready was closed, and the local Docker route was the only one left. Fortunately the host had Docker, because otherwise both paths were gone.

Worth separating two reasons a command can be unavailable, because they call for different responses:

This was the second. az acr build works perfectly well on Basic; the identity could not invoke it. The error names the exact action, which is what you need when deciding whether to request a role or change approach.

Which is today's other half of the same question. In the Jenkins task, work I thought had moved to the agent partly stayed on the controller. Here, work I intended to run in Azure came back to the local host. Both times the machine doing the work was not the one the configuration implied.

Step 3: Three errors, one unset variable

Three commands ran before the export line, and not one of the errors mentions an empty variable:

docker push $ACR.azurecr.io/$ACR:latest -> invalid reference format docker build -t $ACR.azurecr.io/$ACR:latest . -> invalid tag ".azurecr.io/:latest": invalid reference format az acr login -n $ACR -> argument --name/-n: expected one argument 

The middle one is the only useful message, because it echoes back the tag it actually received.

Two symptoms, one cause, and the second is the one that catches people:

A flag whose value is an empty variable does not become an empty string. The shell removes the word entirely, so -n $ACR becomes -n followed by nothing, and the CLI reports a missing argument rather than an empty one. Echo variables after setting them, or use set -u and get an error that names the variable.

What the docs recommend

Two things, and the second corrects something I had written as a limitation.

Verify from the registry, not from the push. The push output says latest: digest: sha256:58110cf4..., which is the client reporting what it believes it did. az acr repository show-tags reads the server's own state, and matching digests is what ties the two together.

While doing that, a nice trap. show-tags lists latest, and show --image $ACR:latest returns "Tags": null. Both commands are correct: show --image addresses one manifest by tag and does not enumerate the tags pointing at it. Same pattern as Day 18's missing contentMd5 and Day 31's sparse create response. The response shape belongs to the command, not to the resource.

Now the correction. Basic's real limitation is the absence of private endpoints, which Premium alone supports, so a private AKS cluster pulling from a Basic registry does not work. I had written that up as a constraint to live with.

It is not a constraint at all. "You can move freely between SKUs as long as the SKU you're switching to has the required maximum storage capacity", and "When you change a registry's SKU, there's no downtime or impact on registry operations." Only the downgrade direction has conditions: geo-replications and connected registries must go before leaving Premium.

Which makes the ACR tier the opposite of nearly every other choice this track has recorded. Blob type is fixed at creation. Hierarchical namespace is a one-way upgrade. A managed disk cannot shrink. An App Service plan's OS and region are fixed when you create it. Choosing Basic commits to nothing, and that is worth knowing precisely because so little else here works that way.

What I learned

Both tasks were configured to run work in one place and ran part of it somewhere else.

The Jenkins one is the more useful finding, because it is invisible and it is load-bearing. Four days of work moving builds onto agents, and the trigger that decides when to build still runs on the controller, every two minutes, forever. The documentation offers the fix in one sentence that is easy to read past: Poll SCM enabled, no schedule required, webhook does the rest.

The Azure one was louder. AuthorizationFailed naming an exact action is about as clear as a cloud error gets, and it told me in one line that the build I intended to delegate had to come home.

And the correction I would not have found without checking: the registry tier I had written off as a commitment is the one thing in this whole track that changes freely, with no downtime, in either direction.

So here is the Day 79 question. In your pipeline, which machine is doing work you believe has been moved somewhere else?

Day 79 down. Twenty-one to go.