Hi guys! In the previous part of this Kubernetes series, I discussed how I used MetalLB, Envoy Gateway, and Gateway API to give multiple applications one shared entry point.
At that stage, I could route a request to the frontend or backend. But there was another problem I didn't want to ignore. My onprem was considered alive.
Every time someone changes the application, who should be responsible for deploying that change to Kubernetes?
I could keep building an image locally, pushing it to a registry, and manually updating a Deployment. That works for the first few tests, it did wonderfully. However, it isn't the workflow I wanted for a platform that would host more applications.
1. The problem wasn't building a Docker image
I already had application containers and a Kubernetes cluster that could run them. The application also had ClusterIP Services and an HTTPRoute, so the traffic side had a place to go.
The problem was the release process.
I didn't want the normal deployment path to look like this:
Developer changes code | v Build an image manually | v Push image to registry | v SSH to a server / access cluster | v kubectl set image or kubectl apply | v Hope the live state matches what is in Git It has a lot of problem to be honest, like manually building image? okay
It has a lot of problem to be honest, like manually building image? okay, but the last step is the real problem. Once a person changes a live Deployment manually, the cluster might be ahead of, or simply different from, the version recorded in the repository.
That also makes rollback confusing. Am I reverting a Git commit? Reusing an older image? Or undoing someone's manual change? too much headache.
I wanted a simpler answer:
Application repositories produce artifacts. The GitOps repository declares desired versions. Argo CD owns reconciliation.
That is not a magical security boundary, by the way. A pipeline with permission to modify the GitOps repository can still influence what the cluster will deploy. But at least it doesn't need a direct Kubernetes API credential to perform the normal delivery path.
2. Three repositories, two kinds of responsibility
My application was split into a frontend and a backend. Each had its own source code, Dockerfile, and CI workflow. A separate cluster-management repository held the Kubernetes manifests.
The intended flow was:
FRONTEND SOURCE REPO BACKEND SOURCE REPO | | push main push main | | v v GitHub Actions GitHub Actions test / build test / build | | v v GHCR frontend GHCR backend image: sha-FE image: sha-BE | | v v frontend-image.yaml backend-image.yaml \ / \ / +----- cluster-management --------+ | v Argo CD | v Kubernetes staging frontend + backend There are two separate concerns here.
| Component | What it owns | What it should not own in this design |
|---|---|---|
| Frontend GitHub Actions | Frontend checks, image build, GHCR publish, frontend image reference update | Direct Kubernetes deployment |
| Backend GitHub Actions | Backend checks, image build, GHCR publish, backend image reference update | Frontend image reference or direct Kubernetes deployment |
| GHCR | The published container images | Which release is currently desired in staging |
cluster-management Git repository | Kubernetes configuration and desired image references | Building application images |
| Argo CD | Reading desired state, rendering manifests, and reconciling Kubernetes | Building source code or deciding an image tag on its own |
Notice that the frontend release and backend release have different commit IDs.
That seems obvious when you look at the two source repositories. But it matters when you design the automation. There is no reason to pretend that both services share one source commit.
3. Why I didn't give GitHub Actions a kubeconfig
I considered the direct model first: build an image in CI, then run kubectl set image from the GitHub Actions runner.
It would have been relatively straightforward. But it also meant giving the CI workflow Kubernetes credentials and making the runner part of the cluster deployment control path.
I didn't want that as the default for this project.
Instead, the access model in my runbook separated three relationships:
GitHub Actions ---> GHCR publishes the application image GitHub Actions ---> GitOps Git repository writes only the application's desired image reference Argo CD ---> GitOps Git repository reads the desired state Argo CD ---> Kubernetes API reconciles the workload So no kubeconfig, Kubernetes ServiceAccount token, or Argo API token needed to be copied into the application CI workflow for the normal deployment process.
However, I want to be precise about what that does, and doesn't, improve.
Removing direct cluster credentials from CI reduces one class of exposure. It doesn't make a compromised CI runner harmless.
If a workflow has permission to modify the GitOps repository, an attacker who compromises that workflow could potentially change the desired state that Argo CD eventually applies. So the GitOps write credential is still a form of deployment authority, even if it doesn't directly access the Kubernetes API.
But there is another side to this.
Because my GitOps repository is private, Argo CD also needs GitHub credentials to read it. Ideally, those credentials should only have read access.
And since Argo CD is running inside my on-premises Kubernetes cluster, I also have to consider what happens if that environment gets compromised.
An attacker who gains access to Argo CD's repository credentials could potentially use them according to the permissions assigned to those credentials. If those permissions are broader than read-only, the impact could be greater. A compromise of Argo CD itself could also expose its Kubernetes access, which is a separate concern.
So technically, I'm not eliminating credential-related risks. I'm separating responsibilities and trying to limit what each component is allowed to do.
In my case, some repository access relied on temporary, account-linked SSH credentials because of restrictions on using deploy keys. That was an acceptable workaround for getting this non-production lab running, but I wouldn't consider it an ideal long-term permission model.
After all, a credential tied to a personal GitHub account could inherit permissions beyond what the workflow or Argo CD actually needs.
For a more mature implementation, I would prefer a dedicated GitHub App or similarly constrained identity, protected branches or approval gates as appropriate, and short-lived credentials where available. I would also limit Argo CD's destinations and resource permissions through an AppProject. None of these safeguards should be assumed to exist just because GitOps is installed.
That distinction is important to me. I don't want to call something "least privilege" merely because the YAML looks neat.
4. The real design problem: two CI pipelines editing the same file
Originally, I could describe frontend and backend images together inside the staging kustomization.yaml:
images: - name: ghcr.io/example/frontend newTag: sha-FRONTEND - name: ghcr.io/example/backend newTag: sha-BACKEND That is a perfectly valid Kustomize pattern.
But if the frontend and backend pipelines both update the same file, a small timing problem appears:
12:00 Frontend CI checks out GitOps commit A 12:01 Backend CI checks out GitOps commit A 12:03 Frontend CI updates the shared YAML, pushes commit B 12:04 Backend CI tries to push its edit based on A 12:04 Push rejected: the remote branch changed This doesn't automatically mean data loss. Git rejected the non-fast-forward push. But now the pipeline needs to reconcile its change with the remote branch.
I didn't want both pipelines repeatedly rewriting the same shared image list. So I split the ownership into two small patch files.
cluster-management/ apps/ marketplace/ base/ frontend/ deployment.yaml service.yaml backend/ deployment.yaml service.yaml overlays/ staging/ kustomization.yaml patches/ frontend-image.yaml <- frontend CI owns this backend-image.yaml <- backend CI owns this routing/ httproute.yaml The base contains the reusable workload definitions. The overlay contains staging-specific configuration, including the selected images.
In the staging kustomization.yaml, the relevant part is:
apiVersion: kustomize.config.k8s.io/v1beta1 kind: Kustomization namespace: demo-staging resources: - ../../base patches: - path: patches/frontend-image.yaml - path: patches/backend-image.yaml These are simplified examples. A real overlay may also include ConfigMaps, NetworkPolicies, routes, and other resources, which need to be declared in the appropriate base or overlay.
The frontend's patch can be as small as:
apiVersion: apps/v1 kind: Deployment metadata: name: frontend spec: template: spec: containers: - name: frontend image: ghcr.io/example/frontend:sha-FRONTEND_COMMIT The backend has a separate file with its own Deployment, container name, and image.
One technical detail here: a strategic-merge patch identifies the container entry by its name. That name must match the actual container name in the base Deployment. I would validate the rendered manifest rather than assume the patch changed the right thing.
Splitting the files does not make concurrent Git pushes automatically safe. It just means that when one runner rebases on top of the other's commit, the edits are usually independent and easier to merge.
5. The image tag is a release record, not latest
I wanted the deployed image reference to point to a specific source revision.
Each source workflow publishes an image tagged with its own commit SHA:
ghcr.io/example/frontend:sha-<frontend-commit> ghcr.io/example/backend:sha-<backend-commit> So staging might contain this combination:
Frontend = sha-a1b2c3... Backend = sha-d4e5f6... The backend does not need to restart just because the frontend changed. Their Deployment Pod templates change independently.
A commit-based tag is also easier to trace than latest. However, a tag is still a mutable registry reference unless the registry prevents overwrites. For stronger artifact immutability, image digests such as @sha256:... are preferable. The source design used SHA-based tags; it should not be described as having guaranteed digest-pinned deployments.
There is another limitation: the frontend in this project was built with Vite, so environment values such as the API base URL may be embedded during npm run build. Those VITE_* build arguments are not a substitute for Kubernetes runtime Secrets. I have to validate the target environment when building an artifact for staging.
6. What the pipeline actually needs to do
I don't think the most useful article is one that pastes 300 lines of GitHub Actions YAML without explaining anything.
The important sequence is:
- Run the source repository's checks.
- Build and publish the appropriate image to GHCR.
- Check out the GitOps repository with Git write permission.
- Update only the component's image patch.
- Render Kustomize and confirm the intended image appears.
- Commit and push that desired-state change, handling a concurrent push safely.
- Let Argo CD detect and reconcile the new Git revision.
For the GHCR part, the workflow needs only the permissions appropriate for publishing the package. A representative excerpt is:
permissions: contents: read packages: write # In a build job after source checkout and tests: # - authenticate to ghcr.io with the workflow GITHUB_TOKEN # - build for the intended node architecture # - push ghcr.io/example/frontend:sha-${{ github.sha }} That is a workflow sketch, not a complete GitHub Actions file. GitHub's official container-publishing documentation describes GITHUB_TOKEN with packages: write for this use case.
I deliberately kept the GitOps write key separate from the GHCR authentication. The built-in GITHUB_TOKEN is normally scoped to the repository running the workflow; it is not automatically authorized to write to an unrelated private GitOps repository.
Validate the change before committing it
Before pushing a new image reference, I would render the overlay:
cd cluster-management kubectl kustomize apps/marketplace/overlays/staging \ > /tmp/staging-rendered.yaml test -s /tmp/staging-rendered.yaml grep -n 'image:' /tmp/staging-rendered.yaml I would then confirm that both rendered Deployments contain the intended image references. That also catches mistakes in patch names, paths, and Kustomize structure before Argo CD sees the commit.
This test is static validation. It does not prove that Kubernetes pulled the image, started a Pod, or served a request.
What about simultaneous GitOps updates?
GitHub Actions' concurrency setting can serialize jobs or runs within one repository. But my frontend and backend workflows were in different repositories, so a concurrency group in the frontend repository does not coordinate with the backend repository's group.
A simplified push strategy is:
# Run from the GitOps checkout after committing a component patch. # This assumes a clean commit that can be safely rebased. for attempt in 1 2 3 4 5; do git fetch origin main git rebase origin/main || { echo 'Rebase conflict: stop and investigate.' >&2 exit 1 } if git push origin HEAD:main; then echo 'GitOps update pushed.' break fi if [ "$attempt" -eq 5 ]; then echo 'GitOps push still failing after retries.' >&2 exit 1 fi done Why not just force-push? Because force-pushing from a CI pipeline risks overwriting the other component's deployment update. In the normal case, separate patch files let Git rebase independent commits cleanly. A genuine rebase conflict should stop the pipeline rather than be hidden.
This is still only one approach. A pull request into a protected GitOps branch would offer stronger review and audit gates, at the cost of another approval step. For staging, the source runbook intentionally favored direct automation.
7. Argo CD owns the cluster-facing part
For this project, Argo CD itself was managed with Helm. That was a platform installation decision. The application's desired state was managed with Kustomize and watched by an Argo CD Application.
The relationship was:
Helm: install/manage Argo CD itself | v Argo CD Application: watch the GitOps source | v Kustomize: render staging resources | v Kubernetes: reconcile Deployments and Services Argo CD natively supports Kustomize sources. The application points at the desired repository path; I didn't need a separate kubectl apply -k command in the application pipeline.
A reduced Application example would look like this:
apiVersion: argoproj.io/v1alpha1 kind: Application metadata: name: demo-staging namespace: argocd spec: project: staging source: repoURL: https://github.com/example/cluster-management.git targetRevision: main path: apps/marketplace/overlays/staging destination: server: https://kubernetes.default.svc namespace: demo-staging syncPolicy: automated: enabled: true selfHeal: true prune: false That example assumes the AppProject, repository credentials, namespace, and required RBAC already exist. A private repository also needs appropriate credentials. The repoURL shown here is illustrative, not a live repository.
There are two Argo CD settings I didn't want to treat casually:
-
selfHealallows Argo CD to reconcile certain live-cluster drift back toward Git's desired state. -
pruneallows deletion of resources that are no longer present in the desired state when automated pruning is enabled.
I preferred starting with manual sync and no automatic pruning, inspecting the diff, then introducing automated sync and self-healing only after the repository accurately represented the existing application. Pruning was a later, separate decision.
That order matters. Automatically reconciling an incomplete or incorrect Git repository can break an application very efficiently.
One more point: Synced is about the relationship between the desired and live resource state. Healthy is a separate assessment. Neither one alone proves that the application's user-facing endpoint works correctly.
8. What I would verify end to end
My runbooks contain detailed acceptance procedures, including a frontend test before activating the backend workflow. They don't contain complete captured logs for every step, so I will describe the validation procedure and expected outcome, not invent a passed test transcript.
| Check | Command or location | What it proves if successful |
|---|---|---|
| Source validation | GitHub Actions job logs | The relevant checks completed |
| Image published | GHCR package and tag | An image reference exists in the registry |
| Desired state updated | GitOps repository commit and patch diff | CI wrote the intended source SHA into Git |
| Manifests render | kubectl kustomize | The selected overlay produces valid YAML and intended image references |
| Argo has accepted source | kubectl get application -n argocd | Argo CD sees the desired revision and reports sync/health status |
| Pod template updated | kubectl get deployment -o jsonpath=... | The workload spec contains the new image reference |
| Rollout completes | kubectl rollout status | The Deployment rollout reached Kubernetes' completion criteria |
| App responds | App-specific HTTP health/smoke tests | The tested endpoint responded through the selected route |
For example, in an example namespace:
kubectl get application demo-staging -n argocd kubectl get deployment frontend -n demo-staging \ -o jsonpath='{.spec.template.spec.containers[*].image}{"\n"}' kubectl rollout status deployment/frontend \ -n demo-staging --timeout=180s Then I would repeat the test with the backend, checking that a backend-only change does not modify the frontend's desired image reference.
There is also a useful negative test: deliberately modify a managed Deployment in the cluster only in a disposable test environment, and check whether self-healing restores the value declared in Git. That's a controlled test of reconciliation, not a suggested operating habit.
The important distinction is that a successful build doesn't mean a successful release. Publishing an image is not the same as writing desired state, and writing desired state is not the same as confirming a healthy application.
9. Two problems that belong in the GitOps story
So while creating it, i have noted some issues that appeared when Argo CD took ownership of existing resources and private repositories.
The first was an application that repeatedly appeared OutOfSync. When developing this, I had included a system-managed metadata.generation value in a Deployment manifest. That value is maintained by Kubernetes rather than being application desired configuration. My note records removing it from the declarative manifest. I would check the live diff and refresh result before treating that as a universally proven root cause for any OutOfSync condition.
The second was a private repository access problem. So I recorded an SSH-agent-related error while Argo CD tried to resolve a Git revision. A repository Secret had also been scoped to a particular Argo CD project, while other Applications needed the same repository. That was a reasonable permission-scope hypothesis to investigate, not enough evidence by itself to say that every SSH_AUTH_SOCK error is caused by project scoping.
These two examples gave me a practical reminder: GitOps introduces its own layer of configuration and debugging. The source revision, rendered manifest, repository credential, AppProject, and live Kubernetes object all need to be inspected separately.
10. The trade-offs I would keep visible
This architecture improved clarity, but I wouldn't call it perfect.
Direct-to-main staging: The workflow I created deploys staging candidates from main. There was no pre-merge review gate in that model. A failing CI job stops that image from becoming the next desired version, but it doesn't remove the bad commit from the source branch.
Temporary Git credentials: I used a dedicated SSH key associated with a personal GitHub account. That should be treated as technical debt, with access narrowed and credentials rotated when an organization-approved service identity is available.
Two independent source repositories: A separate image patch per component reduced file contention, but the pipelines still shared one GitOps branch. A push/rebase retry or an approved merge workflow was needed to handle concurrency.
Tags versus digests: SHA-based tags improved traceability. They did not by themselves prove registry immutability.
Database migrations: I intentionally kept automatic database migration out of the first application-image workflow. Database schema changes need their own compatibility, execution, and recovery strategy. A separate migration Job can be introduced later once that contract is clear.
Rollbacks: Reverting a GitOps image change can trigger Argo CD to redeploy an earlier application image. That does not automatically undo a database migration or guarantee compatibility with external dependencies.
11. What I learned
For me, the interesting part wasn't that GitHub Actions could run a Docker build. That's the easy part, and of course that the world of devops is huge.
The more useful decision was defining who was allowed to change what.
GitHub Actions could produce a frontend or backend image and propose the intended version through Git. Argo CD was responsible for translating that version into a Kubernetes deployment. And the two source repositories needed separate release records so one team's change wouldn't accidentally replace another component's desired version.
I would still tighten the credentials and add stronger release gates before applying this model to more sensitive environments. But for this non-production platform, separating build, desired state, and reconciliation gave me a structure I could reason about.
The next article will move from who is allowed to deploy to which network connections workloads are allowed to receive: testing Kubernetes NetworkPolicy rather than assuming namespace separation provides isolation.

