Building a CI/CD and GitOps Platform with Jenkins, Docker, Kubernetes and Argo CD: My End-to-End Capstone Project

From Jenkins credential errors to a working pipeline, Kubernetes deployment, and automated GitOps reconciliation

Author: Jamiu Olatunji Bakre

Role: Cloud, DevOps and Platform Engineer

Tags: DevOps, Platform Engineering, Jenkins, CI/CD, Docker, Kubernetes, GitOps, Argo CD, Maven, Kustomize, Minikube

Project Overview

This project was an important step in my journey towards becoming a stronger Cloud, DevOps and Platform Engineer.

My goal was to understand how application code moves from a GitHub repository through an automated build process, becomes a Docker image, and is ultimately deployed and managed in Kubernetes.

I wanted to go beyond simply installing tools and following commands from a tutorial. I wanted to understand how the components communicate, where configuration belongs, how credentials are handled, how deployment problems are investigated, and how to verify that an application is actually working.

The journey included configuring Jenkins, working with GitHub repositories, building a Spring Boot application with Maven, packaging the application with Docker, and using Kubernetes and Argo CD to manage the deployment.

One of the most memorable challenges was a Jenkins credential error that took considerable troubleshooting. The pipeline reported that it could not find a credential with the ID github-credentials. After working through the integration issues, I achieved a successful pipeline build on my 20th CI/CD pipeline build.

That experience taught me something important: building a pipeline is not just about writing a Jenkinsfile. It also requires understanding credentials, permissions, repository configuration, networking, container registries, deployment manifests, and the dependencies between each stage.

ArgoCD Output

Scaled Up ArgoCD Output

This documentation explains the architecture, the steps involved, the purpose of each component, the commands used to inspect and validate the system, and the improvements I would make when extending the project.

It also includes reference configurations for frontend and backend applications, persistent storage, monitoring, and production-oriented practices. These examples provide a foundation for future work; they should not be interpreted as proof that every optional component was deployed in the original lab.

1. Project Objectives

The project was designed around the following objectives:

1. Automate application build and verification using Jenkins.
2. Store application source code in GitHub.
3. Package the application into a Docker image.
4. Publish container images to Docker Hub.
5. Define Kubernetes resources declaratively.
6. Separate application source code from deployment configuration.
7. Use Argo CD to monitor the GitOps repository.
8. Test automated reconciliation when the live cluster configuration differs from Git.
9. Verify the deployed application's availability through Kubernetes and an HTTP request.
10. Develop a repeatable troubleshooting and documentation process.

The wider objective was to understand how these capabilities fit together as a small platform engineering workflow.

2. Project Architecture

The architecture separates the application build process from the deployment reconciliation process.

Jenkins runs on an AWS EC2 instance. The Kubernetes application and Argo CD run in a local Minikube environment.

This separation is important because a Jenkins server on EC2 cannot automatically reach a Kubernetes cluster running on a developer's local computer. The two environments have different network boundaries.

Architecture diagram

 Developer | v Application Changes | v GitHub Source Repo | v Jenkins on EC2 | +----------+-----------+ | | | v v v Maven Code Checks Docker Build | | v v Build/Test Docker Hub | v GitOps Repository | v Argo CD | v Local Minikube | v Kubernetes | v Spring Boot App 

The diagram describes the intended delivery flow. The exact automation depends on which stages are implemented in the Jenkinsfile. For example, updating the GitOps repository must be explicitly configured if it is to happen automatically after an image is published.

The role of each component

3. Repository Structure and Separation of Responsibilities

I used two GitHub repositories to separate application development from deployment configuration.

3.1 Application source repository

Repository: platform-cicd-spring-boot-demo-source

URL - https://github.com/Jahmeeu-Cloud/platform-cicd-spring-boot-demo-source

This repository holds the Spring Boot application and its Maven build configuration.

A typical structure looks like this:

platform-cicd-spring-boot-demo-source/ ├── spring-boot-app/ │ ├── pom.xml │ ├── mvnw │ ├── .mvn/ │ └── src/ │ ├── main/ │ │ ├── java/ │ │ └── resources/ │ └── test/ ├── Dockerfile ├── Jenkinsfile ├── .gitignore └── README.md 

This is a recommended structure, not a claim that every file shown was present in the original repository.

The source repository should contain the code required to build the application. Generated output, credentials, and environment-specific secrets should not be committed.

For example, Maven's target/ directory contains generated build artifacts. It is normally excluded from Git.

A suitable .gitignore could include:

target/ *.log .env .env.* !.env.example .idea/ .vscode/ .DS_Store 

The exact exclusions should match the project's development tools and build process.

3.2 GitOps configuration repository

Repository: platform-cicd-gitops-config

URL: https://github.com/Jahmeeu-Cloud/platform-cicd-gitops-config

This repository contains the desired Kubernetes configuration.

The confirmed base directory is:

platform-cicd-gitops-config/ └── apps/ └── spring-boot-app/ └── base/ ├── deployment.yaml ├── service.yaml └── kustomization.yaml 

The Deployment declares how the application should run. The Service provides a stable network endpoint. The Kustomize file identifies the manifests that belong to the application configuration.

The main principle is that the GitOps repository should describe the desired deployment state rather than merely store commands that someone runs manually.

Why keep the repositories separate?

This separation makes it easier to review application changes independently from deployment changes.

It also creates a clear path for future environment promotion:

• Developers change application code.
• Jenkins validates and packages the code.
• A versioned image is published.
• The deployment configuration references that image.
• Argo CD reconciles the cluster with the approved configuration.

In a larger platform, the same approach can support separate development, staging, and production configurations.

[SCREENSHOT: GITHUB SOURCE REPOSITORY HERE]

The source repository contains the application and build-related files.

[SCREENSHOT - GITHUB GITOPS REPOSITORY HERE]

The GitOps repository stores the Kubernetes manifests used to define the desired application state.

4. Understanding the Environment Before Starting

Before troubleshooting a CI/CD pipeline, it is useful to understand where each component is running.

In this project:

• Jenkins ran on AWS EC2.
• The Kubernetes cluster ran locally through Minikube.
• Argo CD ran inside the Minikube cluster.
• The application image was stored in Docker Hub.
• GitHub stored both the source code and the deployment configuration.

This topology has an important implication: Jenkins does not automatically have network access to the local Kubernetes API.

A common mistake is to install kubectl on a remote Jenkins server and assume that this gives it access to a local cluster. It does not. The CLI needs a valid cluster endpoint, credentials, and network connectivity.

In a GitOps architecture, Jenkins can publish the image and update the deployment configuration in Git, while Argo CD performs the cluster-side reconciliation.

This avoids requiring the Jenkins server to connect directly to the local cluster.

4.1 Check Jenkins

On the EC2 instance:

sudo systemctl status jenkins 

If the service is not running:

sudo journalctl -u jenkins -n 100 --no-pager 

Check the Java runtime if the Jenkins service reports startup or compatibility problems:

java -version 

The Java version must be compatible with the installed Jenkins release. Check the official Jenkins documentation for the requirements of the specific version being used.

4.2 Check Docker

docker --version docker info 

These commands confirm that the Docker client is installed and can communicate with the daemon under the current user's permissions.

When Jenkins builds Docker images, test access using the actual Jenkins service account where appropriate:

sudo -u jenkins docker info 

This command assumes a Linux installation with a service account named jenkins.

Do not automatically grant broad privileges to fix an access problem. Membership in the Docker group can effectively provide root-level access to the host. Investigate the specific error and choose a build architecture appropriate to the security requirements.

4.3 Check Minikube and Kubernetes

Run these commands on the machine where Minikube is installed:

minikube status kubectl cluster-info kubectl get nodes -o wide kubectl get pods -A 

Expected results include a running Minikube cluster, a reachable Kubernetes API, and a node reporting Ready.

[SCREENSHOT - MINIKUBE AND KUBERNETES STATUS HERE]

The exact output varies by Kubernetes and Minikube version.

5. How the CI/CD and GitOps Process Works

The complete delivery process has several logical stages.

Stage 1: Source checkout

Jenkins retrieves the application source code from GitHub.

A checkout stage might look like this:

stage('Checkout') { steps { git branch: 'main', url: 'https://github.com/OWNER/REPOSITORY.git' } } 

Replace OWNER/REPOSITORY with the actual repository path.

For a private repository, Jenkins needs a suitable credential. For a public repository, a read-only checkout may not require authentication.

The purpose of checkout is to ensure that the build uses a known source revision rather than files copied manually to the Jenkins server.

Stage 2: Build and test

For a Maven-based Spring Boot application:

mvn clean verify 

The clean goal removes previous build output. The verify lifecycle phase runs the configured checks leading up to verification.

A successful result means the build and configured verification steps passed. It does not prove that the application has no defects.

If the project uses the Maven Wrapper, prefer:

./mvnw clean verify 

The wrapper helps the project use the intended Maven version, subject to the availability of a compatible Java runtime.

Stage 3: Optional code analysis

SonarQube can be included to assess code quality and identify potential issues.

An illustrative command is:

mvn clean verify sonar:sonar \ -Dsonar.host.url="$SONAR_HOST_URL" \ -Dsonar.token="$SONAR_TOKEN" 

This requires a correctly configured SonarQube server, compatible scanner configuration, and a valid token.

Do not add this stage to a pipeline until the scanner, server URL, and credentials are configured. A code-quality stage is optional in the reference implementation and should only be described as completed if it was actually executed.

Stage 4: Build the Docker image

A Docker image packages the application into a standard format that can be stored and run in compatible container environments.

A simple example:

docker build \ -t horlacloud/spring-boot-demo:BUILD_TAG \ . 

Replace BUILD_TAG with a valid version identifier.

For repeatable releases, use a unique build number, release version, or commit-based tag rather than depending only on latest.

For example:

docker build \ -t horlacloud/spring-boot-demo:1.0.0 \ . 

This example assumes the Dockerfile is in the current build context. Adjust the path if it is stored elsewhere.

Stage 5: Publish the image

After building the image, Jenkins authenticates to Docker Hub and pushes it.

A typical manual command sequence is:

docker login docker push horlacloud/spring-boot-demo:1.0.0 

For an automated pipeline, credentials should be supplied securely through Jenkins Credentials rather than typed into a pipeline or stored in the source repository.

A successful push means the registry accepted the image. It does not prove that Kubernetes can pull it or that the application will start correctly.

Stage 6: Update deployment configuration

The GitOps configuration must reference the intended image.

For example:

containers: - name: spring-boot-app image: horlacloud/spring-boot-demo:1.0.0 

If Jenkins is expected to update this value automatically, the Jenkinsfile needs an explicit step to modify the GitOps repository and push the change using an appropriately restricted GitHub credential.

Publishing an image does not automatically update Kubernetes unless the delivery design includes an image-update mechanism or a corresponding configuration change.

Stage 7: Argo CD reconciliation

Argo CD monitors the configured Git repository and path.

When a new desired configuration is committed, Argo CD can detect the difference and synchronise the managed Kubernetes resources according to its application configuration.

This is the GitOps part of the workflow.

[SCREENSHOT - JENKINS PIPELINE STAGES HERE]

[SCREENSHOT - SUCCESSFUL DOCKER HUB IMAGE PUBLICATION HERE]

The build process packages the application into a container image that can be referenced by Kubernetes deployment configuration.

6. The Jenkins Credential Error That Took Hours to Resolve

One of the most valuable lessons from this project came from a Jenkins credential lookup failure.

The error was:

ERROR: Could not find credentials entry with ID 'github-credentials' 

The important detail is the wording: Jenkins could not find the credential entry with the requested ID.

This is different from a rejected token or an access-denied response.

6.1 What is a credential ID?

Jenkins stores credentials separately from the pipeline source code. Each saved credential has an identifier.

For example:

credentialsId: 'github-credentials' 

Here, github-credentials is the credential ID. It is not the token itself.

If the saved credential is named github-token, referencing github-credentials will not work unless a credential with that exact ID exists in the relevant credential store and scope.

6.2 Troubleshooting the problem

I approached the error by focusing on the stage that failed instead of repeatedly rebuilding the application.

The checks to perform are:

1. Identify the exact failed stage in the Jenkins console.
2. Inspect the credential ID referenced by the Jenkinsfile.
3. Open Jenkins Credentials and locate the corresponding entry.
4. Confirm the credential's type.
5. Check the credential's scope and availability to the job.
6. Verify that the credential has the permissions required for the GitHub operation.
7. Confirm that the repository URL and branch are correct.

The exact credential ID must match the identifier used in the Jenkinsfile.

The credential type must also match the pipeline binding. A Secret text credential is not the same as a username/password credential or an SSH private key.

6.3 Example: username and token

If the credential is stored as a username/password pair, an illustrative Jenkins binding is:

withCredentials([ usernamePassword( credentialsId: 'github-credentials', usernameVariable: 'GIT_USERNAME', passwordVariable: 'GIT_TOKEN' ) ]) { sh ''' set +x # Perform the authenticated operation here. ''' } 

The ID and credential type must match the saved Jenkins configuration.

For a Secret text credential, the binding would instead look like this:

withCredentials([ string( credentialsId: 'github-token', variable: 'GIT_TOKEN' ) ]) { sh ''' set +x # Use the token without printing it. ''' } 

These are reference patterns, not the exact final code from my environment.

6.4 Why secrets should not be stored in the Jenkinsfile

A Jenkinsfile is normally stored in version control. Hardcoding a token into it creates a risk of exposing that token through Git history, logs, pull requests, or copied source code.

The safer approach is to store secrets in Jenkins Credentials and reference them by ID.

Use narrowly scoped credentials, protect pipeline changes, and rotate exposed secrets promptly.

6.5 Three similar errors that require different fixes

This distinction saves time because each error points to a different part of the configuration.

[SCREENSHOT - THE SUCCESSFUL PIPELINE BUILD HERE]

By the 20th CI/CD pipeline build, I had worked through the integration problems and achieved a successful pipeline build.

7. Building a Spring Boot Application Container

A container image needs a Dockerfile that explains how the application is built or how its packaged artifact is executed.

There are two common approaches.

Option A: Build the JAR before creating the image

If Maven has already generated the JAR, a simple Dockerfile can copy that artifact into a Java runtime image.

FROM eclipse-temurin:21-jre WORKDIR /app COPY target/*.jar app.jar EXPOSE 8080 USER 10001 ENTRYPOINT ["java", "-jar", "/app/app.jar"] 

This is an illustrative example. The Java runtime version must match the application's requirements, and the build process must ensure that exactly the intended JAR is copied.

The EXPOSE instruction documents the expected container port; it does not publish the port by itself.

The non-root user improves the default security posture, provided the application can run with the required filesystem permissions.

Option B: Use a multi-stage Docker build

A multi-stage Dockerfile can compile the application in a build image and copy only the packaged artifact into the runtime image.

FROM maven:3.9-eclipse-temurin-21 AS build WORKDIR /workspace COPY pom.xml . COPY src ./src RUN mvn -B clean package -DskipTests FROM eclipse-temurin:21-jre WORKDIR /app COPY --from=build /workspace/target/*.jar app.jar EXPOSE 8080 USER 10001 ENTRYPOINT ["java", "-jar", "/app/app.jar"] 

This example assumes the Maven project is at the Docker build context root and produces one intended executable JAR.

Notice that this sample skips tests during packaging. In a complete CI/CD workflow, tests should run in a dedicated stage before image publication, and the pipeline should stop if they fail. Avoid using -DskipTests as a replacement for automated testing.

The runtime image does not need the entire Maven build environment, which can reduce the amount of tooling included in the final container.

Build and test locally

mvn clean verify docker build -t spring-boot-app:local . docker run --rm -p 8080:8080 spring-boot-app:local 

Then, in another terminal:

curl -i http://localhost:8080 

Use an actual application route if the root path is not defined.

8. Deploying the Application with Kubernetes

Kubernetes manages the desired state of containerised workloads. It schedules Pods, monitors workload state, and attempts to maintain the replica count specified in the Deployment.

The project used a Deployment and a Service for the Spring Boot application.

8.1 Deployment manifest

The confirmed GitOps configuration declares three replicas and the image horlacloud/spring-boot-demo:20.

An illustrative Deployment is:

apiVersion: apps/v1 kind: Deployment metadata: name: spring-boot-app labels: app: spring-boot-app spec: replicas: 3 selector: matchLabels: app: spring-boot-app template: metadata: labels: app: spring-boot-app spec: containers: - name: spring-boot-app image: horlacloud/spring-boot-demo:20 ports: - name: http containerPort: 8080 

This shows the core configuration. A production-oriented Deployment should normally also define appropriate probes, resource requests and limits, and a security context.

The replicas field is the desired number of Pods. Kubernetes attempts to maintain that number, but scheduling constraints, image pull failures, application crashes, and insufficient capacity can prevent all replicas from becoming ready.

The selector must match the labels on the Pod template. If the selector and template labels do not match, the Deployment will be invalid or fail to manage the intended Pods.

8.2 Service manifest

A Service provides a stable endpoint for matching Pods.

The project's Service is named spring-boot-app-service. A representative configuration is:

apiVersion: v1 kind: Service metadata: name: spring-boot-app-service spec: type: NodePort selector: app: spring-boot-app ports: - name: http port: 80 targetPort: 8080 protocol: TCP 

Here:

• port: 80 is the Service port.
• targetPort: 8080 is the port on the application container.
• type: NodePort exposes the Service through a port on the Kubernetes node.

The selector identifies the Pods that should receive traffic.

A Service can exist even when it has no ready endpoints. That is why it is important to inspect both the Service and the Pods behind it.

8.3 Validate the resources

kubectl get deployments kubectl get pods -o wide kubectl get services kubectl get endpoints spring-boot-app-service 

To inspect the Deployment in more detail:

kubectl describe deployment spring-boot-app 

To inspect an individual Pod:

kubectl describe pod POD_NAME kubectl logs POD_NAME 

Replace POD_NAME with the name returned by kubectl get pods.

[SCREENSHOT - KUBERNETES DEPLOYMENT AND POD STATUS HERE]

[SCREENSHOT - KUBERNETES SERVICE AND ENDPOINTS HERE]

Kubernetes was managing the application replicas and exposing them through a stable Service.

9. ConfigMaps, Secrets, and Application Configuration

Applications often need configuration that should not be embedded directly into the container image.

Examples include:

• Application mode or environment name.
• Log level.
• API endpoint URLs.
• Database connection settings.
• Authentication credentials.

Kubernetes provides ConfigMaps for non-sensitive configuration and Secrets for sensitive values.

9.1 ConfigMap example

apiVersion: v1 kind: ConfigMap metadata: name: spring-boot-config data: APP_ENV: "development" LOG_LEVEL: "INFO" 

A Deployment can expose these values to the container:

envFrom: - configMapRef: name: spring-boot-config 

This belongs under the container specification.

9.2 Secret example

A Secret can hold sensitive configuration such as a database password.

apiVersion: v1 kind: Secret metadata: name: database-credentials type: Opaque stringData: username: app_user password: REPLACE_WITH_A_SECRET 

This is a template only. Replace the placeholder using an appropriate secret-management process.

Do not commit real passwords, access tokens, or private keys to Git. Kubernetes Secret values are not automatically safe simply because they are stored in a Secret object; access controls and encryption-at-rest configuration also matter.

For production environments, consider a suitable external secret-management integration.

9.3 Why configuration should be separate

Separating configuration from application code makes it easier to promote the same image across environments without rebuilding it for every setting change.

It also reduces the risk of accidentally baking environment-specific values into a reusable image.

10. Persistent Volumes and Persistent Volume Claims

Persistent storage deserves special attention because container storage and application data have different lifecycles.

A container's writable filesystem is generally not intended to be the durable home for important data. If a container is replaced, its writable layer may be lost.

Kubernetes persistent storage provides a way to manage data independently of an individual container's lifecycle.

10.1 Understanding the storage components

PersistentVolume (PV): Represents a piece of storage available to a Kubernetes cluster.

PersistentVolumeClaim (PVC): A request for storage made by an application or workload.

StorageClass: Defines how storage can be provisioned. Depending on the cluster, it may support dynamic provisioning so that a PVC can trigger creation of a volume.

Pod volume mount: Connects the claimed storage to a path inside a container.

The relationship is:

Application Pod | v Volume Mount | v PersistentVolumeClaim | v PersistentVolume | v Underlying Storage 

When dynamic provisioning is enabled, a PVC may trigger the creation of a PV through its StorageClass.

10.2 When should persistent storage be used?

Persistent storage may be appropriate for:

• Database data directories.
• Application files that must survive Pod replacement.
• Durable uploads or other persistent application data.
• Workloads that explicitly require filesystem persistence.

It is generally not necessary for every application container.

For example, a stateless Spring Boot web application may not need a PVC if it stores no durable files locally and uses an external database or object storage.

The right storage design depends on the application's data requirements.

10.3 Inspect available StorageClasses

Before creating a PVC, check the cluster:

kubectl get storageclass kubectl describe storageclass 

The available StorageClasses depend on the Kubernetes environment and installed storage provisioners.

A Minikube installation may provide a default StorageClass, but this should be checked rather than assumed.

10.4 PVC example

This example requests 1 GiB of persistent storage:

apiVersion: v1 kind: PersistentVolumeClaim metadata: name: app-data spec: accessModes: - ReadWriteOnce resources: requests: storage: 1Gi 

This example leaves storageClassName unspecified so that the cluster's default StorageClass can be used when one is configured.

The claim may remain Pending if no suitable StorageClass or matching volume is available.

The access mode ReadWriteOnce generally allows a volume to be mounted read-write by a single node at a time; it does not necessarily mean only one Pod can use it. Actual access behaviour depends on the storage implementation.

10.5 Mount the PVC into a Pod

The claim can be mounted into a container through a Pod or Deployment specification.

apiVersion: apps/v1 kind: Deployment metadata: name: storage-demo spec: replicas: 1 selector: matchLabels: app: storage-demo template: metadata: labels: app: storage-demo spec: containers: - name: storage-demo image: busybox:1.36 command: - sh - -c - "while true; do date >> /data/timestamps.txt; sleep 30; done" volumeMounts: - name: app-data mountPath: /data volumes: - name: app-data persistentVolumeClaim: claimName: Source：https://dev.to/jamiu_cloud/building-a-cicd-and-gitops-platform-with-jenkins-docker-kubernetes-and-argo-cd-my-end-to-end-1nkg
My favorites

Related posts

Is there a nicer syntax for `if let ... Some(destructred_field) else None`?

I currently have: self.filter_map(|entry| { if let TocEntry::FirstTrackInfo { track, control, .. } = entry && control.is

0Rust Users Forum

Convert YAML to JSON in the Browser (and JSON to YAML) Without Uploading the File

If you keep configuration in YAML and hand it to something that wants JSON, the conversion happens constantly — in CI, i

0dev.to

Vercel agent-desktop: computer use for coding agents through the accessibility tree

Vercel Labs released agent-desktop this week, a free, open source tool that gives AI coding agents computer use on macOS

0dev.to

WARN Act and Job Search — How to Use the 60-Day Notice Period to Find Your Next Job

WARN Act and Job Search — How to Use the 60-Day Notice Period to Find Your Next Job A WARN notice is not a pink slip han

0dev.to

Node.js SMS Verification Code: Generating and Sending One-Time Passwords via API Integration

Node.js SMS Verification Code: Generating and Sending One-Time Passwords via API Integration Short answer: use Node.js c

0dev.to

Introducing Takeout Mender – A Free Tool to Recover Metadata from Google Takeout

Hey everyone! I've been working on a project called Takeout Mender, and I'm excited to share it here. Google Takeout exp

0dev.to