Kubernetes GitOps with FluxCD - Part 4 - Helm Chart Automation - Kube Prometheus Stack

In our previous post, we explored how Flux CD enables automated image updates while maintaining GitOps principles. Building on our foundation of basic Flux CD setup, SOPS-based secret management, and image update automation, this article focuses on Helm Chart Automation - a sophisticated capability that further enhances your GitOps workflow by declaratively managing Helm releases.

Helm serves as Kubernetes' package manager, simplifying the deployment of complex applications through charts that bundle related Kubernetes resources. While Helm itself is powerful, managing Helm releases manually contradicts GitOps principles of full automation and declarative configuration. Flux CD bridges this gap with its Helm Controller, allowing teams to define, deploy, and update Helm releases through Git-based declarations rather than imperative commands.

This article demonstrates advanced techniques for automating Helm chart deployments with Flux CD, using the Kube Prometheus Stack as a practical example. We'll explore managing chart dependencies, version control, and environment-specific configurations while adhering to GitOps best practices.

1. Setup Helm Source and Release

First, let's define a Helm Repository and Release at cluster/default/kps-helm.yaml

apiVersion: v1 kind: Namespace metadata: name: monitoring --- apiVersion: source.toolkit.fluxcd.io/v1 kind: HelmRepository metadata: name: prometheus-comminty namespace: monitoring spec: interval: 30m url: https://prometheus-community.github.io/helm-charts --- apiVersion: helm.toolkit.fluxcd.io/v2 kind: HelmRelease metadata: name: kube-prometheus-stack namespace: monitoring spec: releaseName: kube-prometheus-stack interval: 10m chart: spec: chart: kube-prometheus-stack version: "65.8.1" interval: 10m sourceRef: kind: HelmRepository name: prometheus-comminty namespace: monitoring install: crds: CreateReplace upgrade: crds: CreateReplace values: grafana: defaultDashboardsTimezone: "UTC+5:30" prometheus: prometheusSpec: ruleSelector: {} ruleNamespaceSelector: {} ruleSelectorNilUsesHelmValues: false serviceMonitorSelector: {} serviceMonitorNamespaceSelector: {} serviceMonitorSelectorNilUsesHelmValues: false podMonitorSelector: {} podMonitorNamespaceSelector: {} podMonitorSelectorNilUsesHelmValues: false retention: 30d enableOTLPReceiver: true enableRemoteWriteReceiver: true enableFeatures: [otlp-write-receiver]

FluxCD is capable of managing Custom Resource Definitions (CRDs), which is why we've configured crds: CreateReplace. This directive instructs FluxCD to automatically create and replace CRDs during subsequent upgrades—a capability we'll validate later in this post.

Let's push these changes and monitor the reconciliation process to apply this Helm release:

flux -n monitoring events --watch LAST SEEN TYPE REASON OBJECT MESSAGE 48s Normal HelmChartCreated HelmRelease/kube-prometheus-stack Created HelmChart/monitoring/monitoring-kube-prometheus-stack with SourceRef 'HelmRepository/monitoring/prometheus-comminty' 47s Normal NoSourceArtifact HelmChart/monitoring-kube-prometheus-stack no artifact available for HelmRepository source 'prometheus-comminty' 45s Normal ChartPullSucceeded HelmChart/monitoring-kube-prometheus-stack pulled 'kube-prometheus-stack' chart with version '65.8.1' 44s Normal ArtifactUpToDate HelmChart/monitoring-kube-prometheus-stack artifact up-to-date with remote revision: '65.8.1' 47s Normal NewArtifact HelmRepository/prometheus-comminty stored fetched index of size 5.487MB from 'https://prometheus-community.github.io/helm-charts' 0s Normal InstallSucceeded HelmRelease/kube-prometheus-stack Helm install succeeded for release monitoring/kube-prometheus-stack.v1 with chart kube-prometheus-stack@65.8.1

Next, let's access Grafana to verify the deployment:

kubectl -n monitoring port-forward services/kube-prometheus-stack-grafana 3000:80 Forwarding from 127.0.0.1:3000 -> 3000 Forwarding from [::1]:3000 -> 3000

Let's confirm that the default Kubernetes metrics dashboards are functioning correctly:

As we can observe, everything is operational and properly configured.

2. Upgrade helm release

We initially configured our Helm release at version 65.8.1. At the time of writing this post, the latest available version is 69.5.2.

According to the official release documentation, one would typically need to manually upgrade the Custom Resource Definitions before performing a Helm upgrade. However, since FluxCD can manage this process automatically, let's put this feature to the test.

Our approach will be methodical: first, we'll capture the current CRD state in the cluster, then upgrade the Helm release, and finally compare the new CRD definitions to validate the upgrade process.

Let's begin by fetching the current CRD:

kubectl describe crd prometheuses.monitoring.coreos.com > beforeupgrade.yaml head beforeupgrade.yaml Name: prometheuses.monitoring.coreos.com Namespace: Labels: helm.toolkit.fluxcd.io/name=kube-prometheus-stack helm.toolkit.fluxcd.io/namespace=monitoring Annotations: controller-gen.kubebuilder.io/version: v0.16.1 operator.prometheus.io/version: 0.77.2 API Version: apiextensions.k8s.io/v1 Kind: CustomResourceDefinition Metadata: Creation Timestamp: 2025-02-27T07:49:22Z

For the next step, we'll update our Helm Release manifest to version 69.5.2 and push these changes to our Git repository. Then we'll monitor the reconciliation process:

flux -n monitoring events --watch LAST SEEN TYPE REASON OBJECT MESSAGE 14s Normal ChartPullSucceeded HelmChart/monitoring-kube-prometheus-stack pulled 'kube-prometheus-stack' chart with version '69.5.2' 21m Normal NewArtifact HelmRepository/prometheus-comminty stored fetched index of size 5.487MB from 'https://prometheus-community.github.io/helm-charts' 0s Normal UpgradeSucceeded HelmRelease/kube-prometheus-stack Helm upgrade succeeded for release monitoring/kube-prometheus-stack.v2 with chart kube-prometheus-stack@69.5.2

Now, let's examine if the CRDs were successfully updated during the process:

kubectl describe crd prometheuses.monitoring.coreos.com > afterupgrade.yaml head afterupgrade.yaml Name: prometheuses.monitoring.coreos.com Namespace: Labels: helm.toolkit.fluxcd.io/name=kube-prometheus-stack helm.toolkit.fluxcd.io/namespace=monitoring Annotations: controller-gen.kubebuilder.io/version: v0.17.1 operator.prometheus.io/version: 0.80.1 API Version: apiextensions.k8s.io/v1 Kind: CustomResourceDefinition Metadata: Creation Timestamp: 2025-02-27T07:49:22Z

The results confirm the upgrade was successful. We can observe that the controller-gen version has been updated from v0.16.1 to v0.17.1, and the Prometheus operator version has been upgraded from 0.77.2 to 0.80.1.

Additionally, we can verify that all dependencies have been upgraded as well (as seen in the Grafana version displayed in the screenshot below):