Writing Your First Kubernetes Controller in Go: Reconcile, Finalizers, and Mistakes to Avoid

A Kubernetes controller is deceptively simple: watch objects, call Reconcile, make the world match the spec. Then you ship it, and three weeks later someone reports that deleting a namespace hangs forever, your API server audit log shows 40 000 PATCH requests per minute from a single controller pod, and an orphaned bucket is still billing you in production.

All three failures come from the same three concepts: idempotency, finalizers, and status write discipline. This article builds a minimal but correct controller with controller-runtime, scaffolded by kubebuilder, that manages a CRD representing an external resource. We focus on the parts that actually break in production.

Context: why the reconcile loop is harder than it looks

Reconcile is level-triggered, not edge-triggered. You never receive "the user changed spec.size from 10 to 20". You receive "something about default/my-bucket may have changed; go look". The event may be a resync, a status update your own controller just wrote, a cache re-list after an API server restart, or a genuine user edit. It may arrive twice for the same change, or once for three changes.

Three consequences that drive the entire design:

1. Reconcile must be idempotent. Running it 100 times must produce the same cluster state as running it once, and must not create 100 external resources.
2. Reconcile must be restartable. The process can die between "create the external bucket" and "record the bucket ID in status". The next pass must recover without leaking.
3. Garbage collection only covers in-cluster objects. ownerReferences will clean up a Secret you created. Nothing in Kubernetes knows about an S3 bucket, a DNS record, or a row in an external IPAM. That is exactly what finalizers are for.

Scaffolding the project

mkdir bucket-operator && cd bucket-operator kubebuilder init \ --domain example.com \ --repo github.com/acme/bucket-operator \ --plugins go/v4 kubebuilder create api \ --group storage \ --version v1alpha1 \ --kind BucketClaim \ --resource --controller 

The API type. Note observedGeneration and the conditions slice — both are load-bearing, not decoration.

// api/v1alpha1/bucketclaim_types.go type BucketClaimSpec struct { // +kubebuilder:validation:MinLength=3 // +kubebuilder:validation:MaxLength=63 Region string `json:"region"` // +kubebuilder:validation:Enum=Retain;Delete // +kubebuilder:default=Delete ReclaimPolicy string `json:"reclaimPolicy,omitempty"` } type BucketClaimStatus struct { // BucketID is the opaque identifier returned by the storage backend. BucketID string `json:"bucketID,omitempty"` // ObservedGeneration is the .metadata.generation this status reflects. ObservedGeneration int64 `json:"observedGeneration,omitempty"` // +listType=map // +listMapKey=type Conditions []metav1.Condition `json:"conditions,omitempty"` } // +kubebuilder:object:root=true // +kubebuilder:subresource:status // +kubebuilder:printcolumn:name="Region",type=string,JSONPath=`.spec.region` // +kubebuilder:printcolumn:name="Ready",type=string,JSONPath=`.status.conditions[?(@.type=="Ready")].status` type BucketClaim struct { /* ... */ } 

The status subresource is non-negotiable: without it, every Update on status bumps metadata.generation and re-triggers your own controller. That is infinite loop number one.

The generated CRD can carry CEL validation so you reject bad specs at admission time instead of in Go:

apiVersion: apiextensions.k8s.io/v1 kind: CustomResourceDefinition metadata: name: bucketclaims.storage.example.com spec: group: storage.example.com scope: Namespaced names: kind: BucketClaim listKind: BucketClaimList plural: bucketclaims singular: bucketclaim versions: - name: v1alpha1 served: true storage: true subresources: status: {} schema: openAPIV3Schema: type: object properties: spec: type: object required: [region] x-kubernetes-validations: - rule: "self.region == oldSelf.region" message: "region is immutable" properties: region: type: string reclaimPolicy: type: string enum: [Retain, Delete] default: Delete status: type: object properties: bucketID: { type: string } observedGeneration: { type: integer, format: int64 } conditions: type: array items: type: object required: [type, status, lastTransitionTime, reason] properties: type: { type: string } status: { type: string } reason: { type: string } message: { type: string } observedGeneration: { type: integer, format: int64 } lastTransitionTime: { type: string, format: date-time } 

Immutability enforced by CEL removes an entire class of reconcile branches ("what do I do when the user moves the bucket to another region?").

The reconcile loop, step by step

const finalizerName = "storage.example.com/bucket-cleanup" func (r *BucketClaimReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) { log := logf.FromContext(ctx) var bc storagev1alpha1.BucketClaim if err := r.Get(ctx, req.NamespacedName, &bc); err != nil { // NotFound means the object is gone and fully finalized. Nothing to do. return ctrl.Result{}, client.IgnoreNotFound(err) } if !bc.DeletionTimestamp.IsZero() { return r.reconcileDelete(ctx, &bc) } // 1. Finalizer BEFORE any external side effect. if controllerutil.AddFinalizer(&bc, finalizerName) { if err := r.Update(ctx, &bc); err != nil { return ctrl.Result{}, err } // This Update generates a new event; stop here, the next pass continues. return ctrl.Result{}, nil } // 2. Idempotent provisioning, keyed on a deterministic name. name := bucketName(&bc) // e.g. fmt.Sprintf("%s-%s-%s", bc.Namespace, bc.Name, bc.UID[:8]) id, err := r.Storage.EnsureBucket(ctx, name, bc.Spec.Region) if err != nil { meta.SetStatusCondition(&bc.Status.Conditions, metav1.Condition{ Type: "Ready", Status: metav1.ConditionFalse, Reason: "BackendError", Message: err.Error(), ObservedGeneration: bc.Generation, }) _ = r.patchStatus(ctx, &bc) return ctrl.Result{}, err // exponential backoff from the workqueue } // 3. Owned in-cluster object: GC handles deletion, no finalizer needed. if err := r.ensureCredentialsSecret(ctx, &bc, id); err != nil { return ctrl.Result{}, err } // 4. Status last, and only status. bc.Status.BucketID = id bc.Status.ObservedGeneration = bc.Generation meta.SetStatusCondition(&bc.Status.Conditions, metav1.Condition{ Type: "Ready", Status: metav1.ConditionTrue, Reason: "Provisioned", ObservedGeneration: bc.Generation, }) if err := r.patchStatus(ctx, &bc); err != nil { return ctrl.Result{}, err } // Re-check the backend periodically: no watch exists on external systems. return ctrl.Result{RequeueAfter: 10 * time.Minute}, nil } 

Why the finalizer goes first

If you create the bucket and then add the finalizer, there is a window where the controller can crash, the user deletes the CR, the API server removes it immediately, and the bucket leaks. Adding the finalizer is a cheap, idempotent metadata write. Do it before you touch anything outside the cluster, always.

Why provisioning must be keyed on a deterministic name

EnsureBucket must be "create if absent, return existing otherwise" and derive its key from immutable CR data (namespace, name, UID). Never key on status.BucketID alone: if the process dies between the backend call and the status write, the next pass must still find the existing bucket instead of creating a second one. Reading the status you just wrote back from the cache is also unreliable — the informer cache is eventually consistent.

Patch, don't update, for status

func (r *BucketClaimReconciler) patchStatus(ctx context.Context, bc *storagev1alpha1.BucketClaim) error { // base is the object as it was read; MergeFrom computes a minimal patch. return r.Status().Patch(ctx, bc, client.MergeFrom(bc.DeepCopy())) } 

In practice, capture the base object right after Get (base := bc.DeepCopy()), mutate bc, then r.Status().Patch(ctx, bc, client.MergeFrom(base)). A merge patch that computes to an empty diff is a no-op on the API server: no write, no event, no loop. A blind Status().Update() on every reconcile rewrites resourceVersion every time and is the second classic infinite loop — made worse by lastTransitionTime if you rebuild conditions by hand instead of using meta.SetStatusCondition, which only touches the timestamp when the status actually flips.

Deletion: the finalizer path

func (r *BucketClaimReconciler) reconcileDelete(ctx context.Context, bc *storagev1alpha1.BucketClaim) (ctrl.Result, error) { if !controllerutil.ContainsFinalizer(bc, finalizerName) { return ctrl.Result{}, nil // someone else's finalizer, not our turn } if bc.Spec.ReclaimPolicy == "Delete" { err := r.Storage.DeleteBucket(ctx, bucketName(bc)) switch { case err == nil, errors.Is(err, backend.ErrNotFound): // Idempotent: already gone is success. default: r.Recorder.Eventf(bc, corev1.EventTypeWarning, "CleanupFailed", "bucket cleanup failed: %v", err) return ctrl.Result{}, err } } controllerutil.RemoveFinalizer(bc, finalizerName) if err := r.Update(ctx, bc); err != nil { return ctrl.Result{}, client.IgnoreNotFound(err) } return ctrl.Result{}, nil } 

Four rules for finalizers:

• Idempotent cleanup. "Resource not found in the backend" is a success, not an error. Otherwise an already-deleted bucket blocks the CR forever.
• Remove the finalizer only after cleanup succeeded. The order in the code above is the whole point.
• Never block forever. A finalizer that always errors makes the CR undeletable and, worse, makes its namespace stuck in Terminating. Expose the failure as an Event and a condition so an operator can see why, and support an escape hatch (for example, honour storage.example.com/skip-cleanup: "true" as an annotation).
• Check it is your finalizer. Other controllers (and the kubernetes namespace finalizer) add their own. Only ever touch your own string.

Emergency unblocking, for the runbook:

# Inspect what is holding the object oc get bucketclaim my-bucket -o jsonpath='{.metadata.finalizers}{"\n"}' # Last resort: strip the finalizer manually (this LEAKS the external resource) oc patch bucketclaim my-bucket --type=merge -p '{"metadata":{"finalizers":null}}' 

Wiring the controller and filtering events

func (r *BucketClaimReconciler) SetupWithManager(mgr ctrl.Manager) error { return ctrl.NewControllerManagedBy(mgr). For(&storagev1alpha1.BucketClaim{}, builder.WithPredicates(predicate.Or( predicate.GenerationChangedPredicate{}, predicate.AnnotationChangedPredicate{}, ))). Owns(&corev1.Secret{}). WithOptions(controller.Options{MaxConcurrentReconciles: 4}). Named("bucketclaim"). Complete(r) } 

GenerationChangedPredicate drops update events where metadata.generation is unchanged — that includes your own status writes. It does not drop deletions: when the API server marks an object for deletion while finalizers are pending, it bumps generation, so the delete-intent event still reaches you. The trade-off: it also drops periodic resync events, so never rely on resync for drift detection — use the explicit RequeueAfter in the success path instead.

MaxConcurrentReconciles > 1 means the same controller processes different objects in parallel, but the workqueue guarantees a given key is never reconciled concurrently with itself. Your Reconcile must still be safe against two different CRs touching the same backend.

Returning the right thing

Never return an error for a user mistake like an invalid region string: the workqueue will retry it forever, spam your logs, and inflate controller_runtime_reconcile_errors_total. Surface it in status.conditions and stop. Also, do not hand-roll time.Sleep inside Reconcile — you are blocking a worker goroutine; use RequeueAfter.

Conflicts (apierrors.IsConflict) are normal under optimistic concurrency. Just return the error and let the backoff re-read a fresh object; do not wrap your whole reconcile in retry.RetryOnConflict.

Testing against a real API server

envtest runs a real kube-apiserver and etcd binary, which is the only way to validate finalizer semantics and the status subresource.

make test # kubebuilder wires setup-envtest + ginkgo make manifests generate make install # applies config/crd into the cluster make run # runs the manager out-of-cluster against your kubeconfig 

Two tests worth writing on day one: delete the CR while the fake backend returns an error, and assert the object still exists with its finalizer; then make the backend succeed and assert the object disappears.

Security best practices

Least-privilege RBAC. Kubebuilder markers generate the role, so keep them narrow — no groups=* or resources=*:

// +kubebuilder:rbac:groups=storage.example.com,resources=bucketclaims,verbs=get;list;watch;update;patch // +kubebuilder:rbac:groups=storage.example.com,resources=bucketclaims/status,verbs=get;update;patch // +kubebuilder:rbac:groups=storage.example.com,resources=bucketclaims/finalizers,verbs=update // +kubebuilder:rbac:groups=core,resources=secrets,verbs=get;list;watch;create;update;patch;delete // +kubebuilder:rbac:groups=core,resources=events,verbs=create;patch 

Note the dedicated /finalizers subresource permission: it is required as soon as OwnerReferencesPermissionEnforcement is enabled, which is the default on OpenShift.

Scope the cache. A cluster-wide watch on Secrets pulls every secret in the cluster into controller memory. Restrict it:

mgr, err := ctrl.NewManager(cfg, ctrl.Options{ Cache: cache.Options{ ByObject: map[client.Object]cache.ByObject{ &corev1.Secret{}: { Label: labels.SelectorFromSet(labels.Set{"app.kubernetes.io/managed-by": "bucket-operator"}), }, }, }, }) 

Harden the pod. On OpenShift, target the restricted-v2 SCC: do not pin runAsUser, let the namespace range assign it.

apiVersion: apps/v1 kind: Deployment metadata: name: bucket-operator-controller-manager namespace: bucket-operator-system spec: replicas: 2 selector: matchLabels: { control-plane: controller-manager } template: metadata: labels: { control-plane: controller-manager } spec: serviceAccountName: bucket-operator-controller-manager automountServiceAccountToken: true securityContext: runAsNonRoot: true seccompProfile: { type: RuntimeDefault } containers: - name: manager image: quay.io/acme/bucket-operator:v0.1.0 args: ["--leader-elect", "--health-probe-bind-address=:8081"] securityContext: allowPrivilegeEscalation: false readOnlyRootFilesystem: true capabilities: { drop: ["ALL"] } resources: requests: { cpu: 10m, memory: 64Mi } limits: { memory: 256Mi } livenessProbe: httpGet: { path: /healthz, port: 8081 } readinessProbe: httpGet: { path: /readyz, port: 8081 } 

Leader election (--leader-elect with replicas: 2) is a correctness control, not just HA: two active managers racing on the same external backend will double-provision. It needs leases in coordination.k8s.io, namespace-scoped.

Never log secret material. logf.FromContext(ctx) is structured — log the object key and the backend resource ID, never credentials or full object dumps. Credentials belong in a Secret with an ownerReference pointing at the CR, so they are garbage-collected automatically and never written to status.

Validate at admission, not in Go. CEL rules in the CRD (and a validating webhook with a service-ca managed certificate on OpenShift) block bad input before it ever reaches a worker. A NetworkPolicy restricting egress to your backend endpoint closes the loop.

Key takeaways

• Reconcile is level-triggered: make it idempotent, restartable, and keyed on deterministic names derived from immutable fields.
• Add the finalizer before the first external side effect; remove it only after cleanup succeeds, and treat "already gone" as success.
• Use the status subresource with Status().Patch(client.MergeFrom(base)) and meta.SetStatusCondition — blind Update calls are the most common infinite loop.
• Filter events with GenerationChangedPredicate, but keep an explicit RequeueAfter to detect drift in systems you cannot watch.
• Return an error only for transient failures; report user errors as conditions so the workqueue does not retry them forever.
• A finalizer that can never complete is a namespace that can never terminate: always expose the failure and provide an escape hatch.
• Lock down RBAC, scope the informer cache, and run the manager as a non-root, read-only, capability-dropped pod.