Most platform teams end up with the same artifact graveyard: a dozen Helm charts, a values-prod.yaml nobody dares to diff, and a Jenkins job that runs helm upgrade --install whenever someone remembers. It works on day 1. It collapses on day 2, when someone edits a Deployment by hand, when a certificate needs rotating, when a schema migration must run before the rollout, or when a namespace is deleted and leaves orphaned external resources behind.
An Operator fixes the structural problem: it is a control loop, not a one-shot templating engine. And with the Ansible Operator SDK you do not need to write a line of Go. If you already have Ansible roles, you can ship a production-grade Operator, installable through OLM, in a single working day.
This article walks through the whole chain: CRD design, the reconciliation role, the watches.yaml, the bundle, the ClusterServiceVersion, and the OLM install on an OpenShift cluster — plus the RBAC and SCC traps that will bite you in a hardened cluster.
Why an Operator and not another Helm chart
Helm renders templates and applies them. That is a push model: the desired state exists only at the moment of the helm upgrade. Between two pipeline runs, nothing reconciles anything. If an SRE scales a StatefulSet manually at 3 a.m., the drift persists until the next release.
An Operator inverts the model. The desired state lives in a custom resource stored in etcd, and a controller continuously compares it to the actual state. Three concrete consequences:
- Drift correction is free. Any modification to a child object triggers a reconcile through the owner reference watch.
- Day-2 verbs become first-class API. Backup, restore, version upgrade, credential rotation — they are fields in a CRD spec, not pipeline parameters.
- Deletion is handled. Finalizers let you deregister a DNS entry, drop a database, or revoke a Vault role before the namespace disappears.
The Ansible flavour of the Operator SDK gives you this reconcile loop while your business logic stays in YAML tasks executed by ansible-runner. The trade-off is honest: slower reconciles than Go (a few hundred milliseconds of Ansible startup per run), no typed client, no deep caching. For an internal application managed by a platform team, that trade-off is almost always worth it.
Prerequisites
operator-sdk version # operator-sdk version: "v1.37.0", commit: "...", kubernetes version: "v1.30.0", go version: "go1.22" oc version --client oc whoami -c podman login quay.io You need cluster-admin (CRDs are cluster-scoped), a registry the cluster can pull from, and OpenShift 4.12+. Everything below was validated on 4.16.
Scaffolding the project
mkdir internalapp-operator && cd internalapp-operator operator-sdk init \ --plugins=ansible \ --domain acme.io \ --owner "Platform Team" \ --project-name internalapp-operator operator-sdk create api \ --group apps \ --version v1alpha1 \ --kind InternalApp \ --generate-role You now have roles/internalapp/, watches.yaml, config/ (Kustomize bases for CRD, RBAC, manager) and a Makefile that drives the whole build.
Designing a real CRD, not a free-form blob
The scaffolded CRD ships with x-kubernetes-preserve-unknown-fields: true. Never ship that to users: there is no validation, no oc explain, and no safe evolution path. Write a structural schema instead.
# config/crd/bases/apps.acme.io_internalapps.yaml apiVersion: apiextensions.k8s.io/v1 kind: CustomResourceDefinition metadata: name: internalapps.apps.acme.io spec: group: apps.acme.io names: kind: InternalApp listKind: InternalAppList plural: internalapps singular: internalapp shortNames: [iapp] scope: Namespaced versions: - name: v1alpha1 served: true storage: true subresources: status: {} additionalPrinterColumns: - name: Replicas type: integer jsonPath: .spec.replicas - name: Version type: string jsonPath: .spec.image.tag - name: Ready type: string jsonPath: .status.conditions[?(@.type=="Ready")].status - name: Age type: date jsonPath: .metadata.creationTimestamp schema: openAPIV3Schema: type: object properties: spec: type: object required: [image] properties: replicas: type: integer minimum: 0 maximum: 20 default: 2 image: type: object required: [repository, tag] properties: repository: type: string tag: type: string pattern: '^v[0-9]+\.[0-9]+\.[0-9]+$' route: type: object properties: enabled: type: boolean default: false host: type: string databaseSecretRef: type: object required: [name] properties: name: type: string backup: type: object properties: schedule: type: string pattern: '^(\S+\s+){4}\S+$' retentionDays: type: integer minimum: 1 default: 7 status: type: object x-kubernetes-preserve-unknown-fields: true Two things matter here. First, status keeps preserve-unknown-fields because the operator_sdk.util.k8s_status module writes an open-ended condition structure. Second, the regex on image.tag is your first line of defence against someone deploying :latest in production.
The reconciliation role
The role receives the CR spec as Ansible variables (snake-cased by default: database_secret_ref, retention_days). Keep it strictly idempotent, and never use command or shell where a module exists.
# roles/internalapp/tasks/main.yml --- - name: Resolve the database secret kubernetes.core.k8s_info: api_version: v1 kind: Secret name: "{{ database_secret_ref.name }}" namespace: "{{ ansible_operator_meta.namespace }}" register: db_secret no_log: true - name: Fail fast when the secret is missing ansible.builtin.fail: msg: "Secret {{ database_secret_ref.name }} not found in {{ ansible_operator_meta.namespace }}" when: db_secret.resources | length == 0 - name: Apply the application workload kubernetes.core.k8s: state: present apply: true definition: "{{ lookup('template', item) | from_yaml }}" wait: true wait_timeout: 300 loop: - deployment.yaml.j2 - service.yaml.j2 - pdb.yaml.j2 - name: Expose the application through a Route kubernetes.core.k8s: state: "{{ 'present' if route.enabled | default(false) else 'absent' }}" definition: "{{ lookup('template', 'route.yaml.j2') | from_yaml }}" - name: Schedule the logical backup kubernetes.core.k8s: state: "{{ 'present' if backup.schedule is defined else 'absent' }}" definition: "{{ lookup('template', 'backup-cronjob.yaml.j2') | from_yaml }}" - name: Publish status operator_sdk.util.k8s_status: api_version: apps.acme.io/v1alpha1 kind: InternalApp name: "{{ ansible_operator_meta.name }}" namespace: "{{ ansible_operator_meta.namespace }}" status: observedGeneration: "{{ ansible_operator_meta.annotations['acme.io/generation'] | default(0) }}" conditions: - type: Ready status: "True" reason: ReconcileSucceeded message: "Deployment reconciled at version {{ image.tag }}" The Deployment template must carry owner references so that garbage collection and the watch mechanism work:
# roles/internalapp/templates/deployment.yaml.j2 apiVersion: apps/v1 kind: Deployment metadata: name: {{ ansible_operator_meta.name }} namespace: {{ ansible_operator_meta.namespace }} labels: app.kubernetes.io/name: {{ ansible_operator_meta.name }} app.kubernetes.io/managed-by: internalapp-operator spec: replicas: {{ replicas | default(2) }} selector: matchLabels: app.kubernetes.io/name: {{ ansible_operator_meta.name }} template: metadata: labels: app.kubernetes.io/name: {{ ansible_operator_meta.name }} spec: securityContext: runAsNonRoot: true seccompProfile: type: RuntimeDefault containers: - name: app image: "{{ image.repository }}:{{ image.tag }}" imagePullPolicy: IfNotPresent envFrom: - secretRef: name: {{ database_secret_ref.name }} ports: - containerPort: 8080 securityContext: allowPrivilegeEscalation: false capabilities: drop: ["ALL"] resources: requests: cpu: 100m memory: 256Mi limits: memory: 512Mi readinessProbe: httpGet: path: /healthz port: 8080 The SDK injects owner references automatically for resources created by the kubernetes.core.k8s module when manageStatus and the ownership mechanism are active — but the apply: true path requires the objects to live in the same namespace as the CR. For cluster-scoped children, set ownerReferences explicitly or clean them up in the finalizer.
watches.yaml: reconcile period and finalizer
# watches.yaml --- - version: v1alpha1 group: apps.acme.io kind: InternalApp role: internalapp reconcilePeriod: 5m manageStatus: true watchDependentResources: true snakeCaseParameters: true finalizer: name: apps.acme.io/deprovision role: internalapp-deprovision The default reconcilePeriod for Ansible-based Operators is 10 hours. For drift correction on an internal app, drop it to a few minutes — but remember every tick starts an ansible-runner process. With 300 CRs and a 1-minute period, you will saturate the manager. Measure before tuning, and raise --max-concurrent-reconciles through the manager args rather than hammering the period.
The finalizer role is where day-2 deletion logic belongs: dropping a schema, deregistering from an external service registry, deleting an S3 bucket. Make it tolerant of already-deleted resources, otherwise a failing finalizer will wedge the namespace in Terminating forever.
Build, run and iterate
Local run against a live cluster, without deploying the manager:
export KUBECONFIG=~/.kube/config make install # applies the CRD only make run # runs ansible-operator locally against the cluster Then in another shell:
oc new-project demo-internalapp oc create secret generic app-db --from-literal=DB_HOST=pg.internal --from-literal=DB_USER=app oc apply -f - <<'EOF' apiVersion: apps.acme.io/v1alpha1 kind: InternalApp metadata: name: billing namespace: demo-internalapp spec: replicas: 3 image: repository: quay.io/acme/billing tag: v2.4.1 route: enabled: true databaseSecretRef: name: app-db backup: schedule: "0 2 * * *" retentionDays: 14 EOF oc get internalapps -n demo-internalapp # NAME REPLICAS VERSION READY AGE # billing 3 v2.4.1 True 42s When satisfied, build and push the Operator image. On OpenShift, base it on the Red Hat–supported image rather than the upstream one:
# Dockerfile FROM registry.redhat.io/openshift4/ose-ansible-operator:v4.16 COPY requirements.yml ${HOME}/requirements.yml RUN ansible-galaxy collection install -r ${HOME}/requirements.yml --force \ && chmod -R ug+rwx ${HOME}/.ansible COPY watches.yaml ${HOME}/watches.yaml COPY roles/ ${HOME}/roles/ COPY playbooks/ ${HOME}/playbooks/ export IMG=quay.io/acme/internalapp-operator:0.1.0 make docker-build docker-push IMG=$IMG Packaging for OLM
OLM is what turns a Deployment into an installable, upgradable, discoverable operator. The bundle is an OCI image containing the CRDs, the ClusterServiceVersion and metadata.
make bundle IMG=$IMG VERSION=0.1.0 CHANNELS=stable DEFAULT_CHANNEL=stable make bundle-build bundle-push BUNDLE_IMG=quay.io/acme/internalapp-operator-bundle:0.1.0 operator-sdk bundle validate quay.io/acme/internalapp-operator-bundle:0.1.0 \ --select-optional suite=operatorframework The generated CSV needs curation. The parts that matter:
# bundle/manifests/internalapp-operator.clusterserviceversion.yaml (excerpt) apiVersion: operators.coreos.com/v1alpha1 kind: ClusterServiceVersion metadata: name: internalapp-operator.v0.1.0 annotations: categories: "Application Runtime" containerImage: quay.io/acme/internalapp-operator:0.1.0 support: platform-team@acme.io alm-examples: |- [{"apiVersion":"apps.acme.io/v1alpha1","kind":"InternalApp", "metadata":{"name":"billing"}, "spec":{"replicas":3,"image":{"repository":"quay.io/acme/billing","tag":"v2.4.1"}, "databaseSecretRef":{"name":"app-db"}}}] spec: displayName: InternalApp Operator version: 0.1.0 maturity: alpha installModes: - type: OwnNamespace supported: true - type: SingleNamespace supported: true - type: MultiNamespace supported: true - type: AllNamespaces supported: false customresourcedefinitions: owned: - name: internalapps.apps.acme.io version: v1alpha1 kind: InternalApp displayName: Internal Application description: An internally packaged application with day-2 automation. Setting AllNamespaces: false is a deliberate blast-radius decision: the Operator will only ever be granted namespaced RBAC.
For a durable install, publish a file-based catalog and point a CatalogSource at it:
apiVersion: operators.coreos.com/v1alpha1 kind: CatalogSource metadata: name: acme-operators namespace: openshift-marketplace spec: sourceType: grpc image: quay.io/acme/acme-catalog:v4.16 displayName: ACME Internal Operators publisher: Platform Team updateStrategy: registryPoll: interval: 30m --- apiVersion: v1 kind: Namespace metadata: name: internalapp-operator --- apiVersion: operators.coreos.com/v1 kind: OperatorGroup metadata: name: internalapp-operator namespace: internalapp-operator spec: targetNamespaces: - demo-internalapp --- apiVersion: operators.coreos.com/v1alpha1 kind: Subscription metadata: name: internalapp-operator namespace: internalapp-operator spec: channel: stable name: internalapp-operator source: acme-operators sourceNamespace: openshift-marketplace installPlanApproval: Manual startingCSV: internalapp-operator.v0.1.0 oc apply -f olm/ oc get installplan -n internalapp-operator oc patch installplan install-xxxxx -n internalapp-operator \ --type merge -p '{"spec":{"approved":true}}' oc get csv -n internalapp-operator -w # internalapp-operator.v0.1.0 InternalApp Operator 0.1.0 Succeeded For a quick dev loop, operator-sdk run bundle quay.io/acme/internalapp-operator-bundle:0.1.0 -n internalapp-operator creates an ephemeral catalog and subscription in one shot; operator-sdk cleanup internalapp-operator removes it.
Security best practices
Trim the generated RBAC. operator-sdk create api writes a permissive config/rbac/role.yaml with wildcards. Replace it with exactly the verbs your role uses. A wildcard here becomes a cluster-wide privilege escalation path once the CSV is installed, because OLM creates the ServiceAccount and binds it for you.
rules: - apiGroups: ["apps.acme.io"] resources: ["internalapps", "internalapps/status", "internalapps/finalizers"] verbs: ["get", "list", "watch", "create", "update", "patch", "delete"] - apiGroups: ["apps"] resources: ["deployments"] verbs: ["get", "list", "watch", "create", "update", "patch", "delete"] - apiGroups: [""] resources: ["services", "configmaps"] verbs: ["get", "list", "watch", "create", "update", "patch", "delete"] - apiGroups: [""] resources: ["secrets"] verbs: ["get", "list", "watch"] - apiGroups: ["route.openshift.io"] resources: ["routes", "routes/custom-host"] verbs: ["get", "list", "watch", "create", "update", "patch", "delete"] - apiGroups: ["batch"] resources: ["cronjobs"] verbs: ["get", "list", "watch", "create", "update", "patch", "delete"] Note that secrets is read-only. An Operator that can create Secrets cluster-wide can mint ServiceAccount tokens indirectly in many designs — avoid it unless you truly need it.
Never put secrets in the CR spec. A spec.databasePassword ends up in etcd unencrypted-by-default, in oc get -o yaml, in GitOps repositories, and in ansible-runner artifacts on disk. Use a secretRef and read it at reconcile time with no_log: true.
Keep the manager under restricted-v2. The manager Deployment generated by the SDK may still carry legacy fields. Enforce explicitly:
# config/manager/manager.yaml (excerpt) securityContext: runAsNonRoot: true seccompProfile: type: RuntimeDefault containers: - name: manager securityContext: allowPrivilegeEscalation: false readOnlyRootFilesystem: false # ansible-runner needs /tmp writes capabilities: drop: ["ALL"] env: - name: ANSIBLE_GATHERING value: explicit - name: ANSIBLE_DEBUG_LOGS value: "false" resources: requests: cpu: 100m memory: 256Mi limits: memory: 768Mi ANSIBLE_DEBUG_LOGS=true dumps full task results — including resolved Secret content — into the pod logs. Never leave it on outside a debugging session.
Pin and sign images. Reference the Operator image by digest in the CSV, and gate it with a ClusterImagePolicy or Sigstore verification if your cluster enforces it. A floating tag in a CSV means OLM can silently run different code after a registry push.
Use installPlanApproval: Manual in production. Automatic upgrades of a homegrown Operator mean an untested reconcile loop can roll across every tenant namespace at 2 a.m.
Restrict the Operator namespace with a NetworkPolicy. The manager only needs egress to the API server and your registry; deny the rest.
Validate at the API boundary. CEL validation rules (x-kubernetes-validations) in the CRD are cheaper and safer than defensive fail tasks, because they reject bad input before anything reconciles.
Debugging the reconcile loop
oc logs -n internalapp-operator deploy/internalapp-operator-controller-manager -c manager -f oc get events -n demo-internalapp --sort-by=.lastTimestamp oc describe internalapp billing -n demo-internalapp Failures appear as a Failure condition on the CR with the Ansible task name and message. For d...
