In a traditional virtualization world, provisioning a virtual machine often meant downloading a 15GB .iso file, mounting it through an administrative console, clicking through an OS installer wizard, and running post-install scripts.

In a cloud-native platform like OpenShift Virtualization, that manual approach is unacceptable.

We need VMs to spin up in seconds—just like containers—using pre-baked, customized Golden Images (Boot Sources) that update automatically on a schedule.

In this third part of our series, we will walk through the end-to-end production workflow of building custom golden OS images, pushing them to the OpenShift Internal Registry, and automating recurring imports with CDI (Containerized Data Importer) and DataImportCron.

🏗️ Architecture: The Automated Boot Source Pipeline

The automated pipeline consists of three core stages:

  1. Packaging: Wrapping a raw/qcow2 virtual disk into an ultra-lightweight containerDisk.
  2. Distribution: Storing the image inside the secured OpenShift Internal Registry.
  3. Import & Consumption: CDI continuously pulls updates, populates a DataVolume, and registers a ready-to-use DataSource for instant VM cloning.

🛠️ Stage 1: Build the containerDisk from Scratch

A containerDisk is simply an OCI container image containing a disk image located at /disk/. To keep it minimal, secure, and disconnected-friendly, we build it directly FROM scratch without pulling heavy base images like UBI.

1. Prepare File Permissions & Containerfile

Before building, ensure the host qcow2 file has read-only permissions (0440), and assign ownership to UID 107:107 (the internal QEMU user inside KubeVirt pods):

chmod 0440 custom-disk.qcow2 cat << 'EOF' > Containerfile FROM scratch ADD --chown=107:107 custom-disk.qcow2 /disk/ EOF

🚀 Stage 2: Push to the OpenShift Internal Registry

To push images without relying on external registries, we leverage OpenShift's built-in registry.

1. Expose the Internal Registry Route 

oc patch configs.imageregistry.operator.openshift.io/cluster \ --type=merge -p '{"spec":{"defaultRoute":true}}' export REGISTRY_ROUTE=$(oc get route default-route -n openshift-image-registry --template='{{ .spec.host }}') echo "Internal Registry: $REGISTRY_ROUTE"

2. Login and Push 

podman login --tls-verify=false -u kubeadmin -p $(oc whoami -t) $REGISTRY_ROUTE podman build -t $REGISTRY_ROUTE/openshift-virtualization-os-images/custom-os-disk:latest . podman push --tls-verify=false $REGISTRY_ROUTE/openshift-virtualization-os-images/custom-os-disk:latest

🔐 Stage 3: Configure Authentication & TLS Certificates

When the CDI importer pod pulls the image from the internal service address (image-registry.openshift-image-registry.svc:5000), it requires credentials and cluster trust:

1. Create the Pull Secret 

oc create secret generic kubeadmin-reg-secret \ --from-literal=accessKeyId=kubeadmin \ --from-literal=secretKey=$(oc whoami -t) \ -n openshift-virtualization-os-images

2. Extract Cluster CA for TLS Verification

To prevent x509 certificate errors:

oc extract configmap/openshift-service-ca.crt \ -n openshift-virtualization-os-images --to=- > service-ca.crt oc create configmap internal-registry-ca \ --from-file=ca.crt=service-ca.crt \ -n openshift-virtualization-os-images

⏰ Stage 4: Deploy the Automated DataImportCron

Now we wire CDI to monitor the registry and automatically update a DataSource.

Notice the crucial annotation cdi.kubevirt.io/storage.bind.immediate.requested: "true", which forces storage allocation immediately rather than getting stuck in WaitForFirstConsumer:

cat << 'EOF' | oc apply -f - apiVersion: cdi.kubevirt.io/v1beta1 kind: DataImportCron metadata: name: custom-os-image-cron namespace: openshift-virtualization-os-images annotations: cdi.kubevirt.io/storage.bind.immediate.requested: "true" spec: garbageCollect: Outdated importsToKeep: 2 managedDataSource: custom-os-datasource retentionPolicy: None schedule: "*/15 * * * *" template: spec: source: registry: pullMethod: pod url: "docker://image-registry.openshift-image-registry.svc:5000/openshift-virtualization-os-images/custom-os-disk:latest" secretRef: kubeadmin-reg-secret certConfigMap: internal-registry-ca storage: resources: requests: storage: 15Gi EOF

Verify Importer Status 

oc get dataimportcron,dv,pvc,datasource -n openshift-virtualization-os-images

Wait until the DataSource reports READY: True.

🚀 Stage 5: Spin Up a VM in 5 Seconds!

With our managed DataSource active, developers can spin up a production VM with a single CLI command using fast CSI cloning:

virtctl create vm --name production-rhel-vm \ --volume-datasource src:openshift-virtualization-os-images/custom-os-datasource \ | oc apply -f -

⚠️ Battle-Tested Gotchas & Troubleshooting Guide

Here are the top 5 errors engineers face when working with custom boot sources and CDI:

Common Error Root Cause Proven Fix
error: token is not available Relying on stale KUBECONFIG without active login. Always run oc login -u kubeadmin before generating tokens.
Permission denied on VM boot Disk file inside container not owned by user 107. Set chmod 0440 and ADD --chown=107:107 in Containerfile.
x509: certificate signed by unknown authority Importer pod does not trust internal service CA. Extract openshift-service-ca.crt and bind via certConfigMap.
pullMethod: node ignores secretRef Node pull method uses kubelet service accounts, not user secrets. Use pullMethod: pod when pulling with custom secrets.
Stuck in WaitForFirstConsumer StorageClass waits for VM scheduling before binding PVC. Add annotation cdi.kubevirt.io/storage.bind.immediate.requested: "true".

🎯 Key Takeaways

  1. Use FROM scratch and assign ownership to 107:107 for lightweight, secure containerDisk images.
  2. Automate lifecycle updates with DataImportCron so developers always provision from the latest patched OS baseline.
  3. Don't forget immediate binding annotations to eliminate PVC provisioning delays.

⏭️ Next Up: Advanced VM Networking & KubeMacPool

Now that our VMs can boot in seconds, how do we connect them to enterprise networks, VLANs, and external subnets?

In Part 4, we dive deep into:

  • Pod Networking (Masquerade) vs. Bridged Secondary Networks with Multus CNI.
  • Managing MAC address allocation and avoiding duplicate MAC clashes with KubeMacPool.

💬 How do you currently build and distribute OS golden images in your infrastructure? Share your workflow in the comments below!