Three Layers Deep: Getting Confluent Schema Registry to Authenticate with MSK Serverless on EKS

The Setup

When you deploy Confluent Schema Registry on EKS against an MSK Serverless cluster, you're combining three things that each have their own opinions about authentication:

1. Confluent's Docker image — assumes it controls its own environment
2. Kubernetes — helpfully injects environment variables from service discovery
3. MSK Serverless — requires SASL/OAUTHBEARER (IAM) with a specific JAR on the JVM classpath

In isolation, each is fine. Together, they'll fight you in exactly three layers. Here's what I found and how I fixed each one.

The Environment

• EKS 1.31 (Fargate managed node group)
• MSK Serverless (us-east-1)
• confluentinc/cp-schema-registry:7.5.0 as the base image
• MSK IAM auth via IRSA (IAM Roles for Service Accounts)
• aws-msk-iam-auth-2.3.2-all.jar for the SASL callback handler

Layer 1: Kubernetes Poisons the Schema Registry Config

Symptom: Schema registry pod crashes immediately on startup. Logs show something like:

Schema Registry environment variables are not properly configured SCHEMA_REGISTRY_PORT must be set to a valid port number 

Root cause: Kubernetes injects environment variables for every Service in the same namespace. If there's a service named schema-registry, Kubernetes adds:

SCHEMA_REGISTRY_PORT=tcp://10.100.200.50:8081 SCHEMA_REGISTRY_PORT_8081_TCP=tcp://10.100.200.50:8081 SCHEMA_REGISTRY_PORT_8081_TCP_ADDR=10.100.200.50 

Confluent's startup scripts map any env var prefixed with SCHEMA_REGISTRY_ into the config file. So SCHEMA_REGISTRY_PORT=tcp://10.100.200.50:8081 gets translated to port=tcp://10.100.200.50:8081 — which is not a valid port number, causing the crash.

Fix: Add enableServiceLinks: false to the pod spec:

apiVersion: apps/v1 kind: Deployment metadata: name: schema-registry spec: template: spec: enableServiceLinks: false # prevents K8s service env var injection containers: - name: schema-registry image: your-ecr/schema-registry:7.5.0-msk # ... 

This tells Kubernetes to skip environment variable injection from service discovery for this pod. Your own explicitly-defined env vars (like SCHEMA_REGISTRY_KAFKASTORE_BOOTSTRAP_SERVERS) are unaffected.

Same issue hits Kafka broker pods. Any Confluent image that uses the KAFKA_ or SCHEMA_REGISTRY_ env var convention will be crashed by K8s service link injection. Always set enableServiceLinks: false on these deployments.

Layer 2: The MSK IAM Auth JAR Doesn't Reach the JVM

Symptom: Schema registry starts but immediately fails to connect to MSK Serverless:

ERROR org.apache.kafka.clients.NetworkClient - [Producer clientId=...] Connection to node -1 (b-1.xxx.kafka.us-east-1.amazonaws.com/...) failed authentication due to: SSL handshake failed 

Or worse — it silently retries PLAINTEXT/SSL when what MSK Serverless requires is SASL/OAUTHBEARER.

Root cause: MSK Serverless only supports IAM (SASL/OAUTHBEARER). The Confluent 7.5 base image doesn't include the aws-msk-iam-auth JAR. You need to add it to the JVM classpath. But there's a catch: Confluent's entrypoint scripts call java directly and ignore the CLASSPATH environment variable.

Setting CLASSPATH=/path/to/aws-msk-iam-auth.jar in your deployment manifest does nothing.

Fix: Build a custom image that bakes the JAR directly into the directory Confluent's scripts scan:

FROM confluentinc/cp-schema-registry:7.5.0 # Confluent's kafka-run-class hardcodes /usr/share/java/schema-registry/* # and does NOT honour the CLASSPATH env var — the JAR must be here. USER root ADD https://repo1.maven.org/maven2/software/amazon/msk/aws-msk-iam-auth/2.3.2/aws-msk-iam-auth-2.3.2-all.jar \ /usr/share/java/schema-registry/aws-msk-iam-auth.jar RUN chmod 644 /usr/share/java/schema-registry/aws-msk-iam-auth.jar USER appuser 

Using ADD with the Maven Central URL means you don't need to manage the JAR locally — Docker pulls it at build time. The USER root / USER appuser sandwich is required because Confluent's base image runs as appuser (uid 1000) by default, but writing to /usr/share/java/schema-registry/ requires root.

Then configure the SASL settings in your env vars:

env: - name: SCHEMA_REGISTRY_KAFKASTORE_BOOTSTRAP_SERVERS value: "b-1.xxx.kafka.us-east-1.amazonaws.com:9098" - name: SCHEMA_REGISTRY_KAFKASTORE_SECURITY_PROTOCOL value: "SASL_SSL" - name: SCHEMA_REGISTRY_KAFKASTORE_SASL_MECHANISM value: "AWS_MSK_IAM" - name: SCHEMA_REGISTRY_KAFKASTORE_SASL_JAAS_CONFIG value: "software.amazon.msk.auth.iam.IAMLoginModule required;" - name: SCHEMA_REGISTRY_KAFKASTORE_SASL_CLIENT_CALLBACK_HANDLER_CLASS value: "software.amazon.msk.auth.iam.IAMClientCallbackHandler" 

The key is that /usr/share/java/schema-registry/ is the directory Confluent's launch scripts automatically add to the JVM classpath via wildcard (-cp /usr/share/java/schema-registry/*). Putting your JAR there means it's picked up without any script modification.

Why not CLASSPATH? The Confluent Docker image uses launch scripts that construct the java command internally. The CLASSPATH env var is only used if java is invoked without explicit -cp/-classpath flags. Confluent sets its own -cp, so your CLASSPATH override is ignored.

Layer 3: The Ensure Script Fails Pre-Flight Checks

Symptom: Pod starts, gets past the JAR loading, but then hangs for minutes before failing with a connectivity error — even though MSK is accessible. Or it fails immediately with an obscure error from the Confluent ensure script.

Root cause: The Confluent Docker image runs /etc/confluent/docker/ensure as a pre-flight check before launching Schema Registry. This script attempts to connect to Kafka to verify bootstrap connectivity. In the EKS environment, it may run before your IRSA credentials are fully available, or it uses a different code path that doesn't pick up the IAM auth callback handler.

Fix: Override the ensure script with a no-op via a ConfigMap subPath mount:

# ConfigMap apiVersion: v1 kind: ConfigMap metadata: name: schema-registry-scripts data: ensure: | #!/bin/bash # No-op: pre-flight connectivity check skipped in EKS. # IRSA credentials are injected at runtime; the ensure script # runs before the IAM token exchange is complete. exit 0 

# In the Deployment spec volumeMounts: - name: schema-registry-scripts mountPath: /etc/confluent/docker/ensure subPath: ensure volumes: - name: schema-registry-scripts configMap: name: schema-registry-scripts defaultMode: 0755 

The subPath: ensure mount replaces only that single file in the container's filesystem — the rest of /etc/confluent/docker/ is untouched. The defaultMode: 0755 ensures the script is executable.

Is this safe? The ensure script's job is to verify connectivity before startup. Since Schema Registry itself will retry connectivity on startup with proper retry/backoff logic, skipping the pre-flight check doesn't lose safety — it just removes a redundant check that doesn't understand your auth setup anyway.

The Complete Custom Image

FROM confluentinc/cp-schema-registry:7.5.0 # Confluent's kafka-run-class hardcodes /usr/share/java/schema-registry/* # and does NOT honour the CLASSPATH env var — the JAR must be here. USER root ADD https://repo1.maven.org/maven2/software/amazon/msk/aws-msk-iam-auth/2.3.2/aws-msk-iam-auth-2.3.2-all.jar \ /usr/share/java/schema-registry/aws-msk-iam-auth.jar RUN chmod 644 /usr/share/java/schema-registry/aws-msk-iam-auth.jar USER appuser 

Build and push to ECR:

docker build -t <account-id>.dkr.ecr.us-east-1.amazonaws.com/ai-platform/schema-registry:7.5.0-msk \ -f docker/schema-registry/Dockerfile docker/schema-registry/ aws ecr get-login-password --region us-east-1 | \ docker login --username AWS --password-stdin <account-id>.dkr.ecr.us-east-1.amazonaws.com docker push <account-id>.dkr.ecr.us-east-1.amazonaws.com/ai-platform/schema-registry:7.5.0-msk 

The Full Deployment Manifest

apiVersion: apps/v1 kind: Deployment metadata: name: schema-registry namespace: microservices spec: replicas: 1 strategy: type: Recreate # rolling update with 2 concurrent pods is toxic on MSK consumer groups selector: matchLabels: app: schema-registry template: metadata: labels: app: schema-registry spec: enableServiceLinks: false # Layer 1: prevents K8s service env var injection serviceAccountName: schema-registry-sa # IRSA service account containers: - name: schema-registry image: <account-id>.dkr.ecr.us-east-1.amazonaws.com/ai-platform/schema-registry:7.5.0-msk ports: - containerPort: 8081 env: - name: SCHEMA_REGISTRY_HOST_NAME valueFrom: fieldRef: fieldPath: status.podIP - name: SCHEMA_REGISTRY_LISTENERS value: "http://0.0.0.0:8081" - name: SCHEMA_REGISTRY_KAFKASTORE_BOOTSTRAP_SERVERS value: "b-1.xxx.kafka.us-east-1.amazonaws.com:9098" - name: SCHEMA_REGISTRY_KAFKASTORE_SECURITY_PROTOCOL value: "SASL_SSL" - name: SCHEMA_REGISTRY_KAFKASTORE_SASL_MECHANISM value: "AWS_MSK_IAM" - name: SCHEMA_REGISTRY_KAFKASTORE_SASL_JAAS_CONFIG value: "software.amazon.msk.auth.iam.IAMLoginModule required;" - name: SCHEMA_REGISTRY_KAFKASTORE_SASL_CLIENT_CALLBACK_HANDLER_CLASS value: "software.amazon.msk.auth.iam.IAMClientCallbackHandler" startupProbe: httpGet: path: /subjects port: 8081 failureThreshold: 30 periodSeconds: 10 # 300s total — _schemas topic sync can take a while livenessProbe: httpGet: path: /subjects port: 8081 initialDelaySeconds: 60 periodSeconds: 30 volumeMounts: - name: schema-registry-scripts mountPath: /etc/confluent/docker/ensure subPath: ensure # Layer 3: override ensure script volumes: - name: schema-registry-scripts configMap: name: schema-registry-scripts defaultMode: 0755 

Why Each Layer Was Non-Obvious

The real lesson: when you combine a third-party container image with a managed cloud service and a container orchestrator, each one adds an invisible layer of convention. The debug process is peeling them back one at a time — which is exactly why the error messages never tell you the right thing.

The Smoke Test

After all three layers are resolved, Schema Registry starts clean and the MSK Serverless connection goes through IAM:

# From inside the EKS cluster kubectl exec -n microservices deploy/user-service -- \ curl -s http://schema-registry:8081/subjects # -> [] (empty on first boot, connection confirmed) 

And the full path: POST /api/v1/users through the ALB → user-service → MSK Serverless (via Schema Registry for Avro serialization) → notification-service consumer. HTTP 201.

Have you run into similar layering conflicts when combining third-party container images with managed cloud services on Kubernetes? Which of these three layers caught you most off-guard?