Bringing Docker Parity to GitLab Runner's Kubernetes Executor: Logging the Helper Image in CI Job Logs

A deep dive into executor observability, helper image resolution, configuration expansion, and CI trace logging in Go.

GitLab Runner uses helper images to perform several internal operations during CI job execution, including source retrieval, cache handling, and artifact management.

When running jobs through the Kubernetes executor, the helper image is particularly important because it runs as part of the job's execution infrastructure.

However, there was an observability gap: when users configured a custom helper_image, the Kubernetes executor did not expose the selected helper image directly in the CI job trace.

This made debugging container image configuration more difficult than necessary.

I addressed this behavior in GitLab Runner Merge Request !7778, resolving Issue #2387.

This article explores the underlying problem, the difference between runner debug logging and job trace logging, the implementation, configuration edge cases, and the regression tests added to verify the behavior.

1. Understanding the GitLab Runner Helper Image

Before looking at the implementation, it is useful to understand the role of the helper image in GitLab Runner.

GitLab Runner uses a helper image to execute internal operations required by CI jobs.

Depending on the executor and job configuration, these operations can include:

• Cloning and fetching the project repository.
• Downloading job artifacts.
• Uploading job artifacts.
• Restoring caches.
• Uploading caches.

The helper image is separate from the primary job image configured through .gitlab-ci.yml.

For example:

build: image: golang:1.25 script: - go build ./... 

Here, golang:1.25 is the job image.

The helper image is managed by GitLab Runner and is used for internal job preparation and execution operations.

1.1 Why custom helper images are used

Organizations running self-managed GitLab Runner installations may need to override the default helper image.

Common reasons include:

• Pulling images from an internal container registry.
• Operating in restricted or air-gapped environments.
• Using a controlled image distribution process.
• Maintaining consistency with internal infrastructure requirements.
• Avoiding direct dependencies on public container registries.

GitLab Runner provides the helper_image configuration option for this purpose.

A Kubernetes executor configuration might look like:

[[runners]] executor = "kubernetes" [runners.kubernetes] helper_image = "registry.internal.example.com/gitlab/runner-helper:v1.0.0" 

When a custom helper image is configured, knowing exactly which image the executor selected becomes important for troubleshooting.

2. The Observability Gap

The issue addressed in this contribution was a difference in logging behavior between GitLab Runner's Docker and Kubernetes executors.

2.1 Existing Docker executor behavior

The Docker executor already displayed the selected helper image in the CI job output.

For example:

Using helper image: registry.gitlab.com/gitlab-org/gitlab-runner/gitlab-runner-helper:x86_64-<revision> 

When a custom helper image was configured, the log also indicated that the image had been overridden and showed the default image information.

This made the selected image visible to developers directly from the job trace.

2.2 Existing Kubernetes executor behavior

The Kubernetes executor followed a different logging path.

Inside:

executors/kubernetes/kubernetes.go 

the prepareInfrastructure method used s.BuildLogger.Debugln(...) to report the selected helper image.

The relevant implementation was:

if imageNameFromConfig := s.ExpandValue( s.Config.Kubernetes.HelperImage, ); imageNameFromConfig != "" { s.BuildLogger.Debugln(fmt.Sprintf( "Using helper image: %s (overridden, default would be %s:%s)", imageNameFromConfig, s.helperImageInfo.Name, s.helperImageInfo.Tag, )) } else { s.BuildLogger.Debugln(fmt.Sprintf( "Using helper image: %s:%s", s.helperImageInfo.Name, s.helperImageInfo.Tag, )) } 

At first glance, the image information appeared to be logged correctly.

However, the important distinction was the logging destination.

BuildLogger.Debugln writes to the runner's internal debug logging mechanism rather than the regular CI job trace.

Consequently, developers inspecting their job output could not see which helper image was selected.

This was an observability inconsistency between the executors.

3. Understanding the Logging Boundary

The central part of this issue was not helper image resolution itself.

It was understanding where the log message was being written.

GitLab Runner has different logging paths for different audiences.

The Kubernetes executor was sending useful configuration information to the wrong logging destination.

3.1 Why BuildLogger.Debugln was insufficient

The original implementation used:

s.BuildLogger.Debugln("Using helper image: ...") 

This is appropriate when information is intended for internal debug diagnostics.

However, the selected helper image is also operationally relevant to the CI job itself.

A developer debugging an image pull failure should not need access to the runner host's internal logs just to identify the image configured for the job.

Even enabling:

variables: CI_DEBUG_TRACE: "true" 

did not solve the underlying issue of logging through the wrong output mechanism.

The problem was the destination of the message, not simply the verbosity level.

3.2 The desired behavior

The Kubernetes executor should provide the same useful visibility as the Docker executor.

The selected helper image should appear directly in the job trace:

Using helper image: registry.internal.example.com/gitlab/runner-helper:v1.0.0 

For overridden configurations, the output should also identify the override and its default image information.

The objective was to improve observability without changing helper image selection or execution behavior.

4. Implementing the Fix

The implementation was made in:

executors/kubernetes/kubernetes.go 

Specifically, the prepareInfrastructure method was updated.

The important change was replacing:

s.BuildLogger.Debugln(...) 

with:

s.BuildLogger.Println(...) 

The updated implementation:

if s.Config.Kubernetes.HelperImage != "" { s.BuildLogger.Println(fmt.Sprintf( "Using helper image: %s (overridden, default would be %s)", s.getHelperImage(), s.helperImageInfo, )) } else { s.BuildLogger.Println(fmt.Sprintf( "Using helper image: %s", s.helperImageInfo, )) } 

4.1 Why use BuildLogger.Println?

The regular Println method writes the message to the CI job trace.

This makes the helper image visible to the people who actually need the information while investigating a job.

The change does not alter:

• Which helper image is selected.
• How the helper image is configured.
• How the Kubernetes executor creates its infrastructure.
• How containers are pulled or executed.

It changes the visibility of the selected configuration.

This is an important distinction: the fix improves observability without modifying the underlying execution semantics.

4.2 Reusing existing helper image resolution

The implementation uses:

s.getHelperImage() 

for the configured override.

This avoids duplicating helper image resolution logic inside the logging statement.

The default helper image information is obtained through:

s.helperImageInfo 

Reusing these existing values keeps the logging implementation aligned with the executor's established image configuration behavior.

It also avoids introducing a separate image-resolution path purely for diagnostic output.

5. An Important Edge Case: Environment Variable Expansion

One of the more interesting details in this change is how the configured helper image is checked.

GitLab Runner supports variable expansion in configuration values.

For example:

[runners.kubernetes] helper_image = "$CUSTOM_HELPER_IMAGE" 

At runtime, the configured value is expanded using:

s.ExpandValue(...) 

Consider the following scenario:

Configured value: $UNSET_PROBE_VAR Expanded value: "" 

The original implementation checked the expanded value:

if imageNameFromConfig := s.ExpandValue( s.Config.Kubernetes.HelperImage, ); imageNameFromConfig != "" { // Override logging } 

This creates an ambiguity.

The configured value is non-empty, meaning the user explicitly configured an override.

However, the expanded value is empty because the referenced environment variable is unset.

Checking only the expanded value makes the code treat this configuration as though no override had been configured.

5.1 Checking the original configuration

The updated implementation instead checks:

if s.Config.Kubernetes.HelperImage != "" { 

This distinguishes between two different states:

This distinction is important because configuration intent and runtime expansion are not necessarily equivalent.

The logging path should be able to identify that an override was configured even if its expansion produces an empty value.

In that scenario, the output can expose the unresolved configuration rather than silently presenting it as an ordinary default configuration.

This makes debugging configuration problems considerably more straightforward.

6. Regression Testing

A logging change can appear trivial, but testing its output is important because the behavior depends on both configuration and the logging destination.

I added a dedicated test:

TestPrepare_HelperImageLogging 

Location:

executors/kubernetes/kubernetes_test.go 

The test covers three important scenarios.

6.1 Default helper image configuration

The first case verifies the behavior when no custom helper image is configured.

Expected behavior:

Using helper image: <default-helper-image> 

The job trace should contain the default helper image information.

6.2 Custom helper image override

The second case configures:

custom.registry.io/custom-helper:v1.0.0 

Expected behavior:

Using helper image: custom.registry.io/custom-helper:v1.0.0 (overridden, default would be <default-helper-image>) 

This verifies that the override is visible and that the default image information is also retained in the log.

6.3 Unset environment variable

The third case uses:

$UNSET_PROBE_VAR 

The variable is intentionally unset.

This verifies the important distinction between checking the configured value and checking its expanded result.

The test ensures that a non-empty configured override is still identified as an override even when its runtime expansion is empty.

6.4 Running the test

The focused test can be executed with:

go test -v ./executors/kubernetes \ -run '^TestPrepare_HelperImageLogging$' 

This provides a targeted way to validate the behavior without running the entire GitLab Runner test suite.

The test exercises the preparation path and verifies the resulting helper-image log output for the different configuration states.

7. Why This Change Matters Beyond Logging

The implementation is small, but the underlying engineering concern is broader.

In distributed CI infrastructure, observability is part of the user-facing behavior.

When a CI job fails because a helper image cannot be pulled, developers need enough information to understand what the runner attempted to use.

Without that information, debugging may require:

1. Identifying which runner processed the job.
2. Accessing the runner host or its internal logs.
3. Inspecting the runner configuration.
4. Resolving any configured environment variables.
5. Determining the actual image selected at runtime.

Exposing the selected image in the job trace removes an unnecessary diagnostic step.

7.1 Executor parity

GitLab Runner supports multiple executors, each with its own infrastructure implementation.

Users should not have to lose useful diagnostic information simply because their jobs run on Kubernetes instead of Docker.

Matching the Docker executor's helper-image visibility makes the experience more consistent.

7.2 Better debugging in restricted environments

This is especially useful for organizations using private registries.

When an image pull fails, knowing the exact configured image and whether an override was applied helps narrow down possible causes:

• Incorrect registry hostname.
• Incorrect image name or tag.
• Missing image availability.
• Configuration expansion problems.
• Registry authentication or access issues.

The log does not solve all these problems, but it provides a useful starting point.

7.3 Observability without changing execution

A good diagnostic improvement should avoid changing the system behavior it is trying to explain.

This implementation keeps image resolution and infrastructure preparation intact while making the selected configuration visible through the appropriate logging interface.

8. Final Implementation Summary

The contribution can be summarized as follows:

The core implementation involved three decisions:

1. Use the CI job trace logging method instead of internal debug logging.
2. Reuse existing helper image resolution information.
3. Detect an override using the original configuration value rather than its expanded value.

Together, these changes address the observability gap without introducing a new image-selection mechanism.

9. Conclusion

This contribution addressed a small but meaningful inconsistency in GitLab Runner.

The Kubernetes executor already knew which helper image it was using, but the information was written to internal debug logs instead of the CI job trace.

By switching to BuildLogger.Println, reusing the existing helper image information, and checking the original configuration value, the implementation makes helper image selection visible to CI job users.

The regression tests cover the default configuration, a custom registry override, and an override referencing an unset environment variable.

The broader engineering lesson is that observability is not only about collecting logs. It is also about delivering the right diagnostic information to the right audience, at the right point in the execution lifecycle.

This was a focused improvement to GitLab Runner's Kubernetes executor that brings its helper-image logging behavior closer to the Docker executor.

References and Implementation

• Merge Request: GitLab Runner !7778
• Issue: GitLab Runner #2387
• Implementation: executors/kubernetes/kubernetes.go
• Regression Tests: executors/kubernetes/kubernetes_test.go
• Test: TestPrepare_HelperImageLogging

Author: Manish Kumar

GitLab: @manishpatel00

Contributing to GitLab Runner, Kubernetes infrastructure, and open-source developer tooling.