The problem
I've heard this question come up many times over the years, and I've been involved in quite a few discussions about it myself: how do you collect metrics from an application running in a container if the application doesn't expose an HTTP /metrics endpoint?
Maybe it was never designed to do that. Or maybe adding an HTTP server just for monitoring feels like unnecessary overhead.
The question becomes particularly interesting when the application runs a short-lived task: a CLI command, a cron job, or a batch job.
One example I dealt with involved CLI commands in a PHP 7.4 monolith that we were gradually moving into a containerized environment.
The application already existed, did its job, and had years of development behind it. We needed both business metrics, such as counters and histograms, and technical metrics, such as database query execution times.
On physical or virtual servers, this was relatively straightforward. We could accumulate data in files, periodically aggregate it, and send it to the monitoring system.
That approach could work for years.
But moving to Kubernetes changes things.
A container starts, performs its task, and exits. There is no guarantee that Prometheus will perform its next scrape while the container is still running.
Even if the application can produce metrics, we still have a question: when exactly should they be collected, and who makes sure the final state remains available?
Imagine a cron command that finishes in five seconds while the Prometheus scrape interval is fifteen seconds.
Prometheus may never see that particular application instance.
We could send the metrics to a separate service. But that brings up another set of questions: who aggregates the data, how do we make sure the final update gets through, what happens with network delays, and who manages the metrics lifecycle?
What if the application exits with an error?
What if the container has already received a termination signal?
What if the monitoring infrastructure is temporarily unavailable?
What started as a simple request to "collect a few metrics" gradually turns into another service that somebody has to develop, deploy, and maintain.
All just to find out how many operations were processed, how long they took, and how much time the application spent querying the database.
Of course, we could simply stop collecting those metrics. But they're important to the application teams. Without them, it's much harder to understand what's happening inside the application.
And, as a tech lead, my conscience wouldn't let me casually throw away data that could become invaluable at the worst possible moment.
Looking for a solution
One of the first suggestions from the architecture committee was to write our own wrapper.
At first glance, that sounded reasonable: a small helper process that accepts metrics and exposes them to Prometheus.
But a question quickly followed:
Who is going to maintain it?
Writing a prototype is one thing. Making it behave correctly inside a container is another.
You have to deal with termination signals, child processes, exit codes, time limits, application errors, and failures in the metrics-handling code itself.
And under no circumstances should monitoring interfere with the application's business logic or prevent it from completing its job.
The infrastructure team, meanwhile, wasn't enthusiastic about deploying additional components.
And their position made sense.
Kubernetes already had a working monitoring setup. Prometheus was scraping the microservices, everything was configured, and someone was maintaining it.
Why build another infrastructure layer just for a handful of cron jobs?
At some point, the discussion was essentially:
"The monolith can live without those metrics for now. Or spend the time adapting it to the new infrastructure."
But even adapting the application wouldn't solve the fundamental problem of short-lived processes.
You can teach a PHP application to expose /metrics. If it exits before the next scrape, Prometheus may still collect nothing.
What about the existing tools?
Naturally, the first thing to do was look at how other people had already solved this problem.
An application-owned HTTP endpoint
For a long-running service, this is the obvious approach.
The application exposes /metrics, and Prometheus scrapes it periodically.
But with short-lived CLI processes, we run into the lifecycle problem again. The process may exit before the next scrape.
Prometheus Pushgateway
Pushgateway is an established option for certain batch-job scenarios.
The application pushes its metrics to Pushgateway, and Prometheus scrapes them from there.
However, Pushgateway has its own metrics lifecycle. Metrics can remain available after the application has finished, so their freshness and cleanup need to be managed explicitly.
It also introduces another infrastructure component and network communication with it.
For service-level batch jobs, Pushgateway may be exactly the right choice. But it isn't always a good fit for metrics associated with individual instances of short-lived workloads.
Node Exporter Textfile Collector
Another option is to write metrics to a file and expose them through Node Exporter.
For some workloads, this is a simple and effective solution.
But it requires an appropriate file-sharing arrangement and collection infrastructure. In a containerized environment, especially when job instances are created and removed dynamically, that raises additional questions.
A sidecar exporter
We could run a separate exporter container alongside the application.
That separates the application's responsibilities from metrics exposition.
But it also adds a container, consumes resources, requires communication between the two processes, and makes their lifecycles something we need to coordinate.
OpenTelemetry
OpenTelemetry offers a much broader set of capabilities for working with telemetry.
OpenTelemetry Collector can receive metrics from an application and expose them through an HTTP endpoint for Prometheus.
It can also run locally, even inside the same container, so a separate centralized service isn't strictly necessary.
However, the Collector primarily handles telemetry processing and delivery. It doesn't act as a ready-made supervisor for an arbitrary application process.
In our scenario, we would still need to implement the PHP process startup, signal forwarding, child-process cleanup, exit-code preservation, and a bounded wait for the final scrape after the workload finishes.
In other words, we could build the required behavior around OpenTelemetry Collector, but we'd still have to write and maintain our own wrapper.
As it turned out, that wrapper was precisely the part we wanted to make reusable.
All of these approaches are valid. In fact, any one of them might be the best solution for a particular workload.
But we wanted several properties at the same time:
- Keep the existing Prometheus infrastructure unchanged.
- Avoid mandatory additional services.
- Avoid adding another container just for metrics.
- Avoid making the application run its own HTTP endpoint.
- Keep the final metrics available for a limited time after the workload exits.
- Use the same solution with Docker and Kubernetes.
Among the options we examined, we didn't find one that combined all of these properties without additional trade-offs.
Does anyone else even have this problem?
I reached out to colleagues at other companies.
It turned out they had encountered similar situations.
And not necessarily with large monoliths. Sometimes it was an ordinary CLI application, a cron job, or a worker that ran its task and exited.
The workloads were different, but the underlying problem was familiar: the application should do its job, while the infrastructure should be able to collect metrics in the usual way.
During the research, I also used ChatGPT, Gemini, DeepSeek, and YandexGPT to discuss alternatives, challenge assumptions, and explore architectural ideas.
Of course, an LLM saying it hasn't seen a solution doesn't prove that such a solution doesn't exist. The research still had to rely on the documentation of existing tools, comparisons of their capabilities, and our actual requirements.
And gradually, the idea of a separate tool took shape.
Defining the scope
Before writing any code, we needed to establish some boundaries.
There was no point in creating a replacement for existing tools.
If an application already exposes a proper Prometheus endpoint and runs long enough to be scraped regularly, it doesn't need another component.
If Pushgateway fits a particular batch-job scenario, use Pushgateway.
If the infrastructure is already built around OpenTelemetry, there's no reason to abandon it just to introduce something new.
We were interested in a narrower case:
A containerized workload that cannot, or should not, be responsible for exposing its own metrics to Prometheus.
Not bare metal. Not a general-purpose monitoring system. Not a new central metrics aggregator.
Just a container, its application, and the standard Prometheus pull model.
The idea was to place a small adapter directly inside the application container.
It would receive metrics locally, expose them on /metrics, and account for the workload's lifecycle.
No mandatory central service.
No separate sidecar.
No need to rework the application around a new monitoring architecture.
Requirements
Once we had defined the scope, we could put together the basic requirements.
First. The application should start and finish just as it would without the wrapper.
Its exit code must be preserved. Signals must be forwarded correctly. Child processes must be handled and reaped.
Second. The application must be able to submit metrics locally, without communicating with external monitoring infrastructure.
Third. Metrics must be exposed through the standard HTTP /metrics endpoint, so Prometheus can scrape them.
Fourth. The application must be able to submit a complete snapshot of its metrics.
Fifth. For applications that don't want to build their own snapshots, there should be a simple way to work with counters, gauges, and histograms.
Sixth. After the application exits naturally, its final metrics should remain available for a limited time.
There is an important limitation here.
We cannot guarantee that Prometheus will collect those metrics if it's unavailable or hasn't discovered the container in time.
But we can give it another opportunity to scrape them without keeping the container alive indefinitely.
Finally, the whole thing should work inside a single container, with no mandatory additional infrastructure.
And that's how MetricShell came about
The result of this research and implementation was MetricShell.
The idea is fairly simple.
Instead of making the application manage its own HTTP endpoint, we give that responsibility to a small wrapper process.
MetricShell starts as PID 1 inside the container and then launches the user's command.
That might be a PHP CLI application, a cron job, or a worker.
MetricShell supervises the process, receives metrics, and exposes them through /metrics.
Meanwhile, the application continues doing its actual job.
Well, this should be easy, I thought
The plan looked straightforward: start up, accept some metrics, serve them over HTTP, and exit.
There's a law of software development that seems to apply here: the less time you expect something to take, the longer it actually takes.
Or how a couple of hours quietly turned into a couple of months.
The questions started piling up:
- What if the application has already exited, but its last batch of metrics is still being processed?
- What exactly counts as the final state?
- What if a SIGTERM arrives at the same moment?
- What if Prometheus starts reading the metrics right as the snapshot is being updated?
- What if the application has spawned an entire flock of pterodactyls, and they're sending metrics too?
And then there are the details of running as PID 1 on Linux: forwarding signals, dealing with child processes, making sure they're properly cleaned up, and preserving the application's actual exit code.
In the end, we had to design a proper state machine, define event ordering, and deal with races between process termination, metrics ingestion, and HTTP requests.
The hardest part of building a tool for metrics turned out to have very little to do with metrics.
It was about getting process termination right.
How does the application submit metrics?
The current implementation supports two modes.
Snapshot mode
The application builds a complete snapshot of its metrics and submits it to MetricShell.
When a new valid snapshot is accepted, it atomically replaces the previous one.
This is useful when the application already collects its own metrics and can produce their complete state.
Managed Registry
In this mode, MetricShell maintains a local metrics registry.
The application sends operations such as incrementing a counter, setting a gauge, or recording an observation in a histogram.
Communication takes place over a local Unix socket.
This means every application doesn't need to implement a full Prometheus registry of its own.
Both modes use the same mechanism for exposing metrics to Prometheus.
What happens when the application exits?
Imagine that our PHP command has finished processing its data.
Normally, once the main process exits, the container will soon terminate as well.
MetricShell, however, runs as PID 1 and supervises the application process.
When the workload exits naturally, MetricShell finalizes metrics ingestion, freezes the last valid state, and can continue serving it through /metrics.
What happens next depends on the configuration.
MetricShell can exit immediately, wait for a configured duration, or wait for a specified number of eligible completed scrape responses, subject to a finite timeout.
It then exits while preserving the workload's exit status, unless a higher-priority MetricShell-owned failure occurred.
So Prometheus gets another opportunity to collect the final metrics.
This is not guaranteed delivery, and it is not a replacement for durable storage.
But for this particular class of workloads, it allows us to stick with the familiar Prometheus pull model.
Architecture
While the application is running:
┌────────────────────────────────┐ │ Container │ │ │ │ ┌──────────────────────────┐ │ │ │ MetricShell (PID 1) │ │ │ │ │ │ │ │ Snapshot / Registry │◄─┼── Prometheus │ │ GET /metrics │ │ GET /metrics │ └────────────▲─────────────┘ │ │ │ │ │ Local metrics input │ │ │ │ │ ┌────────────┴─────────────┐ │ │ │ PHP CLI / Cron / │ │ │ │ Worker workload │ │ │ └──────────────────────────┘ │ └────────────────────────────────┘ After the workload exits naturally, MetricShell can continue serving the frozen final snapshot until the configured wait condition is met or its timeout expires.
If Prometheus doesn't scrape the metrics in time, MetricShell still exits within the configured limits.
One important detail: this final-scrape waiting behavior applies to natural workload completion. If the container is externally terminated, MetricShell follows its bounded shutdown procedure, and the final wait may be interrupted.
Trying it out
At the time of writing, the current published release is v0.2.2.
MetricShell is written in Go, released under the MIT license, and supports Linux on amd64 and arm64.
You don't need to write an application just to see how it works.
A runnable example is published on GitHub Container Registry:
docker run --rm \ --name metricshell-example \ -p 127.0.0.1:9090:9090 \ ghcr.io/denki77/metricshell-example:0.2.2 In another terminal:
curl http://127.0.0.1:9090/metrics You'll see the metrics exposed by MetricShell.
To integrate it with your own application, there's also an artifact image containing the MetricShell binary, which you can copy into your application's Docker image.
A simplified example:
FROM ghcr.io/denki77/metricshell-artifact:0.2.2 AS metricshell FROM my-application COPY --from=metricshell /metricshell /usr/local/bin/metricshell ENTRYPOINT ["/usr/local/bin/metricshell", "--"] CMD ["my-worker"] This only illustrates how to integrate the wrapper. The application must still submit metrics through one of the supported local interfaces, and Prometheus must be configured to discover and scrape the container.
For production deployments, the documentation recommends pinning images by digest and using the documented configuration.
Docker and Kubernetes examples, protocols, configuration options, and operational limitations are available in the repository.
Limitations and trade-offs
Every architectural decision comes with trade-offs.
MetricShell is no exception.
It introduces another process inside the container and takes responsibility for supervising the workload correctly.
The application must still produce or publish its own metrics. MetricShell cannot magically figure out which business metrics should be measured.
The final snapshot is only available for a bounded period. If Prometheus misses that window, the metrics may be lost.
The metrics registry lives in the memory of a particular MetricShell instance. It does not restore its previous state after a restart.
MetricShell also doesn't handle Prometheus target discovery, time-series storage, or monitoring infrastructure configuration.
These are intentional boundaries.
MetricShell solves a specific problem without trying to grow into an entire monitoring system.
What's next?
At this point, the core functionality of MetricShell is implemented.
The source code, documentation, architectural decisions, and published artifacts are available on GitHub.
But one question remains: how useful is this approach to other engineers?
I don't know yet.
Maybe the existing tools are perfectly sufficient for most teams.
Maybe this problem comes up much more often than it seems.
Or maybe somebody has already solved it in a simpler and more elegant way.
If even three developers or DevOps engineers end up using MetricShell, or recommending it for a suitable scenario, I'll consider the project worthwhile.
After all, one of those three is already accounted for: me.
I'd be interested in technical criticism of the architecture, especially limitations I may have overlooked.
Technical discussions, corrections, and alternative approaches are welcome.
GitHub: https://github.com/Denki77/metricshell
Acknowledgments
Both colleagues and AI tools helped me throughout the research and implementation.
Special thanks to:
- Leonid (SuperJob) — for the enthusiasm and support that kept the engineering exploration moving.
- Alexey (OpticCity) — for sound algorithmic decisions.
- Dima (Tile.Expert) — for the uncomfortable questions and discussions of possible solutions.
- ChatGPT and Codex — for help with research, implementation, and reviewing decisions.
- Google Gemini — for asking the right questions, even when the answers weren't immediately obvious.
And, of course, thanks to everyone I discussed the monitoring of containerized workloads with.
Conversations like these are often where new engineering solutions begin.
