One Hub to Watch Them All: Multi-Tenant Monitoring for 4 Clusters and 400+ Edge Machines with Grafana, Mimir, Loki and Alloy

"How I built a self-hosted, GitOps-managed observability hub on Kubernetes: Mimir for metrics, Loki for logs, Alloy as the only agent, tenants enforced at the gateway, and S3 as the real database."

The problem

Our monitoring was spread across too many places. We run a handful of on-prem Kubernetes clusters (one "hub" plus several "spoke" clusters) and more than 400 edge machines at remote sites. Each place had its own partial monitoring. The rest had none.

That caused the usual pain:

• No single place to look. When something broke at an edge site, we SSH'd in and grepped logs. Live, if the machine was still reachable.
• No history. Local Prometheus instances kept a few days of data on whatever disk they had. Questions like "when did this start?" had no answer.
• No separation. If several teams shared one Prometheus, they all saw each other's data, and one noisy sender could hurt everyone.
• Hand-made config. Nobody could say exactly what was deployed or why.

I wanted one central place for the metrics and logs of every cluster and every machine. It had to be self-hosted, multi-tenant, cheap on disk, and fully described in Git.

This post covers what I built, how it works, and the mistakes I made along the way.

The stack at a glance

The design rests on four ideas:

1. Hub and spoke. Mimir, Loki and Grafana run only on the hub. Every cluster and machine runs Alloy and pushes to the hub over HTTPS. The hub never has to reach into a site.
2. Object storage is the real database. Mimir and Loki keep only recent data on local disks. Everything else goes to S3. Retention is 15 days for metrics and 30 days for logs.
3. The gateway decides the tenant, never the sender. Each sender authenticates with its own login, and the gateway maps that login to a tenant.
4. Everything is Git. Nothing is applied by hand. Secrets come from a vault, so Git holds no credentials.

1. Collection: Alloy everywhere

I used Grafana Alloy as the single agent. A single Alloy DaemonSet per cluster collects:

• pod logs (Kubernetes API discovery),
• node / systemd journal logs,
• node metrics (built-in node_exporter),
• kubelet and cAdvisor metrics,
• kube-state-metrics.

The edge machines run the same Alloy binary as a systemd service, with a smaller config that ships host logs and host metrics.

Every series and log stream gets a cluster label, so one set of dashboards works for every cluster:

prometheus.remote_write "default" { endpoint { url = "https://metrics.example.com/api/v1/push" basic_auth { username = sys.env("PUSH_USER") password = sys.env("PUSH_PASSWORD") } // Informational only: the gateway overrides it from the login headers = { "X-Scope-OrgID" = "spoke-a" } } external_labels = { cluster = "spoke-a" } } loki.write "default" { endpoint { url = "https://logs.example.com/loki/api/v1/push" tenant_id = "spoke-a" basic_auth { username = sys.env("PUSH_USER") password = sys.env("PUSH_PASSWORD") } } external_labels = { cluster = "spoke-a" } } 

Small gotcha: loki.write has a tenant_id field. prometheus.remote_write does not, so you set the X-Scope-OrgID header yourself.

"Why still run kube-state-metrics if Alloy collects metrics?" My team lead asked exactly that. Alloy scrapes. kube-state-metrics produces metrics about Kubernetes objects: desired vs. available replicas, pod phase, restarts, PVC status, and so on. Nothing else exposes those, and every "is my deployment healthy?" panel needs them. I run one replica per cluster, with no autoscaling, because two replicas would just double every series. Plain-host edge machines don't need it.

2. Metrics: Mimir in classic mode

Mimir runs in distributed mode on the hub:

gateway → distributor → ingester ×3 (RF 3, WAL on SSD) → blocks to S3 ↑ query-frontend → query-scheduler → querier → store-gateway ×3 (reads S3) compactor: merges blocks, enforces 15-day retention memcached: chunks, index, metadata and results caches 

What I configured, and why:

• Classic architecture (no Kafka). Recent chart versions default to the Kafka-based "ingest storage" architecture. That's a lot of extra machinery for our scale, so I switched it off with ingest_storage.enabled: false.
• Replication factor 3 on ingesters and store-gateways.
• Per-tenant limits sized for reality. The defaults (10k samples/s, 150k series) are far too low for hundreds of senders.
• out_of_order_time_window: 10m. Edge sites sit on flaky links and sometimes deliver late samples.
• Stateless parts autoscale with HPAs: distributor 3–6, querier 3–8, query-frontend 2–3, gateway 2–4. Stateful parts are scaled by hand.

mimir: structuredConfig: multitenancy_enabled: true tenant_federation: enabled: true # lets Grafana query "tenant-a|tenant-b"  ingest_storage: enabled: false # classic: distributor → ingester over gRPC  ingester: ring: replication_factor: 3 limits: compactor_blocks_retention_period: 15d ingestion_rate: 200000 max_global_series_per_user: 5000000 out_of_order_time_window: 10m 

A chart quirk worth knowing: the mimir-distributed chart only has a native HPA for the gateway. For distributor, querier and query-frontend it offers only KEDA-based autoscaling, which is marked experimental and needs the KEDA operator. I didn't want another operator just for CPU/memory scaling, so I defined plain autoscaling/v2 HPAs as extra objects and set replicas: null on those Deployments. That way Argo CD and the HPA never fight over the replica count.

3. Logs: Loki in distributed mode

Loki looks almost the same: gateway → distributor → 3 ingesters → chunks and TSDB index in S3, with index-gateways, a compactor for retention, and a pattern-ingester for Grafana's Logs Drilldown.

The settings that matter:

• TSDB schema v13 with a 24h index period.
• 30-day retention, enforced by the compactor (retention_enabled: true).
• Per-tenant rate limits on ingestion and on streams.
• Chunks flush after 30 minutes idle or 2 hours max age.

4. Multi-tenancy: the gateway decides

This is the part I'm happiest with.

Mimir and Loki are multi-tenant through the X-Scope-OrgID header. The catch: whoever sets the header picks the tenant. If senders set it themselves, a compromised or misconfigured edge machine could write into another tenant's data, or read it.

So the senders' header doesn't count. Both gateways (nginx) require basic auth, then derive the tenant from the login. Any header the client sent is overwritten:

map $remote_user $tenant { hub-push hub; # the hub cluster's own Alloy grafana $http_x_scope_orgid; # only Grafana may choose tenants ~^edge-sites- edge-sites; # every edge machine: edge-sites-<site-id> default $remote_user; # spoke clusters: login = tenant } # in every proxied location proxy_set_header X-Scope-OrgID $tenant; 

What this gives us:

• Spoke clusters log in as their own cluster name and land in their own tenant.
• Each of the 400+ edge machines has its own login (edge-sites-<site-id>). They all land in one shared tenant, edge-sites. One machine's credentials can be revoked without touching the others. A regex in the map does it, so adding a site needs no config change.
• Only Grafana may pick tenants itself. Its datasource sends hub|spoke-a|edge-sites and Mimir's tenant federation merges them in a query.

I tested this with a throwaway nginx container before rolling it out:

• a forged X-Scope-OrgID gets overwritten;
• edge logins map to the edge tenant;
• a wrong password returns 401.

That test was worth it. My first version was wrong: return 200 in a test location skips auth_basic entirely, so the test passed for the wrong reason. Test with a real proxy_pass backend.

Where the credentials live

Nothing secret is in Git. OpenBao holds the values, and External Secrets Operator turns them into Kubernetes Secrets. I keep two kinds of credentials at separate paths:

The sender needs the plain password to log in. The gateway needs only the bcrypt hash. Keeping them apart means each consumer gets only what it needs.

The edge team hands over their htpasswd lines. I store them at a separate path, and an ExternalSecret template joins both lists into the single .htpasswd the gateway mounts:

apiVersion: external-secrets.io/v1 kind: ExternalSecret metadata: name: loki-gateway-basic-auth spec: secretStoreRef: kind: ClusterSecretStore name: vault-store target: name: loki-gateway-basic-auth template: engineVersion: v2 data: # strip Windows line endings in case the list was edited there  .htpasswd: | {{ .hub | replace "\r" "" | trim }} {{ .edge | replace "\r" "" | trim }} data: - secretKey: hub remoteRef: { key: monitoring/loki-gateway-basic-auth, property: htpasswd } - secretKey: edge remoteRef: { key: monitoring/edge-sites-htpasswd, property: htpasswd } 

Two details:

• bcrypt cost 5 (htpasswd -nbB -C 5 <user> <password>). The gateway checks the hash on every push request. The default cost burns CPU for no real gain on long random machine passwords.
• nginx ignores # lines in htpasswd. So a # edge logins placeholder keeps the path valid before the first real login exists.

5. Grafana: stateless and HA

• 2–4 replicas behind an HPA.
• State in PostgreSQL (CloudNativePG): users, dashboards and alert rules. Grafana needs no PVC, and any replica can serve any request.
• HA alerting. The replicas gossip over a headless service, so each alert notification goes out once.
• Grafana-managed alert rules. Mimir's ruler and Alertmanager and Loki's ruler are switched off. Everything lives in one place, and one rule can query both metrics and logs.
• Dashboards from Git. Community dashboards (Kubernetes Views, Node Exporter Full) are stored as JSON in the repo. Kustomize turns them into ConfigMaps, and the dashboard sidecar loads them into folders.

# kustomization.yaml generatorOptions: disableNameSuffixHash: true labels: grafana_dashboard: "1" annotations: # dashboard JSON exceeds the 256 KB last-applied annotation limit  argocd.argoproj.io/sync-options: ServerSideApply=true configMapGenerator: - name: grafana-dashboards-kubernetes namespace: monitoring options: annotations: grafana_folder: Kubernetes files: - dashboards/kubernetes/k8s-views-global.json - dashboards/kubernetes/k8s-views-pods.json 

6. Storage: S3 for history, SSD for the hot path

• S3 (MinIO on a NAS) holds every Mimir block and every Loki chunk. It's cheap, large, and the compactors enforce retention on it. Each component has its own bucket and its own access key, scoped to that bucket.
• Longhorn SSD volumes hold only ingester WALs, recent data and index caches.

I started on iSCSI HDD and it was too slow for ingester WALs.

Longhorn keeps one replica per volume, because Mimir and Loki already replicate data 3× at the application level. Another 3× underneath would just be 9 copies.

If a volume is lost, at most the last ~2 hours that weren't uploaded yet are at risk, and the other two ingester replicas cover even that.

7. GitOps and secrets

Everything is an Argo CD Application under one app-of-apps. Each Helm-based component uses multi-source Applications: the upstream chart from its Helm repo, plus a values.yaml from our Git repo. Chart upgrades become a one-line version bump in a pull request.

The rules:

• No kubectl apply or kubectl edit on these components. Change Git, push, let Argo sync.
• Every credential lives in OpenBao: S3 keys, push credentials, htpasswd lists, the Grafana admin login, DB credentials. External Secrets Operator syncs them. The Git repo is safe to share inside the company.
• VPA in recommend-only mode (updateMode: "Off") for every workload. It never restarts anything, but after a couple of weeks of real traffic it tells me what the requests and limits should be.

Lessons learned (the hard way)

1. Removing one Helm values block can silently re-architect a component.

Someone tidied up the Mimir values and deleted the ingester: block. It looked redundant next to ingest_storage.enabled: false. It isn't: one configures how data reaches the ingesters, the other configures the ingester pods.

Without the block, the chart fell back to its default: zone-aware ingesters (ingester-zone-a/b/c) with no StorageClass. We have no default StorageClass, so the new pods sat Pending and ingestion stopped.

Fix: restore the block and leave a loud comment saying why it must stay.

2. Two memberlist clusters on one network can merge.

Mimir and Loki both use memberlist gossip on port 7946, both with a ring called ingester. After pods restarted and IPs were reused, Loki's ring contained Mimir's ingesters. Setting a distinct cluster_label on each fixed it for good.

3. RKE2's DNS service isn't called kube-dns.

The gateways' nginx resolver crash-looped until I set global.dnsService: rke2-coredns-rke2-coredns.

4. Don't put Grafana behind PgBouncer in transaction mode.

Prepared statements break (unnamed prepared statement does not exist). Grafana connects straight to the Postgres primary.

5. An HPA needs requests on every container in the pod.

Grafana's dashboard sidecar had none. The HPA reported missing request for cpu, and Argo CD marked the app Degraded.

6. A browser login popup in Grafana means a datasource returned 401.

The gateway's WWW-Authenticate header goes all the way through to the browser. Fix the datasource credentials; the popup isn't the real problem.

What we got out of it

• One place for everything. Metrics and logs from every cluster and every edge site, in one Grafana, correlated by the same cluster label.
• Real history. 15 days of metrics and 30 days of logs, regardless of whether the edge machine is still online.
• Isolation by design. Senders can't pick their tenant. Each tenant has its own limits, so a noisy sender can't starve the rest. Revoking one machine is one line in the vault.
• Horizontal scale where it matters. The write and read paths scale with HPAs. Storage grows with the object store, not with node disks.
• Cheap and self-hosted. No per-series SaaS bill. The expensive part (history) lives on commodity S3.
• Reproducible. The whole stack can be rebuilt from Git plus the vault. Every change has a commit and a reviewer.
• One agent to maintain. Alloy handles logs and metrics on Kubernetes and on bare hosts.

What's next

• More worker nodes. Replication factor 3 doesn't protect against losing the node it all runs on, and our hub currently has a single worker. Anti-affinity is already set to preferred, so pods will spread on their own once the nodes exist.
• SSO for Grafana and IP allowlists on the push endpoints.
• Per-tenant limit overrides, so the edge tenant and the hub can have different budgets.
• Self-monitoring of Mimir, Loki and Grafana, plus Kubernetes events as logs.

If you're building something similar, my strongest advice is about the multi-tenancy boundary. Put it at the gateway and derive it from the identity, not from a header the client controls. Everything else in this stack is a sizing question.

Questions or a different approach? Leave a comment, I'd love to compare notes.