In every event-driven system I've operated, there's one metric that predicts user-visible trouble before anything else: consumer lag. Not CPU, not memory, not pod count — lag. If your consumers can't keep pace with the topic, everything downstream of that offset is silently falling behind while your dashboards stay green.
Remember why this matters: the front end of your system serves real requests and signaling messages. When lag grows, those answers arrive late — or don't arrive at all — and the impact lands directly on end subscribers.

This is the guide I wish someone had given me the first time I had to explain at 2 AM why notifications were "sent" but nobody received them.

What consumer lag actually is

Lag is the distance between the end of the log and the last offset your consumer group has committed:

lag = log end offset − committed offset

Per partition, per consumer group. It is not one number — it's a vector. A group can be perfectly caught up on 15 partitions and hopelessly behind on one, and the average will hide it.

Measuring it: three levels of effort

Level 1 — The CLI, right now, from anywhere:

kafka-consumer-groups.sh \ --bootstrap-server <broker:9092> \ --group <group-id> \ --describe

The output gives you CURRENT-OFFSET, LOG-END-OFFSET, LAG, and CONSUMER-ID per partition. This is your first look: is the lag big, and is it concentrated on specific partitions?

Level 2 — Continuously, with kafka-exporter + Prometheus:

The kafka-exporter exposes kafka_consumergroup_lag per group and partition. The alert that actually works is not on absolute lag but on its trend:

deriv(kafka_consumergroup_lag[10m]) > 0

Lag that grows continuously is an incident. A spike that recovers is a batch job or a traffic burst — different conversation.

Level 3 — Lag as a first-class SLO. Define it per group: "notification dispatch group must stay under 10,000 messages of lag." That converts an infrastructure metric into a promise someone can be paged on.

Reading lag like an operator

Four patterns, four different actions:

1. Lag grows steadily on all partitions. Consumers are slower than producers. Check consumer processing time before scaling anything — if each message takes 200 ms and you're receiving 100 msg/s with 10 consumers, no amount of restarts will save you. Scale consumers or parallelize processing.

2. Lag grows on one partition only. Classic partition key skew. One hot key (one big customer, one busy region) is hammering a single partition. Scaling consumers doesn't help — the partition is the ceiling. The fix is upstream: review the partition key.

3. Lag spikes and recovers. Usually a batch producer or a restart. If it recovers within your SLO, tune the alert, don't chase the incident.

4. Lag jumps with no producer change. Suspect a rebalance storm. Consumers leaving and joining the group repeatedly pause consumption during each rebalance. Check kafka-consumer-groups.sh --describe for changing CONSUMER-IDs, and look at session timeout vs processing time — a consumer that takes longer than its session timeout to poll will get kicked, rejoin, and repeat.

The scaling ceiling most people discover too late

Here's the trap: you see lag, you scale consumers, lag stays. Why? A consumer group can never have more active consumers than partitions. If your topic has 6 partitions and you run 12 consumers, 6 of them are idle. Before scaling consumers, check the partition count:

kafka-topics.sh --bootstrap-server <broker:9092> --topic <topic> --describe

If you need more parallelism than partitions, you need to repartition the topic — which is a much bigger decision (offsets, ordering, consumer state).

A quick health one-liner

When I want a single command that answers "is anything falling behind right now," across all groups:

kafka-consumer-groups.sh \ --bootstrap-server <broker:9092> \ --all-groups --describe 2>/dev/null \ | awk '$6 ~ /^[0-9]+$/ && $6 > 1000 {print $1, $2, $3, $6}' \ | sort -k4 -nr | head -20

Any partition with more than 1000 messages behind (group, topic, partition, lag), sorted worst first. Zero output means you can go back to sleepp.

Summary

Consumer lag is the earliest honest signal in an event-driven system. Measure it per partition, alert on its trend rather than its absolute value, know the four patterns (global slowness, key skew, bursts, rebalances), and remember the partition ceiling before you scale consumers. Treat lag as an SLO and it will tell you about incidents before your users do.

Note: this article was drafted with AI assistance and reviewed, edited and tested by me based on my own operations experience.