Quick answer

Prometheus is a monitoring system that pulls numbers from your app on a schedule and stores them as time series, so you can ask how many orders failed in the last hour or how slow checkout became after a deploy. Your app exposes the numbers on a /metrics endpoint, Prometheus scrapes it, and you query the results with PromQL. This post builds that first half with Python: a small shop records five real orders.

Picture a small online shop in Karachi. Orders come in all day. Then a customer messages to say checkout felt slow, and nobody can say whether it was slow for everyone, or whether it started an hour ago or last week. The logs hold the answer, but only after digging. Metrics change the question from "what happened?" to "how many, and how slow?" That is the gap Prometheus fills: it turns events your app already knows about into numbers you can graph and alert on.

What I would check before using it

My first check: does anyone look at the numbers? Prometheus stores data and answers queries, but it sends no alert until you write a rule for it. If the plan is to check the dashboard after an outage, a log search does the same job with less machinery. It earns its place when someone watches a few numbers and acts when they move.

My second check is whether the app can expose metrics at all. Prometheus pulls data from your app; it does not receive data pushed to it. A job that runs for ten seconds and exits needs the Pushgateway instead. The demo app is a long-running process with an HTTP server.

My third check is cardinality, the number of distinct label combinations. A counter labelled by order status has two series, paid and failed. A counter labelled by customer id creates a new series for every customer, and that growth makes a Prometheus server slow and expensive. Keep labels to a small set of known values, and never put a user id or email in one.

My fourth check: who runs the server? Prometheus needs disk, retention rules, and backups if the history matters. If nobody owns that, a managed metrics service is the honest alternative. My verdict, as judgement: adopt Prometheus when a service runs long enough to be scraped and someone will act on its alerts. Skip it for a weekend script. The recommendation flips when labels grow without limit.

Five orders, one endpoint, and a counter that refuses to go down

Five orders arrive at the same small shop: Ayesha buys a kurta and dupatta for Rs 12,400, Bilal buys sneakers for Rs 7,800, Hina buys a handbag and wallet with no total, Ayesha returns for a dupatta at Rs 3,100, and Bilal buys sneakers and socks for Rs 8,600.

One file defines both metrics and processes the orders.

"""Shop orders app for The Practical Developer (Prometheus demo). A small online shop records two metrics with prometheus_client: a Counter for orders placed (labelled by status) and a Histogram for how long each order takes to process. The metrics are served on http://localhost:8000/metrics in Prometheus exposition format, ready for a Prometheus server to scrape. Run: python3 run_demo.py """ import time from prometheus_client import Counter, Histogram, start_http_server ORDERS_TOTAL = Counter( "shop_orders_total", "Orders processed by the shop, labelled by outcome", ["status"], ) ORDER_SECONDS = Histogram( "shop_order_seconds", "Time spent processing one order, in seconds", buckets=(0.1, 0.25, 0.5, 1.0, 2.5, 5.0), ) ORDERS = [ # (order_id, customer, items, total_rs, processing_seconds) (101, "Ayesha", ["kurta", "dupatta"], 12400, 0.42), (102, "Bilal", ["sneakers"], 7800, 0.31), (103, "Hina", ["handbag", "wallet"], None, 0.05), (104, "Ayesha", ["dupatta"], 3100, 0.28), (105, "Bilal", ["sneakers", "socks"], 8600, 0.55), ] def process_order(order_id, customer, items, total, seconds): """Process one order and record both metrics.""" start = time.monotonic() time.sleep(seconds) if total is None: ORDERS_TOTAL.labels(status="failed").inc() print( f"Order {order_id} for {customer}: FAILED, no total to charge" ) else: ORDERS_TOTAL.labels(status="paid").inc() joined = ", ".join(items) print( f"Order {order_id} for {customer}: paid Rs {total} " f"for {joined}" ) ORDER_SECONDS.observe(time.monotonic() - start) def main(): start_http_server(8000) print("Metrics server listening on http://localhost:8000/metrics") for order in ORDERS: process_order(*order) print("All five orders processed. Metrics are live on /metrics.") if __name__ == "__main__": main() 

The second script runs it end to end.

"""Run the Prometheus demo end to end and print the real /metrics output. First it shows the mistake: a refund path tries to subtract from a Counter with inc(-1), and prometheus_client refuses with ValueError. Then it starts the shop app in a background thread, processes five real orders, and fetches http://localhost:8000/metrics with urllib, printing the shop lines. Run: python3 run_demo.py """ import threading import urllib.request from prometheus_client import Counter import shop def show_refund_mistake(): refunds = Counter( "shop_refund_attempts_total", "Refund attempts in this demo" ) try: refunds.inc(-1) except ValueError as exc: print("ValueError:", exc) print("A counter only goes up. A refund needs its own counter,") print("or a Gauge, not a negative increment.") def fetch_metrics(): with urllib.request.urlopen("http://localhost:8000/metrics") as resp: return resp.read().decode("utf-8") def main(): print("--- The mistake first: subtracting from a Counter ---") show_refund_mistake() print() print("--- The shop, five orders ---") thread = threading.Thread(target=shop.main, daemon=True) thread.start() thread.join() print() print("--- What Prometheus would scrape: /metrics ---") body = fetch_metrics() for line in body.splitlines(): if line.startswith("shop_"): print(line) if __name__ == "__main__": main() 

Here is the real output, with the _created lines and the demo-only refund counter left out.

--- The mistake first: subtracting from a Counter --- ValueError: Counters can only be incremented by non-negative amounts. A counter only goes up. A refund needs its own counter, or a Gauge, not a negative increment. --- The shop, five orders --- Metrics server listening on http://localhost:8000/metrics Order 101 for Ayesha: paid Rs 12400 for kurta, dupatta Order 102 for Bilal: paid Rs 7800 for sneakers Order 103 for Hina: FAILED, no total to charge Order 104 for Ayesha: paid Rs 3100 for dupatta Order 105 for Bilal: paid Rs 8600 for sneakers, socks All five orders processed. Metrics are live on /metrics. --- What Prometheus would scrape: /metrics --- shop_orders_total{status="paid"} 4.0 shop_orders_total{status="failed"} 1.0 shop_order_seconds_bucket{le="0.1"} 1.0 shop_order_seconds_bucket{le="0.25"} 1.0 shop_order_seconds_bucket{le="0.5"} 4.0 shop_order_seconds_bucket{le="1.0"} 5.0 shop_order_seconds_bucket{le="2.5"} 5.0 shop_order_seconds_bucket{le="5.0"} 5.0 shop_order_seconds_bucket{le="+Inf"} 5.0 shop_order_seconds_count 5.0 shop_order_seconds_sum 1.611112356999456 

Five orders took about 1.61 seconds in total, about 0.32 seconds each. Four finished within half a second; the one under a tenth is order 103, the failed one, which stopped early with no total to charge. The counter tells the other half: four paid, one failed, and that failed count is the number to alert on.

The refund is the thing that went wrong. The natural move is to subtract a refunded order with inc(-1), and prometheus_client refuses with ValueError: a Counter can only go up. That rule keeps counters safe to add across restarts. The fix is a second counter, shop_refunds_total, or a Gauge.

A Prometheus server needs a scrape job pointing at the app. This configuration was not run: no Prometheus server binary is installed on this machine.

scrape_configs: - job_name: "shop" static_configs: - targets: ["localhost:8000"] 

Once the server scrapes that endpoint, rate(shop_orders_total{status="failed"}[5m]) gives the failed-order rate over five minutes, and histogram_quantile(0.95, rate(shop_order_seconds_bucket[5m])) gives the 95th percentile order time.

Common mistakes and how to fix them

First, putting a user id or order id in a label. The symptom is a server that grows slower every week, with millions of series and queries that time out. The fix is to label only by a small set of known values, like status, and keep ids in logs or traces. OpenTelemetry traces one order instead.

Second, subtracting from a Counter. The symptom is ValueError: Counters can only be incremented by non-negative amounts, when a refund path calls inc(-1). The fix is a separate counter for refunds, or a Gauge for a value that moves both ways.

Third, scraping an app that already exited. The symptom is a target shown as down, or a curl that finds nothing listening, because start_http_server runs on a daemon thread and the process ends when main returns. The fix is to keep the process alive, as serve.py does.

Fourth, choosing buckets that hide the interesting range. The symptom is a histogram where every observation lands in one bucket, so the 95th percentile tells you nothing. The fix is to pick buckets around the latencies you care about.

Frequently Asked Questions

Does Prometheus store my logs? No, it stores numbers over time, called time series. Keep logs for the story of one request, and metrics for the shape of all requests.

How is Prometheus different from Sentry? Sentry catches one error with a stack trace. Prometheus counts how often things happen and how slow they get, so you see a trend before it becomes a crash. The Sentry post shows the error side of the same shop.

Can I use Prometheus with a short-lived script? Not directly, because Prometheus pulls data from a running endpoint. A ten-second script should push its numbers to the Pushgateway instead.

Do I need Kubernetes to run Prometheus? No. The demo runs on one machine with no containers at all. Kubernetes only matters when you want Prometheus to discover targets automatically, as the Kubernetes post covers.

What happens when Prometheus restarts? Counters start again from zero, and Prometheus treats a drop as a reset. Gauges keep their last scraped value until the next scrape.

What to remember

  • Prometheus pulls numbers from /metrics on a schedule; it does not receive them.
  • A Counter only goes up; refunds need their own counter, or a Gauge.
  • Labels stay small and known; never put a user id or order id in one.
  • A Histogram answers how slow things get; its buckets must match the latencies you care about.
  • Keep the process alive while the endpoint is scraped.

Where this leaves you

Prometheus earns its place the day a customer says checkout felt slow and you answer with a number instead of a guess. The demo proved the three things that matter: a real app exposed real metrics, the counter showed four paid orders and one failed, and the histogram showed the average order took about a third of a second. That is the whole job. Start with one counter and one histogram on your own machine, and add a failed-order alert when the numbers look right. A graph you never look at is just decoration.

Further reading

Waqas Shahid is a software engineer from Karachi with more than 12 years of experience, including 6+ years in engineering leadership. He writes The Practical Developer.