Postmortem: The Tuesday Our AI Bill Had No Owner

I've read a lot of postmortems. The best ones aren't about blame. They're about the gap between what happened and what the system let you see.

This one is about that gap. Specifically, the gap between "our AI bill went up" and "here's exactly why."

This is a composite of patterns I've seen across platform teams, written in the standard blameless format. The names are made up. The lesson isn't.

Postmortem: The Tuesday Our AI Bill Had No Owner

1. Summary

At 07:40 on a Tuesday, our daily cost alert fired. Vertex AI spend in the shared platform project had roughly tripled overnight.

Nine services call Gemini from that project. All of them use the same service account. All of them call the same model.

The billing console told us exactly how much we spent. It could not tell us who, which feature, or which step.

The root cause turned out to be a retry loop in the review step of a nightly ticket summarization workflow. Finding it took over two hours of Slack threads, log digging, and educated guessing.

The deeper root cause: none of our model calls carried request labels. We had cost data, but no way to slice it.

2. Impact

• Customers: None. The job's output was eventually correct.
• Cost: About 3x a normal day's AI spend for that project.
• People: One on-call SRE, two developers, and a FinOps analyst pulled into a two-hour investigation.
• Trust: Finance asked, fairly, how we'd know if this happened again. We didn't have a good answer.

3. Timeline

Monday, 16:30 A developer ships prompt_version=v4 for the ticket summarization workflow. The new prompt asks the review step to be stricter about tone.

Monday, 23:00 Nightly ticket summarization job starts. The stricter reviewer starts rejecting drafts. Each rejection triggers a rewrite with the full ticket history plus the previous draft plus review notes. Context grows on every loop.

Tuesday, 07:40 Daily cost alert fires.

07:52 On-call SRE opens Cloud Billing. Filters by project. Sees one project, one service (Vertex AI), one SKU group. No breakdown below that.

08:15 Slack thread in #platform: "Did anyone ship anything AI-related yesterday?"

08:40 Three teams reply "no." One replies "maybe? we updated a prompt."

09:05 SRE starts grepping application logs across services for model call volume. Logs are inconsistent. Some services log token counts, most don't.

09:30 A developer notices the summarization job ran for six hours instead of the usual forty minutes.

09:54 Root cause confirmed: review step looping on rejections, with context growing each time.

10:10 Prompt rolled back to v3. Retry cap lowered. Job rerun successfully.

4. Root cause analysis (5 whys)

Why did spend triple?
The review step in the summarization workflow rejected most drafts, causing repeated rewrites with growing context.

Why didn't we see it during the night?
Our only cost signal was the daily billing alert. Billing data isn't real-time, so it can't catch a loop while it's happening.

Why did root cause take two hours?
Billing showed one number for the whole project. Nothing told us which service, workflow, or step was responsible.

Why couldn't billing break it down?
None of our Vertex AI calls carried request labels.

Why weren't we labeling?
We label infrastructure (VMs, buckets, clusters) through Terraform. Model API calls have no resource to label, so they fell through the cracks. Nobody owned the gap.

5. What went well

• The daily cost alert existed and fired.
• The team responded fast and without blame.
• Rollback was clean once we knew where to look.

6. What went wrong

• No per-call attribution for AI spend.
• No real-time token metrics, only delayed billing data.
• No retry ceiling on the review loop.
• Prompt changes weren't tracked anywhere cost-related.

7. Background: how Vertex AI request labels work

Before the action items, here's the feature we were missing. If you already know it, skip to section 8.

Naming note: Vertex AI docs are moving under the Gemini Enterprise Agent Platform name. If a docs link redirects there, it's the same feature.

In Google Cloud, labels live at three layers:

Request labels are key-value pairs you attach to each model call. Vertex AI forwards them to Cloud Billing, where you can filter and group costs in billing reports and query them in the BigQuery billing export.

The rules, from the docs:

• Up to 64 labels per call for Google models, up to 32 for partner models
• Keys: 1 to 63 characters, starting with a lowercase letter
• Keys and values: lowercase letters, numbers, underscores, and dashes only
• Google models: labels go in the request body on generateContent and streamGenerateContent
• Partner models: labels go in an X-Vertex-AI-Labels header (base64-encoded JSON) on rawPredict and streamRawPredict
• Labels only reach Cloud Billing on the PayGo consumption option
• The direct Gemini API (AI Studio keys) does not support this field

Here's the simplest possible Gemini call with labels:

from google import genai from google.genai import types client = genai.Client(vertexai=True, project="your-project-id", location="global") response = client.models.generate_content( model="YOUR_GEMINI_MODEL", contents="Summarize this support ticket in two lines.", config=types.GenerateContentConfig( labels={ "team": "support", "env": "prod", "use_case": "ticket_summary", } ), ) 

That's it. Three labels would have turned our two-hour investigation into one query.

8. Action items

Each one is below, with the actual implementation.

AI-1: Define and publish a label taxonomy

The first temptation is to let every team pick their own labels. Don't. Six months later you'll have team=Support, team=support, team=support-team, and team=cx all meaning the same thing.

We settled on four groups:

Ownership (who pays)

• team: the owning team
• cost_center: the finance code

Lifecycle (where it runs)

• env: dev, staging, prod

Product (what it's for)

• app: the service
• use_case: the specific feature, like ticket_summary

Technical (how it ran)

• agent: which agent made the call
• step: which workflow step, like plan, retrieve, draft, review
• prompt_version: like v3
• route: which routing rule picked the model, like default or fallback

Notice that step and prompt_version alone would have pointed straight at our root cause: step=review, prompt_version=v4.

What never goes in a label:

• User IDs, emails, session IDs, request IDs
• Anything sensitive

Two reasons. Cardinality: billing data is aggregated, and a label with a million unique values makes every report useless and your queries expensive. Visibility: label values land in billing data, and anyone with billing viewer access can read them.

Our rule: if a label can have more than about 100 values, it belongs in logs, not billing.

And we treat the taxonomy like an API contract. It's written down, versioned, and changes go through review. Renaming a key later breaks every historical query.

AI-2: Ship a labeled client wrapper

Application code no longer calls the model directly. Every call goes through one wrapper that merges default labels, validates required keys, sanitizes values, and logs token usage.

# llm_client.py import json import re import contextvars from contextlib import contextmanager from google import genai from google.genai import types REQUIRED_KEYS = ("team", "env", "use_case") _bad_chars = re.compile(r"[^a-z0-9_-]") _ctx_labels = contextvars.ContextVar("llm_labels", default={}) def _clean(value: str) -> str: return _bad_chars.sub("_", str(value).strip().lower())[:63] def _clean_key(key: str) -> str: key = _clean(key) return key if key[:1].isalpha() else f"k_{key}"[:63] @contextmanager def llm_labels(**labels): """Attach labels to every model call made inside this block.""" token = _ctx_labels.set({**_ctx_labels.get(), **labels}) try: yield finally: _ctx_labels.reset(token) class LabeledClient: def __init__(self, project: str, location: str, defaults: dict): self._client = genai.Client(vertexai=True, project=project, location=location) self._defaults = defaults def generate(self, model: str, contents, labels: dict | None = None, config: types.GenerateContentConfig | None = None): merged = {**self._defaults, **_ctx_labels.get(), **(labels or {})} missing = [k for k in REQUIRED_KEYS if k not in merged] if missing: raise ValueError(f"Missing required LLM labels: {missing}") merged = {_clean_key(k): _clean(v) for k, v in merged.items()} config = config or types.GenerateContentConfig() config.labels = {**(config.labels or {}), **merged} response = self._client.models.generate_content( model=model, contents=contents, config=config ) _log_usage(model, merged, response) return response def _log_usage(model: str, labels: dict, response): usage = response.usage_metadata print(json.dumps({ "message": "llm_usage", "model": model, "labels": labels, "prompt_tokens": getattr(usage, "prompt_token_count", None) or 0, "output_tokens": getattr(usage, "candidates_token_count", None) or 0, "cached_tokens": getattr(usage, "cached_content_token_count", None) or 0, "total_tokens": getattr(usage, "total_token_count", None) or 0, })) 

Three design decisions worth explaining:

• Fail loud on missing labels. A ValueError in dev is annoying for exactly one afternoon. After that, everyone labels.
• Sanitize instead of reject. Team: "Customer Support" quietly becomes customer_support. A capital letter should never cause a prod outage.
• Log usage with the same labels. This is what makes AI-7 possible. Same dimensions in billing and in monitoring.

App code stays clean:

llm = LabeledClient( project="your-project-id", location="global", defaults={"team": "support", "env": "prod", "app": "helpdesk"}, ) response = llm.generate( model="YOUR_GEMINI_MODEL", contents="Summarize this support ticket in two lines.", labels={"use_case": "ticket_summary", "prompt_version": "v3"}, ) 

AI-3: Propagate labels through multi-agent workflows

Our summarization workflow has four steps across three agents. Passing a labels dict through every function signature would never survive code review.

The llm_labels context manager in the wrapper solves this. Set labels once at the top, and every call inside inherits them:

with llm_labels(use_case="ticket_summary", prompt_version="v4", agent="orchestrator"): plan = llm.generate(model="YOUR_GEMINI_MODEL", contents=plan_prompt, labels={"step": "plan"}) with llm_labels(agent="writer_agent"): draft = llm.generate(model="YOUR_GEMINI_MODEL", contents=draft_prompt, labels={"step": "draft"}) with llm_labels(agent="reviewer_agent"): review = llm.generate(model="YOUR_GEMINI_MODEL", contents=review_prompt, labels={"step": "review"}) 

contextvars works correctly with asyncio, so this holds up when agents run concurrently.

If this had been in place on Monday night, the billing breakdown would have shown one line towering over the rest: agent=reviewer_agent, step=review, prompt_version=v4. Investigation over.

AI-4: Cover partner models via header

Some of our teams also call partner models through Vertex AI. For those, labels travel as a base64-encoded JSON header. Here's a Claude call as the example:

import base64 import json import google.auth import google.auth.transport.requests import requests PROJECT = "your-project-id" MODEL = "YOUR_CLAUDE_MODEL" creds, _ = google.auth.default(scopes=["https://www.googleapis.com/auth/cloud-platform"]) creds.refresh(google.auth.transport.requests.Request()) labels = {"team": "support", "env": "prod", "use_case": "ticket_summary"} labels_header = base64.b64encode(json.dumps(labels).encode("utf-8")).decode("utf-8") url = ( f"https://aiplatform.googleapis.com/v1/projects/{PROJECT}" f"/locations/global/publishers/anthropic/models/{MODEL}:rawPredict" ) resp = requests.post( url, headers={ "Authorization": f"Bearer {creds.token}", "Content-Type": "application/json", "X-Vertex-AI-Labels": labels_header, }, json={ "anthropic_version": "vertex-2023-10-16", "max_tokens": 512, "messages": [{"role": "user", "content": "Summarize this support ticket in two lines."}], }, ) 

Two things to watch:

• The 32-label limit for partner models. If your wrapper adds a lot of defaults, trim them for partner calls.
• Check the supported model list in the docs before assuming labels work for a newly added partner model.

We folded this into the same wrapper so app code never needs to know which path it's on.

AI-5: Enforce labels in CI

A wrapper only helps if people use it. Five lines of bash close the gap:

#!/usr/bin/env bash # Fail the build if anyone calls the model directly instead of through LabeledClient. if grep -rn "models.generate_content(" --include="*.py" src/ | grep -v "llm_client.py"; then echo "Direct model calls found. Use LabeledClient from llm_client.py so labels are attached." exit 1 fi 

It's crude. It works. If you want something smarter later, swap it for an AST-based lint rule. Ship the crude one first.

AI-6: Build label coverage and cost queries

The billing console is fine for a quick look. For real analysis, we use the BigQuery billing export. The standard export table is named gcp_billing_export_v1_<BILLING_ACCOUNT_ID>, and labels live in an array of key-value structs.

Query 1: Label coverage. This is the first number we put on a dashboard. What percentage of AI spend actually carries labels?

SELECT DATE(usage_start_time) AS day, ROUND(SUM(cost), 2) AS total_cost, ROUND(SUM(IF(EXISTS(SELECT 1 FROM UNNEST(labels) WHERE key = 'team'), cost, 0)), 2) AS labeled_cost, ROUND(100 * SAFE_DIVIDE( SUM(IF(EXISTS(SELECT 1 FROM UNNEST(labels) WHERE key = 'team'), cost, 0)), SUM(cost) ), 1) AS coverage_pct FROM `your_billing_project.billing_dataset.gcp_billing_export_v1_XXXXXX_XXXXXX_XXXXXX` WHERE usage_start_time >= TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 30 DAY) AND service.description LIKE '%Vertex AI%' GROUP BY day ORDER BY day DESC; 

On the day of the incident, ours was effectively 0%. Our target is 95% in prod.

Query 2: Net cost by team, use case, and step. The query we wish we'd had at 07:52.

WITH ai AS ( SELECT DATE(usage_start_time) AS day, (SELECT value FROM UNNEST(labels) WHERE key = 'team') AS team, (SELECT value FROM UNNEST(labels) WHERE key = 'use_case') AS use_case, (SELECT value FROM UNNEST(labels) WHERE key = 'step') AS step, (SELECT value FROM UNNEST(labels) WHERE key = 'env') AS env, cost + IFNULL((SELECT SUM(c.amount) FROM UNNEST(credits) c), 0) AS net_cost FROM `your_billing_project.billing_dataset.gcp_billing_export_v1_XXXXXX_XXXXXX_XXXXXX` WHERE usage_start_time >= TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 30 DAY) -- Check how AI usage appears in your export first: -- SELECT DISTINCT service.description FROM ... and adjust this filter. AND service.description LIKE '%Vertex AI%' ) SELECT day, team, use_case, step, env, ROUND(SUM(net_cost), 2) AS net_cost FROM ai GROUP BY day, team, use_case, step, env ORDER BY day DESC, net_cost DESC; 

Note the credits line. The cost column doesn't include credits, so skip that and your numbers won't match the invoice.

Query 3: Unit economics. Join labeled cost with your own business metrics:

SELECT a.day, ROUND(SUM(a.net_cost), 2) AS ai_c...