Two-Plane Observability on Google Cloud: Signal Contracts, Alert Policies and a Hexagonal Dispatcher

We stopped learning about production failures from other teams by splitting observability into two planes: telemetry that's always on, and notification that's deliberate and rare. This is how we built it: the contracts, the config and the sharp edges we hit along the way.

Our services happen to be written in Go, so the code samples are Go. The architecture isn't: every boundary is OpenTelemetry, Cloud Monitoring or JSON on Pub/Sub, so a workload in any language can plug in.

Context in 60 seconds

• The fleet: a GKE cluster of Go services plus more than a dozen Cloud Run jobs, across three environments (dev, preprod, prod), each in its own GCP project.
• Before: every component instrumented differently, logs as the only real signal, and one Teams channel per service.
• The trigger: a sub-step in a Cloud Run job failed; the job swallowed the error and exited 0. No alert fired. A downstream team noticed hours later.
• The goal: any failure that needs a human reaches one within minutes, through one predictable channel. Everything else stays quiet, but visible.

Architecture at a glance

What's language-specific, and what isn't

The only code that has to exist per language is the instrumentation library. Everything downstream depends on what lands in Google Cloud, not on how it was produced:

A Python, Java or Node.js service joins the platform by emitting the same signals and publishing the same JSON. The policies, dashboards and dispatcher don't change.

Two planes that never mix

The telemetry plane is always on. Every workload emits the same signals, all the time, straight to Google Cloud Observability. Dashboards and Error Reporting read Cloud Monitoring and Cloud Logging directly. Nothing in this path goes through a service of ours.

The notification plane is deliberate and rare. Nothing reaches a person unless someone decided, on purpose, that it's worth interrupting someone for. Most of the time that decision is a Cloud Monitoring alert policy on the shared signals. Occasionally it lives in code, for app-level events that can't be expressed as a metric.

Pub/Sub carries alerts, never telemetry. It's tempting to send metrics through the same bus, but then a service of ours would have to aggregate time series, store history and evaluate alert conditions, all things Cloud Monitoring already does. It would also sit in the path of every signal: if it went down, we'd lose visibility exactly when we needed it.

One instrumentation library per language

If every team instruments differently, you can't build anything on top. So each language gets exactly one small library, and it's the only sanctioned way to wire up telemetry. Our fleet is Go, so we have one. This is what using it looks like in a Cloud Run job:

ocfg := obs.Config{ GCPProject: cfg.Project, // empty = telemetry off (local runs) ServiceName: "report-syncer", ServiceVersion: buildSHA, Environment: cfg.Env, // dev | preprod | prod ComponentType: obs.ComponentJob, // or obs.ComponentService } shutdown, err := obs.InitOTel(ctx, ocfg) if err != nil { return fmt.Errorf("init otel: %w", err) } defer func() { // Fresh context: the caller's ctx may already be canceled, // and this flush carries the run's final result. ctx, cancel := context.WithTimeout( context.Background(), 5*time.Second) defer cancel() if err := shutdown(ctx); err != nil { slog.Error("otel shutdown", "err", err) } }() slog.SetDefault(slog.New(obs.NewCloudLoggingHandler(ocfg))) // Root span + run.duration + run.result{status}, same everywhere. return obs.RunJob(ctx, doWork) 

The decisions behind that call apply in any language:

• ComponentType picks the export path. Cloud Run jobs use the in-process Google Cloud exporters: no collector, flush on shutdown. GKE services export OTLP to an in-cluster OpenTelemetry Collector. A service configured without a Collector address fails at startup instead of silently exporting to localhost and dropping everything. Both paths must land on the same metric types, or every template further down breaks; check that first when you add a Collector.
• Sampling follows the workload. Jobs sample every run, because volume is low and every run matters. Services use ParentBased(TraceIDRatioBased(0.1)), so a distributed trace stays complete across hops.
• There's an activation gate. With no GCP project set, or in local mode, the library installs nothing and returns a no-op shutdown, so local runs need no credentials. The W3C tracecontext + baggage propagator is installed either way, so context still flows through a process that isn't exporting.
• A wrapper emits the job contract for you. RunJob opens the root span, times the run, records the outcome and marks the span as errored on failure. Teams can't get the contract wrong because they don't write it.
• The shutdown flush gets its own deadline. For a job, the final flush is the run result. Flush under an already-canceled context (or skip the flush on exit) and you silently drop the one data point every alert depends on.

Porting it to another language

A library in Python, Java or Node.js has the same six jobs. If it does these, the rest of the platform can't tell the difference:

1. Initialize the OpenTelemetry SDK with the Google Cloud exporters for jobs, or OTLP to the Collector for services.
2. Name instruments <service>.<suffix>, with exactly the suffixes, units and label values in the contract below.
3. Wrap the unit of work (a job run, a request) so the contract signals are emitted automatically.
4. Map log fields to what Cloud Logging expects (next section).
5. Flush on shutdown with its own timeout.
6. Publish app events as the JSON shape described in the alert event contract.

The conformance test is simple: run a sample job and check the metric types and labels that appear in Cloud Monitoring. If they match, the dashboards and policies work unchanged.

The logging gotcha that broke Error Reporting

Cloud Logging reads a field called severity, but most JSON loggers write level by default: Go's slog, Python's common JSON formatters, Node's pino, Logback's JSON encoders. Ship that as-is and every line lands at default severity: severity filters, log-based metrics and Error Reporting all quietly misclassify everything.

The library's log handler fixes that in one place:

Two details matter. The trace field needs the full resource name, not a bare trace ID, or the log-to-trace link never appears. And the stack trace is captured at the log call site, written in the language's native stack format (Go's goroutine … [running]: layout for us) so Error Reporting can parse it, with the logging library's own frames dropped. Leave those frames in and Error Reporting groups every error under the logger instead of the code that failed.

That was the cheapest win of the whole project. Once error logs are shaped correctly, Error Reporting groups them fleet-wide with zero configuration. You get "all the problems in one place" on day one, before a single dashboard exists.

The signal contract

A small, fixed vocabulary that every workload emits:

The first three are built into the library. self_heal is a naming convention on top of the generic obs.Count helper, so the library doesn't enforce its label values; code review does.

GKE services emit errors and self_heal too, and get request rate, errors and latency from OpenTelemetry's standard HTTP and gRPC instrumentation (otelhttp / otelgrpc in Go; every language has an equivalent) instead of inventing their own. Pod health comes from GKE system metrics and Managed Service for Prometheus at the platform level, not from the app.

Why the service name is in the metric name

Every instrument is named <service_in_snake_case>.<suffix>, so report-syncer emits report_syncer.run.result, which lands in Cloud Monitoring as workload.googleapis.com/report_syncer.run.result. That's on purpose: the Google Cloud metric type comes from the instrument name, so a prefix guarantees two services can never collide on a metric with a different meaning or shape.

The trade-off is that there's no single run_result metric to query across the fleet. One component's run outcomes look like this:

sum by (status) ( increase(workload_googleapis_com:report_syncer_run_result[1h]) ) 

What makes it scale anyway is that the suffixes and labels are identical everywhere. The fleet dashboard, the per-component dashboards and the baseline alert policies are all Terraform templates that loop over the list of components and stamp out the same query for each one. Adding a component means adding a name to a list.

Baseline alert policies

Every component gets two policies by default, so none can be silent:

• Failure: run.result{status="failure"} greater than zero → severity=critical.
• Heartbeat: no run.duration sample within the expected interval (metric absence).

The heartbeat has three edges worth knowing before you rely on it:

1. A metric-absence condition only fires after the metric has existed. A brand-new component that has never written a point can't trip it. Make "first successful run in each environment" part of onboarding.
2. The absence window maxes out at 23.5 hours. That's fine for jobs that run hourly. For a once-a-day job, the schedule plus normal start-time jitter doesn't fit, so write its heartbeat as a PromQL condition over a longer range instead.
3. Every execution is a new time series. The library sets service.instance.id to the Cloud Run execution and task index, so each run writes its own series. Absence is evaluated per series, so aggregate across executions (sum by service) first. Otherwise every finished run's series looks "absent" and the heartbeat fires constantly.

Labels: bounded, and no owners

Service identity travels as resource attributes (service.name, service.namespace, service.version, deployment.environment, service.instance.id and a custom component_type), set once by the library alongside what the GCP resource detector finds. There are deliberately no team or owner labels: routing is by severity, triage context is by service, and ownership metadata goes stale faster than anything else in a label set.

The other hard rule: metric labels stay bounded. status, error_kind, outcome and a few other fixed enums. Never object paths, URIs, run IDs or user IDs: those belong on spans and logs.

Alert policies into Pub/Sub

Most alerts start as Cloud Monitoring policies. Cloud Monitoring already does evaluation, incident lifecycle and its own dedup well, so we let it, and give each policy a Pub/Sub notification channel pointing at the environment's alerts topic.

The setup details that cost us time:

• IAM. Monitoring publishes as its notification service agent, service-<PROJECT_NUMBER>@gcp-sa-monitoring-notification.iam.gserviceaccount.com. It needs roles/pubsub.publisher on the topic. Without it, incidents open in the console and nothing ever arrives.
• User labels carry routing. Every policy sets severity, service and component_type as user labels; the Terraform template sets them, so nobody forgets. They arrive in the incident payload (policy_user_labels), and the dispatcher's Pub/Sub adapter maps them onto the same event shape app code publishes.
• Incidents open and close. The payload carries state: open | closed. Carry that through as a resolved notification for the same incident, not as a new alert, or every recovery looks like a second incident.

The alert event contract

Whether an alert comes from a policy or from code, the dispatcher works with one shape. The contract is the JSON on the wire, not a type in any one language. Our library ships a Go struct for producers, but the dispatcher doesn't import it: the two sides share only the documented schema, so either can evolve on its own as long as the JSON holds. In a polyglot fleet, publish that schema as JSON Schema and generate the types per language:

type Event struct { // critical | warning | info Severity string `json:"severity"` // component name, as in the metric prefix Service string `json:"service"` // job | service ComponentType string `json:"component_type"` // one human-readable line Summary string `json:"summary"` // rare routing override; usually empty Channel string `json:"channel,omitempty"` // dashboard, trace, logs Links map[string]string `json:"links,omitempty"` // dedupe key: <service>/<kind> Fingerprint string `json:"fingerprint,omitempty"` // extra context, never secrets Attributes map[string]string `json:"attributes,omitempty"` } 

Two fields do the heavy lifting:

• Severity expresses how much this matters.
• Fingerprint expresses what this is, for example report-syncer/run-blocked, so repeated occurrences collapse into one incident instead of a hundred messages.

There's no environment in the event or the fingerprint, and that's deliberate. Each environment has its own project, topic and dispatcher, so a fingerprint is unambiguous inside the pipeline that receives it.

The producer declares intent, not delivery. It says how severe something is; it doesn't pick a Teams channel, decide whether to email or open a ticket. Routing falls out of severity:

func (e Event) ResolveChannel() string { if e.Channel != "" { return e.Channel // explicit override, used rarely } switch e.Severity { case SeverityCritical: return ChannelBlocker case SeverityWarning: return ChannelWarn default: // unknown severity: never dropped, never pages return ChannelDigest } } 

The default branch is a small decision we're happy with: a typo in a severity lands in the quietest bucket. The event isn't lost, and nobody gets woken up by a mislabeled one.

ResolveChannel is the producer-side reference for the primary channel. The dispatcher applies the same mapping, and its routing config can add mirrors on top, such as email for criticals.

One hard rule is baked into the contract: the payload never carries sensitive data. No request bodies, no payment fields, no customer identifiers in Summary or Attributes. Alerts travel through Pub/Sub and land in chat tools, which is the wrong place for anything you wouldn't put on a whiteboard.

Pub/Sub in the middle

The topic between producers and the dispatcher is there on purpose:

• Decoupling. Producers don't know or care how many channels exist, or whether the dispatcher is up. They publish and move on.
• One door for two sources. Alert policies and app events arrive the same way, so the dispatcher has one input.
• Fan-out. One event can drive several subscribers: the dispatcher now, an archive or a ticket integration later.

One sharp edge, stated plainly because it bit us in planning: by default, a topic doesn't keep messages; subscriptions do. Only subscriptions that exist at publish time receive a message. Create the topic today and the dispatcher's subscription tomorrow, and everything published in between is gone.

Two ways around it: create the subscription in the same Terraform apply as the topic, or enable topic message retention, which lets a subscription created later seek back to a timestamp and pick up what it missed.

The dispatcher: ports and adapters

The dispatcher turns an incoming event into the right notification: a blockers message, a warnings message, an email or a digest entry, or nothing at all if it's a duplicate. It uses a ports-and-adapters (hexagonal) architecture, so adding a new way to notify never touches the core.

Four boundaries, only three of them ports

Three decisions hide in that table:

• The core doesn't read from Pub/Sub; Pub/Sub calls the core. The core offers one function, Dispatch(ctx, event) error. The Pub/Sub adapter receives each message, calls it, and turns the result into an ack or a nack. The core never sees a subscription, so it's trivial to test and doesn't care how events arrive.
• Normalizing payloads is the adapter's job. Two very different JSON shapes arrive on the topic: Cloud Monitoring incidents and app events. Translating both into one Event is technology-specific parsing, so it lives in the Pub/Sub adapter. The core never sees a Cloud Monitoring payload.
• Static config doesn't need a port. Routing is data that's loaded once, so it's a constructor argument. It would only earn a port if it had to reload at runtime.

The outbound port

package dispatch // the core: no Teams, no SMTP, no URLs // ChannelID names a configured destination, e.g. "blocker" or // "oncall_email". It comes from routing config; the core never // sees URLs or channel types. type ChannelID string // State follows the incident: Cloud Monitoring reports open and close. type State string const ( StateOpened State = "opened" StateResolved State = "resolved" ) // Notification is what the core asks a channel to deliver: what // happened, not how it looks. Each adapter renders it (Adaptive // Card, email, ticket). type Notification struct { Key string // idempotency key: fingerprint + state State State Severity Severity Service string ComponentType string Summary string Links map[string]string // dashboard, trace, logs Occurrences int // repeats folded into this incident FirstSeen time.Time } // ErrPermanent marks failures a retry can't fix: // rejected payload, deleted workflow, revoked credentials. var ErrPermanent = errors.New("permanent notification failure") // Notifier is the outbound port. One instance per configured channel. // // Contract: // - nil means the channel ACCEPTED the notification. Delivery may // still be asynchronous (Teams Workflows answers 202 first). // - Wrap ErrPermanent when a retry can't help; any other error // is transient. // - Respect ctx: the core sets a deadline per send. // - May be called more than once with the same Key (at-least-once). // Adapters that can dedupe on Key should. type Notifier interface { Notify(ctx context.Context, n Notification) error } 

Most of the design lives in the comment, not the signature. What nil means, which errors are worth retrying and whether a duplicate call is safe are exactly the questions Pub/Sub acks and dedupe depend on.

The core, end to end

type DeliveryLog interface { // Claim reports whether key should go to ch now: not already // sent there within window. Safe under concurrent redeliveries. Claim(ctx context.Context, key string, ch ChannelID, window time.Duration) (bool, error) // Release undoes a claim after a transient failure, // so a redelivery retries it. Release(ctx context.Context, key string, ch ChannelID) error } type Service struct { routes Routes // severity → []ChannelID, from config channels map[ChannelID]Notifier // wired in main, validated at startup log DeliveryLog window time.Duration } // Dispatch is the inbound port. // nil → ack // ErrPermanent → ack and log loudly // anything else → nack; the redelivery retries only failed channels func (s *Service) Dispatch(ctx context.Context, ev Event) error { key := ev.Fingerprint + "#" + string(ev.State) var transient, permanent []error // ev.Channel override, or severity → channels for _, ch := range s.routes.For(ev) { ok, err := s.log.Claim(ctx, key, ch, s.window) if err != nil { transient = append(transient, err) continue } if !ok { continue // already sent to this channel within the window } sendCtx, cancel := context.WithTimeout(ctx, 10*time.Second) err = s.channels[ch].Notify(sendCtx, s.notification(ev, key)) cancel() switch { case err == nil: case errors.Is(err, ErrPermanent): permanent = append(permanent, fmt.Errorf("%s: %w", ch, err)) default: _ = s.log.Release(ctx, key, ch) transient = append(transient, fmt.Errorf("%s: %w", ch, err)) } } if len(transient) > 0 { return errors.Join(transient...) } if len(permanent) > 0 { return fmt.Errorf("%w: %v", ErrPermanent, errors.Join(permanent...)) } return nil } 

Three details carry the weight:

• Claims are per (key, channel), not per fingerprint. A critical goes to both blocker and oncall_email. If Teams succeeds and email fails, the redelivery retries only email. With a single claim per fingerprint you'd either post to Teams twice or lose the email.
• The claim key includes the state. Otherwise a close that arrives inside the window would be swallowed as a duplicate of the open it resolves.
• Errors map onto Pub/Sub. Transient failures nack, so Pub/Sub redelivers. Permanent ones ack, because retrying a rejected payload forever helps nobody. A dead-letter topic with a max-delivery-attempts limit on the subscription is the backstop for anything stuck in between.

The Teams adapter

MS Teams is the first adapter. It posts through Teams Workflows (Power Automate): one workflow per channel, built on the "When a Teams webhook request is received" trigger, which posts the card into that channel.

The trigger expects an Adaptive Card wrapped in a message attachment, and the adapter is also where HTTP status codes become the port's error classes:

// adapters/teams: the only package that knows about Workflows. type Workflow struct { url string // from Secret Manager, injected in main http *http.Client } func (w *Workflow) Notify( ctx context.Context, n dispatch.Notification, ) error { body := message{ Type: "message", Attachments: []attachment{{ ContentType: "application/vnd.microsoft.card.adaptive", Content: adaptiveCard(n), // opened or resolved layout }}, } resp, err := postJSON(ctx, w.http, w.url, body) if err != nil { return err // network or timeout: transient } defer resp.Body.Close() switch { case resp.StatusCode == 400, resp.StatusCode == 401, resp.StatusCode == 403, resp.StatusCode == 404: return fmt.Errorf("%w: teams workflow returned %d", dispatch.ErrPermanent, resp.StatusCode) case resp.StatusCode >= 300: // 429, 5xx: transient return fmt.Errorf("teams workflow returned %d", resp.StatusCode) } return nil // 202: accepted, not necessarily posted } 

Three things about Workflows shape the adapter:

• 202 Accepted means queued, not posted. The flow runs asynchronously, so a ba...