Email verification tests are small, but their network path is not. A typical CI run starts a pod, calls an application, reaches a mail provider, reads a temporary inbox, and then cleans up the test data. When every one of those connections is allowed by default, a flaky test is hard to explain and an accidental outbound call is easy to miss.

The fix I have found useful is to treat email test traffic as a separate network contract. The test pod gets only the egress it needs, the inbox identity belongs to one CI run, and a blocked connection produces evidence instead of a generic timeout. This is especially helpful when Kubernetes runs inside AWS and several teams share a cluster.

Why email test traffic needs a boundary

An email test normally needs three different paths:

  1. The test runner reaches the application under test.
  2. The application or verifier reaches an approved mail API.
  3. The verifier reads the message and reports the result to CI.

Those paths should not automatically become one broad rule. If a test container can reach every public address, a bad fixture or a compromised dependency can use the same permissions to exfiltrate data. If it can reach no external service, the test may fail with a timeout that looks like a provider incident.

The boundary is not a replacement for test assertions. It is a way to make the allowed dependencies explicit. A search term such as “temp mail for facebook” might describe test data someone is looking for, but it says nothing about which host a production-like CI job should contact. The host allowlist and the run identity are the controls that matter.

I also keep the inbox scope narrow. A free disposable email address can be useful for a short-lived test, but the address should be tied to the run ID and should expire. A shared inbox turns an old message into a false positive.

Define the network contract

Before writing a NetworkPolicy, write down the expected flow. Mine usually has these entries:

  • Application endpoint: the Kubernetes Service and port the verifier must call.
  • Mail API: the provider endpoint and port, normally HTTPS only.
  • DNS: the cluster DNS service, if the client resolves names at runtime.
  • Telemetry: the internal collector, if CI evidence is sent there.
  • Everything else: denied and logged where the CNI supports it.

This list should be owned by the job, not copied from a namespace-wide policy. The rule are easier to review when one resource shows what one test actually needs. Keep the provider API and inbox read API separate if they are separate services; that split makes credentials and failure messages clearer.

Record a small event for each boundary crossing: destination class, timestamp, run ID, and outcome. Do not log tokens, full message bodies, or reset links. A DNS failure, a denied egress event, and a provider HTTP 429 need different responses, even if all three first appear as “message not received.”

A Kubernetes NetworkPolicy pattern

Here is a starting point for a verifier pod. The exact CIDRs depend on the provider and the cluster networking setup, so I prefer stable egress gateways or private endpoints where they are available rather than embedding a frequently changing public IP list.

apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: ci-email-verifier-egress namespace: ci-tests spec: podSelector: matchLabels: component: email-verifier policyTypes: - Egress egress: - to: - namespaceSelector: matchLabels: kubernetes.io/metadata.name: app-staging ports: - protocol: TCP port: 8080 - to: - namespaceSelector: matchLabels: kubernetes.io/metadata.name: kube-system ports: - protocol: UDP port: 53 - protocol: TCP port: 53

This policy permits the application and DNS paths only. In a real cluster, the mail provider path can be added through an egress proxy, a private AWS endpoint, or a carefully maintained destination rule. Be aware that a namespace selector controls Kubernetes destinations; it does not magically validate a public hostname. That distinction has caught me more than once.

Pair the policy with a Job deadline and a cleanup TTL. A blocked request should fail inside the test's own deadline, not keep a pod running until the CI system gives up. The Kubernetes email failure budgets pattern is useful here because it separates provider retries from pod retries.

Keep AWS access narrow and observable

If the verifier runs on EKS, use a dedicated service account and workload identity for any AWS API calls. For example, a test that creates an SQS receipt or reads a secret should have permissions for that queue or secret only. It should not inherit the node role's broad access.

The same principle applies to security groups and egress gateways. Name the rule after the CI component, attach an owner, and record why the destination exists. A rule called allow-internet is operationally cheap today and expensive during an incident. A rule called email-verifier-provider-egress gives the next engineer a place to start.

Do not confuse successful DNS resolution with successful delivery. Capture the DNS result class, connection outcome, HTTP status, provider message ID, and inbox observation time. These fields let you distinguish a NetworkPolicy denial from a provider delay. The first one needs a manifest or route fix; the second may need a bounded retry.

How to debug blocked or flaky tests

When a run fails, check the path in order:

  1. Pod scheduling: Was the verifier running, and did it restart?
  2. DNS: Did the pod resolve the application and provider names?
  3. Policy: Did the CNI report a denied flow?
  4. Transport: Was TCP or TLS established?
  5. Provider: Was the send accepted, rejected, or throttled?
  6. Inbox: Was the observed message created by this run?

Give each test message a unique run ID and reject messages with the wrong ID. This turns stale data into a clear assertion failure. A useful email test failure taxonomy can help standardize those categories across browser and API tests.

Sometimes the logs says “timeout” even though the policy is working exactly as configured. That is why I add a short diagnostic mode that prints destination class and policy evidence, with secrets redacted. It makes triage faster without changing the production-like deny rules.

An operating checklist

Before merging a Kubernetes CI email test, verify:

  • The verifier has an explicit egress contract.
  • DNS, application, provider, and telemetry paths are distinct.
  • The inbox identity contains a run-scoped value and has an expiry.
  • NetworkPolicy and cloud egress rules have named owners.
  • AWS permissions belong to a dedicated workload identity.
  • Denied flows and provider responses are observable without message secrets.
  • Pod retries, provider retries, and CI retries are counted separately.
  • A cancelled run still deletes its inbox and temporary resources.

Network boundaries will not make email tests perfect. They make the failures honest. Once the cluster can say “DNS denied,” “provider throttled,” or “stale message rejected” instead of only “timeout,” the test becomes an operations signal that a team can trust.