Email verification tests often fail for reasons that have nothing to do with the application. A shared inbox is slow, an old message is read by the wrong test, or a CI retry consumes a token that the first attempt already used. The result is a red pipeline and very little evidence about what really happened.

In cloud work, I treat test email as an infrastructure boundary. The application under test should send to a short-lived fixture owned by that job, while the rest of the AWS environment should have no path to real customer mailboxes. This makes failures easier to classify and keeps test data from drifting into places nobody checks.

Why cloud email tests need a boundary

The common setup is deceptively simple: create an address, run the signup flow, wait for a message, and click the link. It becomes less simple when several matrix jobs run at the same time. A shared address creates cross-talk, and a long-lived address makes cleanup somebody elses problem.

A better contract is:

  1. The CI job receives one fixture identity and one run ID.
  2. The application can send only to that fixture during the test.
  3. The test records message ID, received time, subject, and token state.
  4. The fixture expires when the job finishes, including after a failed retry.

This is also where a temp mailid can be confusing: the address is not the test state. The state belongs to the run, and the address is only one input to it.

The AWS boundary I use

For an AWS-based pipeline, I separate three concerns:

  • Control: the CI runner requests a fixture through a small internal service.
  • Delivery: the application sends through the staging mail path, never a production sender.
  • Evidence: logs and message metadata go to a run-scoped location with a short retention period.

The service returns a fixture ID, address, expiration time, and a correlation ID. The application receives those values as environment variables. It does not receive broad permissions to create mailboxes, read all test messages, or inspect unrelated runs.

In IAM, the useful rule is usually a narrow resource policy rather than another role with a large wildcard. Allow the test worker to read messages for its own fixture and write evidence to its own prefix. Deny access when the correlation ID does not match the job. The exact implementation depends on the inbox provider, but the boundary should be visible in policy review.

For network controls, put the fixture service behind a private endpoint when possible. Egress from the test workload should be limited to the service and the approved staging mail provider. This sounds small, but it save a surprising amount of debugging when a test accidentally points at a public endpoint.

A CI job with a short-lived fixture

The pipeline should create and destroy the fixture in separate, guaranteed steps. A simplified GitHub Actions shape looks like this:

jobs: email-flow: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - name: Create email fixture id: fixture run: ./ci/email-fixture create --ttl 20m > fixture.json - name: Run verification test env: TEST_EMAIL: ${{ steps.fixture.outputs.address }} RUN_ID: ${{ github.run_id }} run: ./ci/test-verification.sh - name: Collect evidence if: always() run: ./ci/email-fixture evidence --run-id "$RUN_ID" - name: Destroy email fixture if: always() run: ./ci/email-fixture destroy --run-id "$RUN_ID" 

The real workflow needs to export the address from fixture.json; the example keeps the sequence clear. The important detail is if: always() on evidence and cleanup. A setup is more reliable when a failed assertion still leaves behind enough data to explain the failure, then removes the fixture afterwards.

For retry safety, make the verification API idempotent. A retry should query the existing run state instead of generating a second token and overwriting the first one. These replay-safe verification APIs describe the same principle from the backend side. Matrix jobs should also use stable GitHub Actions fixtures, so a parallel job never guesses which inbox belongs to it.

Failure signals and cleanup

I record four timestamps: fixture creation, message accepted, message received, and token consumed. With those values, the team can distinguish a provider delay from an application bug or a test that waited too little. The logs becomes much more useful when every line includes the run ID and fixture ID.

Do not log the complete verification URL or token. Store a redacted digest, the message ID, and the URL host. If the test needs to prove that the link was correct, assert the host and path separately. This gives useful evidence without turning CI artifacts into credentials.

Cleanup should have a second safety net. The job cleanup is the fast path; a scheduled AWS task should remove fixtures whose expiry time is in the past. Keep that sweeper boring and observable. In practise, a metric for expired fixtures and a warning for cleanup failures catches more issues than a complex reconciliation system.

Practical checklist

  • Generate one fixture per CI run or matrix shard.
  • Put the run ID in the fixture record and every log event.
  • Restrict IAM and network access to the staging mail path.
  • Make token creation and consumption idempotent.
  • Collect metadata with if: always().
  • Redact tokens and full verification URLs from artifacts.
  • Destroy fixtures in the job and with an expiry sweeper.
  • Alert on fixture leaks, provider latency, and unread messages.

The first version of this setup was more simple than the final design: one fixture, one test, and one cleanup command. That was enough to expose the real problem. Once parallel CI and retries arrived, ownership, expiry, and evidence became the useful parts. Thats the boundary I would put in place before adding more retries or increasing test timeouts.