Turning a Broken Test Suite into a Deploy Gate

In September I ran yarn test on our NestJS API and 71 of 87 suites failed. Nobody had noticed, because nothing ran the tests. By the end of the month the suite was green, ran in 95 seconds instead of 335, and a failing test blocked the deploy. Here's how I got there.

How it got that bad

I'm the only engineer at a real estate platform in Taiwan. The API runs on Cloud Run, and our Cloud Build pipeline had two steps: build the image, deploy it. Tests never ran, so broken ones piled up.

Almost every failure looked like this, before a single assertion ran:

Nest can't resolve dependencies of the RentalService (PrismaService, GoogleService, FormatHelperService, GoogleMapService, ?, UserService). Please make sure that the argument SystemService at index [4] is available... 

That one hurt. The rental service spec had 19 tests for real behavior, and they used to pass. When SystemService was added to its constructor, all 19 broke, and with 70 other suites already red, nobody noticed.

Delete what doesn't test behavior

69 of the failing suites were Nest CLI scaffolding. nest g service generates a spec with a single it('should be defined') and provides the service without its dependencies, so any service with a constructor argument fails from day one. These had never passed.

For each file I asked whether deleting it would lose anything we know about how the code behaves. A should be defined stub only checks that Nest can wire the class, which the app already proves every time it starts. Keeping them would mean maintaining a mock list per service and updating it on every constructor change.

So I fixed the 21 real tests first. The rental spec only needed mocks for its new dependencies. The listing controller spec was failing because AuthGuard needed a UserService. Instead of providing that, I overrode the guard:

const module = await Test.createTestingModule({ controllers: [ListingController], providers: [{ provide: ListingService, useValue: listingServiceMock }], }) .overrideGuard(AuthGuard) .useValue({ canActivate: () => true }) .compile(); 

The guard has its own tests in the auth module. Pulling its dependencies into every controller spec is how this one broke in the first place.

Then I kept the stubs for five core modules that handle data and permissions, as an early warning when a constructor changes, and deleted the other 62. I didn't reach for describe.skip, because it would turn the suite green while hiding the problem.

Before calling it done, I ran the suite with the database URL and LINE credentials blanked out. It stayed green, which meant every test was properly isolated and CI wouldn't need any secrets.

Make tests block deploys

Once the suite was green, I added a test step at the start of the Cloud Build pipeline. Cloud Build stops at the first failing step, so a failing test means no image and no deploy.

Its first three runs on staging all failed, and none of the causes showed up on my laptop:

• The Prisma client wasn't generated in the CI container, so every model type was missing. Fix: run prisma generate before the tests.
• Jest ran out of memory while type-checking the generated client. Fix: raise Node's heap size and recycle workers with --workerIdleMemoryLimit.
• One module created an SDK client from credentials at import time and crashed before any test ran. Fix: dummy values for those variables in CI.

Move it to GitHub Actions

The gate worked, but every build now reinstalled dependencies the Dockerfile had already cached, and failures were buried in Cloud Build logs. So I moved the tests to GitHub Actions and had the workflow start Cloud Build only after they pass.

The risk is that the build no longer protects itself. The workflow has to make sure only tested commits get deployed. The relevant parts look like this:

concurrency: group: deploy-${{ github.ref }} cancel-in-progress: false jobs: test: strategy: fail-fast: true matrix: shard: [1, 2, 3] steps: # checkout, setup-node, yarn install, prisma generate - run: yarn test --ci --shard=${{ matrix.shard }}/3 deploy: needs: test permissions: id-token: write steps: # authenticate with Workload Identity Federation - run: gcloud builds triggers run "$TRIGGER" --sha="${{ github.sha }}" ... 

• --sha builds the commit that passed. Without it, triggers run builds whatever the branch points to at that moment, which could be a newer push nobody tested.
• The per-branch concurrency group, with the deploy job waiting for the build to finish, means an older build can never finish after a newer one and overwrite it.
• Workload Identity Federation means no service account key is stored in GitHub. The identity provider only accepts this repo's staging and main branches.
• Three shards, because GitHub's runner was noticeably slower than Cloud Build, and a single test job would have made deploys slower than before.

I also used triggers run rather than gcloud builds submit. submit uploads the working directory as the build source, and after a test run that directory is full of node_modules.

Checking that it works

I pushed a test I'd broken on purpose. One shard failed, the deploy job was skipped, and Cloud Build never saw that commit.

The first timed run also caught a mistake of mine. Push to deploy took over 18 minutes, because I'd left the old test step in Cloud Build and every commit was being tested twice. After removing it, it dropped to about 12.

Where it ended up

Deploys take about as long as before, but every one of them is tested now. If I started over, I'd add the gate on day one. Most of this post is cleanup from not doing that.