Flaky Playwright tests in CI are usually tests that wait on the clock instead of the page, share state across workers, or depend on file order. Playwright gives you the tools to see them: retries mark a test flaky when it fails then passes, trace: 'on-first-retry' records what happened, and --repeat-each plus --fail-on-flaky-tests reproduce and gate them. Use retries to collect evidence, not to turn the build green.

One Playwright test failing then passing: where flaky shows up and where the default JUnit report loses it
Measured on @playwright/test 1.65.0. The list and JSON reporters keep the flaky outcome; the default JUnit file records a pass unless includeRetries is on.

The e2e job passes on the second attempt often enough that nobody looks anymore. Playwright already knows which tests are doing it, because a retry that passes is a distinct outcome in its model. The work is making that outcome reach a place someone reads.

Everything here was checked against @playwright/test 1.65.0, the current release on 2026-11-12, with a test written to fail on its first attempt and pass on its retry.

What Playwright calls flaky

Playwright sorts every test into three outcomes: passed on the first run, flaky (failed on the first run but passed when retried), and failed on the first run and every retry. By default failing tests are not retried, so there is no flaky outcome until you set retries.

A retry is not a rerun in place. After a failure Playwright discards the worker process and its browser and starts a new one, so the retry starts clean in memory. State outside the process, such as a database row or a shared account, survives, which is why a test that flakes from shared state often passes on retry.

The config below follows the docs for CI and adds two lines. failOnFlakyTests (added in 1.52) makes a flaky result fail the run, and the reporters write flaky results to files you can query. In the 1.65.0 check, the list reporter ended with 1 flaky, and the run exited 0 by default and 1 with --fail-on-flaky-tests.

import { defineConfig } from '@playwright/test'; export default defineConfig({ fullyParallel: true, forbidOnly: !!process.env.CI, retries: process.env.CI ? 2 : 0, workers: process.env.CI ? 1 : '50%', failOnFlakyTests: !!process.env.NIGHTLY, reporter: process.env.CI ? [['list'], ['json', { outputFile: 'results.json' }], ['junit', { outputFile: 'junit.xml', includeRetries: true }]] : 'html', use: { trace: 'on-first-retry' }, });

Outcome definitions from playwright.dev/docs/test-retries; config lines from playwright.dev/docs/test-configuration and the TestConfig API page; the 1.45 and 1.52 dates from the Playwright release notes. All read 2026-11-12. NIGHTLY is a variable you set yourself, so pull requests keep retries and a scheduled run fails on any flake.

The default JUnit report hides flaky tests

This is the setting that decides whether anything downstream ever sees a flake. With retries on and the JUnit reporter at its defaults, a test that failed then passed is written as an ordinary passing testcase, and the suite reports failures="0". Any tool reading that file, your own script or a flaky-test service, counts it as green.

Set includeRetries: true on the reporter, or PLAYWRIGHT_JUNIT_INCLUDE_RETRIES=1 in the environment, and the same test carries a flakyFailure element with the first attempt's error. The docs mention includeRetries only in passing, in the onlyFailures row of the reporter table. The JSON reporter needs no setting: the test's status is flaky and stats.flaky counts them.

```Output captured locally, @playwright/test 1.65.0

default JUnit: the flaky test looks like any pass

with includeRetries: the first attempt is kept

JSON reporter: list the flaky tests

jq -r '.. | objects | select(has("specs")) | .specs[]
| select(any(.tests[]; .status == "flaky")) | "(.file):(.line) (.title)"' results.json

 ## Wait on the page, not the clock Most Playwright flakes in our experience are a wait that guesses. The `page.waitForTimeout` documentation is blunt: "Tests that wait for time are inherently flaky." A runner with fewer cores than your laptop turns a sleep that was long enough locally into one that is not. Web-first assertions wait for the condition instead. `await expect(locator).toBeVisible()` retries until the element is visible or the expect timeout runs out, 5000 ms by default. `expect(await locator.isVisible()).toBe(true)` checks once, immediately; the best practices page says such a test "won't wait a single second." ```tests/checkout.spec.ts // flaky: guesses how long the request takes, then checks once await page.getByRole('button', { name: 'Pay' }).click(); await page.waitForTimeout(2000); expect(await page.getByText('Payment received').isVisible()).toBe(true); // stable: waits for the outcome, up to the expect timeout await page.getByRole('button', { name: 'Pay' }).click(); await expect(page.getByText('Payment received')).toBeVisible();

Workers, fullyParallel and shards change who runs together

By default, test files run in parallel and tests in one file run in order in the same worker. That order is a dependency nobody wrote down: a test that only passes after its neighbor set something up breaks the day you turn on fullyParallel, or the day sharding splits the file. With fullyParallel, shards split by test; without it, by file, so adding files reshuffles which tests share a worker.

Workers default to half the logical CPU cores, and the CI docs recommend one worker on CI for stability. If a test fails only when workers run together, it shares something: a user account, a fixture file, a port. Give each worker its own with testInfo.workerIndex, which the parallelism docs suggest for exactly this.

jobs: e2e: runs-on: ubuntu-latest strategy: fail-fast: false matrix: shard: [1, 2, 3, 4] steps: - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 - uses: actions/setup-node@949feb2413d6458794dcd2491c4babbbce0c15c1 # v7.1.0 with: node-version: lts/* cache: npm - run: npm ci - name: Install Chromium run: npx playwright install --with-deps chromium - name: Run tests run: npx playwright test --shard=${{ matrix.shard }}/4 - if: ${{ !cancelled() }} uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2 with: name: playwright-${{ matrix.shard }}-attempt-${{ github.run_attempt }} path: | test-results/ results.json junit.xml

Reproduce before you fix

Run the suspect test many times with retries off and the CI worker count, so a flake has room to show. --repeat-each runs each test N times; --last-failed reruns only what failed last time. The trace from the first retry, in test-results/ under a directory ending in -retry1, shows what the page looked like when it failed.

npx playwright test tests/checkout.spec.ts --repeat-each=30 --retries=0 --workers=1 npx playwright test --last-failed npx playwright show-trace test-results/<test-dir>-retry1/trace.zip

Browser install time is not test time

Install only the browsers you test, as in the workflow above: the best practices page says npx playwright install chromium --with-deps saves download time and disk space on CI. The CI docs advise against caching browser binaries, because restoring the cache takes about as long as downloading them, and the Linux system dependencies cannot be cached at all.

Keep the install in its own step. A download that fails there is a network failure, covered in Playwright browser download failed in CI, and should never be counted as a flaky test.

FAQ

Why are my Playwright tests flaky in CI but not locally?

CI runners usually have fewer cores, run your tests next to other workers, and start from a cold browser install. Fixed waits that were long enough locally stop being long enough, and tests that share accounts or data collide under parallel workers. Replace waits with web-first assertions and give each worker its own data.

How do I make Playwright fail on flaky tests?

Pass --fail-on-flaky-tests on the command line, available since 1.45, or set failOnFlakyTests: true in the config, added in 1.52. A run with a test that failed then passed then exits nonzero. Many teams keep retries on pull requests and turn this on for a scheduled run.

What does trace on first retry do in Playwright?

With trace: 'on-first-retry', Playwright records a trace only when a failed test is retried for the first time, so a passing suite pays no tracing cost. The trace lands in the retry's directory under test-results/ and opens with npx playwright show-trace.

Should I cache Playwright browsers in GitHub Actions?

Playwright's CI documentation recommends against it: restoring the cache takes about as long as downloading the browsers, and the Linux system dependencies cannot be cached. Install only the browsers you test instead.

References