How We Cut Maven & JUnit 5 Test Runs by 61% Across Apache Repos Without Missing Regressions

Most "predictive test selection" write-ups for Java come from vendors, and most of them give you a percentage with no way to check it. This one comes with the commit hashes. We replayed 800 consecutive commits from four open-source Java projects and counted what a dependency-based test selector would have run, what it would have skipped, and whether any skipped test would have caught a real bug.

The tool is blastradius, an Apache-2.0 test impact analysis plugin for Maven (and Gradle) with JUnit 5. The headline: 61.4% fewer test executions across the four replays, with 0 would-miss cases after the exclusions described below. The rest of this post is about what those two numbers do and do not mean.

The idea: record what tests actually load

Test impact analysis (TIA) answers one question on a pull request: which tests could be affected by this diff? Static guesses (package names, module graphs) are cheap but loose. Blastradius records ground truth instead:

1. Track. On a base-branch build, a java.lang.instrument agent watches every class loaded while each test runs, and records which production classes each test really touched.
2. Diff. On a change branch, it diffs the commit against the Git merge base, so only the PR's own changes count, not whatever landed on main after it diverged.
3. Select. A test runs if one of its recorded dependencies changed, if it is new or modified, if an executed class reaches a changed class through one or two recorded static invocation edges, or if a non-source change (build files, resources) triggers the conservative "run everything" fallback.
4. Gate. The selection narrows Surefire/Failsafe through the standard -Dtest= filter, so it keeps working with JaCoCo and custom argLine setups.

Missing or unusable baseline data always means a full run.

The experiment

For each project, the validation harness replays 200 consecutive commits, treating each commit as the change it introduced over the one before it. For every pair it asks the selector what it would run, then compares against ground truth. A would-miss is a test that caught a real regression or an injected mutant but that selection chose not to run. That is the failure that matters.

On the same window, the harness also runs bounded mutation validation: it injects synthetic faults into the head commit and checks whether the selected tests catch them.

Results: test executions skipped

The 61.4% is the aggregate: 966,368 selected executions out of 2,501,105 in the full suites. Note what it is: a count of test executions in a replay, not measured CI wall-clock time. Slow tests and fast tests count the same, and your savings will depend on where your slow tests sit.

Results: injected faults

A "killing test" is one that passed on head, failed on the mutant, and stayed failed on confirmation. Those are the tests selection must not skip. Counts are per mutant, so a test that kills five mutants counts five times.

What we excluded, and what did not come out clean

A benchmark is only worth reading if the awkward parts are on the page. Here they are.

• Excluded commit pairs (34 of 800). Pairs the harness could not measure count neither for nor against selection. In httpcomponents-client, 11 of 12 hit the build timeout on a slow multi-module rebuild and 1 hit a Surefire report containing an XML-illegal character (since fixed). In jsoup, all 10 exceeded the 40-minute mutation-build timeout. In commons-io, 11 hit the mutation-validation timeout and 1 failed to build under the harness.
• Flaky tests were excluded from the verdict. HttpClientPluginConfigurationTest (shenyu) and TestTlsHandshakeTimeout#testTimeout (httpcomponents-client, flaky under parallel build load) were removed. Excluding the second is what brings httpcomponents-client's would-miss to 0.
• commons-io had an unresolved case. A first recorded run showed a would-miss on URIOriginTest#testGetInputStream, a test parameterized on a live network fixture. A plain mvn clean test at that commit passed (6,376 tests, 0 failures), but the pair then failed to build under the harness and could not be root-caused. It is excluded, so the consolidated would-miss is 0, but that is "not a confirmed miss," not "proven clean."
• commons-io skipped 3.2% of killing executions. 1,011 of 31,648 killing test executions were skipped, spread over 42 commit pairs on the diff-targeted or whole-tree-fallback path rather than dependency-match selection. Every one of the 850 killed mutants still had at least one killing test selected, so the gap is partial under-selection, not a blind spot. This figure is provisional.
• jsoup exposed a real gap, which we then fixed. The first 200-pair run skipped 5,808 killing executions: 5,682 from 12 QueryParser mutations and 126 from two Normalizer mutations. jsoup's CSS selector engine is a string DSL, which is exactly where class-load tracking is weakest: a selector string reaches code that the test never loads directly. Adding a bounded two-hop static-invocation rule fixed it: targeted replays of all 14 mutation identities selected all 16,642 killing executions. Those were targeted replays, not a fresh full 200-pair rerun.

The lesson from jsoup generalizes. Reflection, string-dispatched APIs, generated code, and call paths deeper than two edges sit outside the evidence. If your project is built around selector engines, expression languages or query parsers, weigh that before adopting.

So, "without missing regressions"?

Within this window and these exclusions, blastradius had 0 would-miss cases across 800 commit pairs on four projects, and it selected a killing test for every mutant the full suites caught. That is the claim, and it is a measured one rather than a guarantee. We recommend running the full suite on a regular cadence (daily is the recommendation) as a safety net, so an occasional gap is caught within a day rather than never. Fast, sound-by-default selection on every build, backstopped by a full run, is the intended trust model.

Try it on your project

Add the plugin from Maven Central:

<plugin> <groupId>io.github.baokhang83.blastradius</groupId> <artifactId>blastradius-maven-plugin</artifactId> <version>0.4.0</version> <executions> <execution> <phase>process-test-classes</phase> <goals><goal>select</goal></goals> </execution> </executions> <configuration> <baseRef>main</baseRef> </configuration> </plugin> 

On your base branch it tracks; on other branches it selects. For separate CI runners, persist and restore the .blastradius/ directory with your CI cache.

If you would rather check the mechanism against your own history before changing CI, the validator harness is the same shadow-mode tool that produced the numbers above: it replays your repository's commits, reports what would have been skipped, and flags would-miss cases without touching your build.

• Repository: github.com/baokhang83/blastradius
• Full methodology and footnotes: HISTORICAL_REPLAY_ANALYSIS.md
• Maven guide: docs/maven-junit5-test-impact-analysis.md
• Maturity and trust model: MATURITY.md

FAQ

What is test impact analysis (predictive test selection) for Java?

It is a way to run only the tests a change could affect, instead of the whole suite, on every pull request. Blastradius does it by recording which production classes each test loads on the base branch, then selecting the tests whose recorded dependencies overlap the diff.

How is this different from running tests by module or package?

Module and package filters guess from project structure. Blastradius uses classes actually loaded at runtime during each test, so a test in one module that exercises code in another is still selected when that code changes.

Does it work with JaCoCo and custom argLine settings?

Yes. Selection works through the standard Surefire/Failsafe -Dtest= filter, so it does not replace your test runner or conflict with JaCoCo or a custom argLine.

What happens when it cannot decide safely?

It runs everything. A missing or unusable baseline, or a non-source change such as a build file or resource, triggers the full-suite fallback. New and modified tests are always selected.

Do I still need to run the full suite?

Yes. Selection is sound by default, not guaranteed. We recommend a full-suite run on a regular cadence (daily) as a backstop, so any gap is caught within a day.

Does 61.4% fewer test executions mean 61.4% faster CI?

No. The figure counts test executions in a replay, not wall-clock time. Your speedup depends on where your slow tests fall, plus fixed costs such as compilation and startup. The tracking build on your base branch also runs the full suite once.

Where does it struggle?

Code reached only through string-dispatched APIs, reflection or generated code, or through call paths deeper than two recorded edges, sits outside the evidence. jsoup's selector engine exposed exactly this in the replay. Projects built around query parsers, expression languages or selector DSLs should weigh it.

Does it support Gradle and Kotlin?

Kotlin in the conventional src/main/kotlin and src/test/kotlin layout is supported, with a full-suite fallback for Kotlin inline functions. The Gradle plugin is in the repository, but at the time of writing it is not yet published to the Gradle Plugin Portal. The Maven plugin is on Maven Central.

Can I run it on separate CI runners?

Yes. Persist and restore the .blastradius/ directory with your CI cache. An S3-compatible shared index store is available as an option if that suits your runners better.

Can I check it against my own repository before adopting it?

Yes. The validator harness replays your own commit history in shadow mode, reports what would have been skipped, and flags would-miss cases without changing your build.