Predictive Test Selection Without SaaS Lock-In: Storing TIA Indexes in GitHub Actions Cache or S3

Hosted predictive test selection products solve a real problem, and they come with a procurement conversation: another vendor, another bill, and a security review about sending build and test metadata to someone else's servers. Some teams clear that review. Plenty do not, or cannot justify the budget for a suite that is only slow, not on fire.

There is a smaller, older idea that needs none of it: test impact analysis from a saved dependency map, stored wherever you already store build state. This post shows a complete self-hosted CI architecture for it with blastradius, using either the GitHub Actions cache or a private S3 bucket. I ran both paths end to end on fresh clones so the setup, and the places it breaks, are tested rather than described.

What actually gets stored

Before you decide where it can live, look at what it is. After a base-branch build, blastradius writes an index that maps each test to the production classes it loaded. A single entry looks like this:

{ "test": { "className": "demo.ParserCase34Test", "methodName": "applies" }, "dependsOnClasses": ["demo.ParserCase34Test", "demo.Parser"] } 

The whole file also holds the commit SHA the index was built from, a timestamp, direct-invocation records, and a list of "ambient" classes (JDK and framework classes, mostly). It is class names and test names. It is not source code, test output or data. For my 120-test fixture the index was about 175 KB, and about 53 KB compressed in a cache tarball.

That is still metadata about your codebase's structure, so treat it as internal. Two practical rules from the project docs: do not put credentials or other secrets under .blastradius/, and remember that GitHub Actions caches are readable by pull-request workflows.

On "nothing leaves your infrastructure": the tool reads and writes a local directory or the S3 bucket you configure. I searched the plugin, core and S3 modules for hard-coded service endpoints and found none beyond the AWS SDK itself. That is a grep, not an audit, so do your own review if your policy requires one.

The architecture

 push to main pull request ───────────── ──────────── mvn verify restore index └─ TRACK: full suite + agent └─ SELECT: diff vs merge base │ │ ▼ ▼ .blastradius/<main-sha>/index.json ──store──▶ run only matching tests (GitHub Actions cache or S3 bucket) (missing index → full suite) 

Two jobs, one artifact:

1. Trunk job (TRACK). Runs the full suite with a Java agent attached and writes the index for that commit.
2. PR job (SELECT). Restores the newest trunk index, diffs the PR against its Git merge base, and narrows Surefire/Failsafe to the tests whose recorded classes changed. If no usable index exists, it runs the full suite.

Everything else is plumbing: how the index travels from job 1 to job 2.

Option 1: GitHub Actions cache (no extra infrastructure)

For most teams this is the right default. No bucket, no IAM, nothing outside GitHub.

name: build on: push: branches: [main] pull_request: jobs: verify: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 with: fetch-depth: 0 # the merge base must be reachable - uses: actions/setup-java@v4 with: distribution: temurin java-version: '21' cache: maven - name: Restore Blastradius index id: blastradius-index uses: actions/cache/restore@v4 with: path: .blastradius key: blastradius-index-${{ runner.os }}-${{ github.sha }} restore-keys: | blastradius-index-${{ runner.os }}- - run: mvn -B --no-transfer-progress verify - name: Save Blastradius index (main only) if: ${{ github.ref == 'refs/heads/main' && success() && steps.blastradius-index.outputs.cache-hit != 'true' }} uses: actions/cache/save@v4 with: path: .blastradius key: blastradius-index-${{ runner.os }}-${{ github.sha }} 

Why it is shaped this way:

• Save only from successful main builds. PRs read an index made by the trusted base branch and never write one.
• Key by commit SHA. Each main build saves an immutable snapshot. A PR has a new SHA, so the exact lookup misses and restore-keys falls back to the newest main snapshot, which is what you want.
• success() before saving. A failed build should not become tomorrow's baseline.

And the plugin, with the base reference set to origin/main (see the first gotcha below for why):

<plugin> <groupId>io.github.baokhang83.blastradius</groupId> <artifactId>blastradius-maven-plugin</artifactId> <version>0.4.0</version> <executions> <execution> <phase>process-test-classes</phase> <goals><goal>select</goal></goals> </execution> </executions> <configuration> <baseRef>origin/main</baseRef> </configuration> </plugin> 

Option 2: a private S3 bucket

Use S3 when your runners cannot share a reliable workspace cache: self-hosted runners on different machines, several CI systems, or a cache you do not trust to stay warm. The index is written to and read from the bucket directly, so no cache steps are needed:

<configuration> <baseRef>origin/main</baseRef> <indexStore>s3</indexStore> <s3Bucket>ci-dependency-indexes</s3Bucket> <s3Prefix>blastradius</s3Prefix> <s3Region>eu-central-1</s3Region> <!-- <s3Endpoint>https://minio.example.com</s3Endpoint> optional, S3-compatible stores --> </configuration> 

The object key is derived from the commit: blastradius/.blastradius/<commit-sha>/index.json. Credentials come from the standard AWS credential chain, so nothing secret goes in the POM.

For GitHub Actions, prefer OIDC and a short-lived role over long-lived access keys. The plugin needs only s3:GetObject and s3:PutObject on the prefix, and does not need to list the bucket:

{ "Version": "2012-10-17", "Statement": [{ "Effect": "Allow", "Action": ["s3:GetObject", "s3:PutObject"], "Resource": "arn:aws:s3:::<BUCKET>/<PREFIX>/*" }] } 

Give the main job a role that can write, and give PR jobs a separate read-only role (s3:GetObject only) if PRs may read the bucket at all. Do not grant either to untrusted fork pull requests; let those fall back to the full suite.

permissions: contents: read id-token: write steps: - uses: actions/checkout@v4 with: fetch-depth: 0 - uses: aws-actions/configure-aws-credentials@v4 with: role-to-assume: arn:aws:iam::<ACCOUNT_ID>:role/blastradius-index-writer aws-region: eu-central-1 - run: mvn -B --no-transfer-progress verify 

What I tested

All runs used a small synthetic fixture (six utility classes, 60 two-test JUnit 5 classes, so 120 tests), Maven 3.9.12, JDK 25 and blastradius 0.4.0. Each job ran in a fresh clone with no shared files, mimicking actions/checkout on an empty runner. The PR changes one line in one utility class.

Caveats, so you read this correctly:

• The cache path was simulated by tarring and untarring .blastradius/, which is what actions/cache does. I did not run it on real GitHub-hosted runners.
• The S3 path ran against a local S3-compatible server (moto), not real AWS S3. I did not test IAM, OIDC role assumption or bucket policies. The endpoint, bucket and prefix behavior is the same code path as MinIO or any S3-compatible store, but verify your IAM setup separately.
• 83.3% reflects an evenly partitioned toy project. The realistic figure is the historical replay: 61.4% fewer test executions across 800 commits on four open-source projects, counted as executions rather than wall-clock time.

The cache-miss row is the safety property that matters for this architecture: losing the index costs you speed, never correctness. A missing, unreadable or mismatched index produces a full run, not a guess.

Gotchas I hit

1. actions/checkout on a PR has no local main, and the build fails. On a pull request, actions/checkout leaves you on a detached commit with only remote-tracking refs. With <baseRef>main</baseRef> the build failed outright:

base reference "main" does not resolve 

Setting <baseRef>origin/main</baseRef> fixed it, and it still behaves as the base reference on the push-to-main job, where it is the current commit. Pair it with fetch-depth: 0 so the merge base is reachable. Note that a -DbaseRef=origin/main flag did not override a <baseRef> hard-coded in the POM in my test, so set it in the POM (or make the POM read a property).

2. Cache eviction turns into full runs. GitHub evicts caches that have not been accessed for a while and enforces a per-repository size limit. If main is quiet for long enough, the next PR misses and runs everything. That is safe but slow, so make sure something on main runs regularly (a scheduled workflow works) to keep a fresh snapshot.

3. Tracking can fail without failing the build. If the tracking agent is not attached to the test JVM, the build passes and logs tracking failed ... falling back to the full suite, and no index is saved. I found this with JaCoCo plus an explicit @{argLine} in Surefire; the fix is in the JaCoCo post in this series. Whatever your storage choice, grep the main job's log for [blastradius] TRACK and for the absence of tracking failed.

4. Forks. Fork PRs do not receive secrets or OIDC roles, so they cannot use S3, and they may not restore your cache. They fall back to the full suite, which is the correct behavior.

Cache or S3?

Start with the cache. Move to S3 when eviction, runner topology or access control gives you a reason.

What you give up versus a hosted service

Be honest about the trade. A dependency-based tool like this one is deterministic and self-contained, and it has no model that learns from failure history, no hosted dashboard and no cross-project analytics. You own index freshness, and you own the policy for when to fall back. If those hosted features are what you need, and your security and budget allow it, a hosted product may be the better fit. If your constraint is "no new vendor, no data leaving our accounts", this is a workable alternative.

Either way, keep a regular full-suite run (daily is the recommendation) as a backstop. Selection is sound by default, not a guarantee.

FAQ

Can I run predictive test selection in Maven without a SaaS product?

Yes. Dependency-based test impact analysis needs only a saved map of which classes each test loads. Blastradius builds that map on your base branch and stores it in a local directory, the GitHub Actions cache or an S3 bucket you own.

What data does the index contain?

Test class and method names, the production class names each test loaded, the commit SHA and a timestamp. It does not contain source code, test output or data. It is still internal metadata, so keep it out of public buckets and do not store secrets alongside it.

Do I need S3?

No. The default file store plus a GitHub Actions cache is enough on GitHub-hosted runners. S3 is for runners that cannot share a cache, or when you want IAM-level access control.

What happens if the cache or bucket is empty?

The PR build runs in FALLBACK mode and executes the full suite. Nothing is skipped on a guess.

Why does my PR build fail with "base reference does not resolve"?

actions/checkout on a pull request provides no local main branch. Use origin/main as the base reference and fetch-depth: 0.

Does it work with Gradle?

There is a Gradle plugin in the repository with the same cache and S3 settings, but at the time of writing it is not yet published to the Gradle Plugin Portal. The Maven plugin is on Maven Central.

Is skipping tests safe?

Selection is sound by default but not guaranteed. Reflection, generated code and string-dispatched APIs can escape dependency tracking. Keep a scheduled full-suite run as a safety net.

Disclosure: I am the author of blastradius.