Running tests is one of the most important parts of a software delivery pipeline. But as codebases grow, running the entire test suite for every code change can become increasingly expensive.
A small change to one module might trigger hundreds or thousands of tests, even when most of them have no relationship to the modified code.
An obvious optimization is to run only the tests affected by a change.
But there's a catch.
How do you know which tests are safe to skip?
Selecting too many tests reduces the benefit. Selecting too few can allow regressions to slip through.
I wanted to explore this problem by building a dependency-aware test selection tool that could identify potentially impacted tests while favoring correctness over aggressive optimization.
That project became Smart Test Impact Analyzer.
The problem: Running everything isn't always efficient
Consider a Python application with the following structure:
src/ payments/ calculator.py service.py notifications/ email.py tests/ test_calculator.py test_payment_service.py test_notifications.py Suppose a developer changes:
src/payments/calculator.py
A traditional CI pipeline runs the entire test suite.
But the notification tests may have no dependency on the payment calculator at all.
If we can understand the relationships between modules and tests, we may be able to identify a smaller set of relevant tests.
The challenge is that these relationships aren't always direct.
For example:
calculator.py | v service.py | v test_payment_service.py A change to calculator.py can affect test_payment_service.py, even though the test doesn't import the calculator directly.
This is why simply matching changed files against test filenames isn't sufficient.
We need to understand dependencies.
The approach: Dependency-aware test selection
Smart Test Impact Analyzer analyzes Git changes, builds a static dependency graph from Python imports, and traverses that graph to identify tests that may be affected.
The high-level workflow looks like this:
flowchart TD A[Git Base and Head References] --> B[Git Diff Parser] B --> C[Changed File Analysis] C --> D[Python AST Parser] D --> E[Dependency Graph] E --> F[Reverse Dependency Traversal] F --> G[Impacted Test Selection] C --> H{Safe to Analyze?} H -->|No| I[Full Suite Fallback] H -->|Yes| G G --> J[Report Generator] I --> J J --> K[Text / JSON / Markdown] K --> L[CI Artifact] The implementation is intentionally divided into independent stages so each part can be tested and extended separately.
1. Detecting changes with Git
The first step is identifying what changed between two Git references.
Smart Test Impact Analyzer uses:
git diff --name-status --find-renames BASE...HEAD This provides the changed file paths and their statuses.
The implementation handles added, modified, deleted, renamed, and copied files.
The CLI allows developers to specify the repository and comparison references:
test-impact --repo . --base main --head HEAD The analyzer uses committed Git changes rather than uncommitted working-tree modifications.
This distinction matters because test selection should be based on a clearly defined comparison between repository states.
2. Building a dependency graph with Python AST
Once changed files are identified, the next challenge is understanding how Python modules depend on one another.
I chose Python's built-in ast module to inspect source code.
AST stands for Abstract Syntax Tree. Python can parse source code into a structured representation that allows us to inspect imports without executing application code.
For example:
from payments.calculator import calculate The analyzer extracts the import relationship and records that the current module depends on payments.calculator.
It also handles:
- Standard
importstatements. -
from ... import ...statements. - Relative imports.
- Recognized package and module prefixes.
The resulting dependency graph represents relationships between discovered Python modules.
Conceptually:
payments.service depends on payments.calculator tests.test_payment_service depends on payments.service The graph stores each module and the modules it depends on.
One deliberate design decision was to include discovered modules even when they have no imports. This ensures that modules without dependencies are still represented in the graph.
Static analysis also has limitations. Dynamic imports and runtime-generated dependencies cannot reliably be discovered through this approach alone.
3. Finding impacted tests through reverse dependency traversal
Knowing what a module depends on is useful, but our question is slightly different:
Which modules depend on the module that changed?
This is where reverse dependency traversal comes in.
Given this graph:
calculator.py ^ | service.py ^ | test_payment_service.py A change to calculator.py should identify both service.py and test_payment_service.py as dependents.
Smart Test Impact Analyzer constructs a reverse graph and traverses it using a queue-based breadth-first search.
The traversal continues through multiple dependency levels, not just direct imports.
Conceptually:
visited = {changed_module} queue = [changed_module] while queue: current = queue.pop(0) for dependent in reverse_graph[current]: if dependent not in visited: visited.add(dependent) impacted.add(dependent) queue.append(dependent) This allows the analyzer to identify tests affected through transitive dependencies.
Once impacted modules are identified, the analyzer compares them against discovered test files and selects the relevant tests.
It also handles directly modified test files by selecting those tests themselves.
4. Conservative full-suite fallback
This is probably the most important design decision in the project.
Test selection is an optimization. Test coverage is a correctness requirement.
If the analyzer cannot confidently identify impacted tests, it should not silently return an incomplete selection.
Instead, it recommends running the full suite.
Smart Test Impact Analyzer uses full-suite fallback in several situations:
| Situation | Why full-suite fallback? |
|---|---|
| Deleted, renamed, or copied files | Dependency relationships may have changed |
| Test configuration changes | Test behavior or discovery may be affected |
conftest.py changes | Shared fixtures can affect multiple tests |
| CI workflow changes | Execution behavior may have changed |
| Unsupported non-Python changes | Dependency analysis is incomplete |
| Invalid or unreadable Python source | The dependency graph cannot be trusted |
| Changed module cannot be mapped | Impact cannot be determined confidently |
| No impacted tests identified | Avoid returning an empty selection that could hide risk |
For example, changing pyproject.toml triggers full-suite mode.
Likewise, if a changed Python module cannot be found in the dependency graph, the analyzer falls back instead of assuming that no tests are affected.
Documentation changes are ignored by dependency analysis, but if no impacted tests are identified overall, the analyzer also uses the full-suite fallback.
This approach deliberately favors safety over maximizing the number of tests skipped.
5. Making the results useful through structured reports
Test selection is only useful if developers can understand why particular tests were selected.
Smart Test Impact Analyzer generates reports in three formats:
- Text: For local development and command-line inspection.
- JSON: For downstream automation and integrations.
- Markdown: For CI artifacts and pull request workflows.
The reports include:
- Selection mode: targeted or full suite.
- Changed files.
- Selected tests.
- Total discovered tests.
- Selection ratio.
- Fallback reason, when applicable.
- Reasons explaining why each test was selected.
For example, a targeted report might look like this:
Smart Test Impact Analyzer ================================================== Selection mode: TARGETED Changed files: 1 Selected tests: 2 / 5 Selection ratio: 40.0% Selected Tests -------------------------------------------------- tests/test_calculator.py Reason: Depends directly or transitively on src/payments/calculator.py tests/test_payment_service.py Reason: Depends directly or transitively on src/payments/calculator.py The example illustrates the report format; actual selections depend on the repository's dependency graph.
I wanted the output to be useful both to a developer investigating a change and to a future CI integration consuming the results programmatically.
6. Integrating test impact analysis into GitHub Actions
The project includes a GitHub Actions workflow that runs on pushes, pull requests, and manual dispatches.
For pull requests, it performs test impact analysis and uploads a Markdown report as a workflow artifact.
One important decision: the workflow still runs the complete test suite.
The test selection report is currently advisory. It allows us to validate the accuracy of the analyzer before using its results to skip tests in CI.
I consider this an important intermediate step.
Before allowing an optimization to change which tests execute, we need confidence that the dependency analysis is sufficiently reliable.
The current workflow therefore separates two concerns:
- Verify the code using the full test suite.
- Independently evaluate and report which tests the analyzer believes are impacted.
This gives us a foundation for measuring selection quality before introducing selective execution.
7. Testing the implementation
I wanted the analyzer itself to be testable without requiring a large external application or CI environment.
The project includes unit and integration tests covering:
- Git diff parsing.
- Dependency graph construction.
- Direct and transitive dependency traversal.
- Directly changed test selection.
- Configuration and deleted-file fallbacks.
- Invalid Python source handling.
- Report generation.
- CLI output.
- A real Git repository integration scenario.
The test suite currently has 21 passing tests.
I also kept the project free of runtime dependencies. It uses Python's standard library for Git integration, AST parsing, dependency analysis, and report generation. Pytest is used for development and testing.
Engineering tradeoffs and current limitations
There are several things this implementation intentionally does not try to solve yet.
Static imports are not the same as runtime dependencies.
Dynamic imports, reflection, and runtime-generated dependencies aren't fully represented by the AST-based graph.
Python is the current focus.
The dependency analyzer doesn't currently support multiple programming languages.
Pytest conventions drive test discovery.
Test discovery currently recognizes Python files beginning with test_ or ending with _test.py.
Fixtures and plugins need deeper modeling.
Shared pytest fixtures and framework-specific behavior can introduce dependencies that aren't captured by ordinary import relationships.
Selective execution isn't enabled in CI yet.
The current GitHub Actions workflow runs the full suite and publishes the analyzer's recommendations. It doesn't yet skip tests based on the selection.
These are deliberate boundaries for the initial implementation, rather than problems I want to hide behind an overly ambitious description.
What I'd explore next
There are several directions I'd like to investigate:
- Better modeling of pytest fixtures and shared test dependencies.
- Configuration-driven test discovery.
- Support for additional programming languages.
- Historical test execution data to improve selection decisions.
- Feedback mechanisms to measure selection accuracy and identify missed dependencies.
- A gradual transition from advisory test selection to selective CI execution, with full-suite safeguards.
I'm particularly interested in measuring the tradeoff between CI efficiency and the risk of missing regressions.
Reducing the number of executed tests is useful only if the selected tests provide sufficient confidence in the change.
Final thoughts
Building Smart Test Impact Analyzer helped me explore a practical developer productivity problem from a different angle.
The interesting part wasn't simply finding a way to run fewer tests. It was designing a system that could reason about code dependencies, identify potentially impacted tests, explain its decisions, and recognize when it shouldn't make an optimization.
The principle I kept coming back to was:
Optimize what you can prove safe. Fall back when you cannot.
That's the foundation I'd want before introducing more aggressive test optimization into a real engineering pipeline.
If you're working on test intelligence, CI optimization, or developer productivity, I'd love to hear how your teams approach this problem.
Explore the project:
Smart Test Impact Analyzer on GitHub
Feedback and suggestions are welcome!

