🔎 Python Project Scanner Architecture: Static Analysis, Multi-Ecosystem Dependencies, and Technical Mapping

Taking over an existing software project often begins with the same questions.

Which technologies does it use? What libraries does it depend on? Which external services are integrated? How do its different components relate to one another?

In a recent, well-documented project, the answers may be relatively easy to find. In a legacy project, a monorepo, or an application composed of multiple services, they are often scattered across configuration files, dependency manifests, Dockerfiles, and source code.

Manual analysis becomes a task of its own.

An alternative approach is to build a technical inventory directly from the filesystem without executing the project being analyzed.

This article presents the architecture of Project Inventory, a static analysis engine built with Python 3.11+, capable of inventorying dependencies across ten ecosystems, identifying subprojects, analyzing Docker configurations, and mapping relationships between components.

1. Analyzing a Project Without Executing It

The first architectural principle is to work exclusively with observable elements within the project.

Software project │ ▼ Filesystem exploration │ ▼ Technical file detection │ ▼ Manifest and configuration analysis │ ▼ Source code reference analysis │ ▼ Consolidation and mapping │ ▼ HTML, JSON, and CSV reports 

The scanner does not need to start the application, install its dependencies, or launch its Docker containers.

It builds a technical representation from the available files.

This distinction is important: a static inventory describes what can be observed within a project, but does not necessarily establish its actual runtime behavior.

A service may be declared in Docker Compose without running. A library may be installed without being directly imported. An environment variable may exist without being used.

The report must preserve these distinctions.

2. A Modular Architecture

The engine separates responsibilities across several categories of components.

project-inventory/ │ ├── scanner/ │ ├── __init__.py │ ├── filesystem.py │ └── project_scanner.py │ ├── detectors/ │ ├── __init__.py │ ├── environment.py │ ├── infrastructure.py │ ├── languages.py │ ├── packages.py │ ├── projects.py │ └── services.py │ ├── analyzers/ │ ├── __init__.py │ ├── dependencies.py │ └── relationships.py │ ├── exporters/ │ ├── __init__.py │ ├── csv_export.py │ ├── html_export.py │ └── json_export.py │ ├── docs/ ├── output/ │ └── .gitkeep │ ├── LICENCE.md ├── LICENSE.md └── app.py 

This organization corresponds to the documented project structure.

The separation between detection and analysis is a central aspect of this architecture.

A detector can identify a package, an environment file, or a Docker service. An analyzer can then correlate these observations to establish relationships between components.

This prevents data discovery from being mixed with interpretation.

3. Recursive Filesystem Exploration

The first step is to traverse the selected directory and its subdirectories.

However, not every file within a project is relevant to its technical inventory.

Directories containing installed dependencies, generated files, or development tool data can significantly increase the volume of files to analyze.

The scanner excludes directories including:

.git .idea .vscode __pycache__ node_modules vendor .venv venv dist build coverage 

An educational example using the Python standard library could look like this:

import os from pathlib import Path EXCLUDED_DIRS = { ".git", ".idea", ".vscode", "__pycache__", "node_modules", "vendor", ".venv", "venv", "dist", "build", "coverage", } def walk_project(root: Path): for current, directories, files in os.walk(root): directories[:] = [ directory for directory in directories if directory not in EXCLUDED_DIRS ] for filename in files: yield Path(current) / filename 

This example illustrates recursive traversal and directory exclusions. It does not reproduce the production code of Project Inventory.

Modifying the directories list in place prevents os.walk() from descending into excluded directories when using its default top-down traversal.

This is particularly useful when a project contains a large node_modules directory or Python virtual environment.

The objective is not to analyze every installed file, but to identify the elements that describe the project's architecture.

4. Detecting Technical Files by Name and Extension

A multi-ecosystem scanner must recognize several file formats.

Some have conventional names, such as package.json or requirements.txt. Others are identified by their extension, such as .csproj files.

A simplified representation of the detection rules could look like this:

KNOWN_FILES = { "requirements.txt": "python", "pyproject.toml": "python", "package.json": "node", "composer.json": "php", "go.mod": "go", "Cargo.toml": "rust", "Gemfile": "ruby", "pom.xml": "java", "pubspec.yaml": "dart", "mix.exs": "elixir", } def detect_ecosystem(path): if path.name in KNOWN_FILES: return KNOWN_FILES[path.name] if path.suffix == ".csproj": return "dotnet" return None 

This example is intentionally incomplete. The documented engine also supports lock files and other configuration formats.

Detecting the file type makes it possible to select the appropriate processing logic.

JSON, TOML, XML, and Gradle configuration files cannot necessarily be interpreted using the same parser.

The scanner identifies relevant files before passing their contents to specialized detectors.

5. Inventorying Dependencies Across Ten Ecosystems

Project Inventory currently supports ten ecosystems.

This diversity requires distinguishing the format of a file from the meaning of the information it contains.

A manifest generally describes the dependencies requested by a project.

A lock file can specify resolved versions and include transitive dependencies.

However, not every file serves exactly the same purpose. For example, go.sum contains module checksums and should not be interpreted as a conventional lock file that independently represents the complete set of active dependencies.

The analysis must account for the semantics of each ecosystem.

Example: Node.js Manifest

{ "dependencies": { "express": "^4.18.2", "axios": "^1.7.0" }, "devDependencies": { "vite": "^5.0.0" } } 

This file declares version constraints.

The presence of express in the manifest does not necessarily mean that the installed version is exactly 4.18.2.

A lock file may provide a more precise resolved version.

The inventory should preserve this distinction rather than replacing both observations with a single value.

6. Consolidating Dependencies Without Losing Their Sources

The same dependency can appear in multiple technical files.

In a Dart project, for example, a package may be declared in pubspec.yaml and locked in pubspec.lock.

The engine consolidates information by ecosystem and package.

Example from the documentation:

dio pubspec.lock ==5.7.0 [locked] pubspec.yaml ^5.7.0 

The package appears once, while its different sources remain visible.

This approach avoids two problems:

• Counting the same dependency multiple times
• Losing the distinction between declared and locked versions

A conceptual representation could look like this:

{ "ecosystem": "dart", "package": "dio", "sources": [ { "file": "pubspec.yaml", "version": "^5.7.0", "type": "declared" }, { "file": "pubspec.lock", "version": "5.7.0", "type": "locked" } ] } 

This JSON illustrates the consolidation principle. It does not necessarily correspond to the exact schema used by the engine's exports.

A dependency discovered only in a lock file also remains in the inventory.

This makes it possible to identify transitive packages that are not directly declared in the main manifest.

Keeping Ecosystems Separate

Two packages with the same name in different ecosystems do not necessarily represent the same library.

A conceptual consolidation key could therefore be:

key = (ecosystem, package_name) 

This distinction prevents a Python package and a Node.js package with identical names from being arbitrarily merged.

In projects containing multiple subprojects, source files and their locations also remain important for identifying the context of each dependency.

7. Identifying Subprojects Within a Single Directory Tree

A software repository may contain several independent applications or components.

For example:

application/ │ ├── backend/ │ ├── requirements.txt │ └── app.py │ ├── frontend/ │ ├── package.json │ └── src/ │ ├── worker/ │ ├── pyproject.toml │ └── worker.py │ └── docker-compose.yml 

A global dependency inventory alone is insufficient to describe this architecture accurately.

The individual subprojects must also be identified.

application │ ├── backend │ └── Python │ ├── frontend │ └── Node.js │ └── worker └── Python 

Project Inventory identifies subprojects from detected manifests and configuration files.

Each subproject can be associated with one or more ecosystems and the technical files that led to its identification.

This step provides the context needed to associate dependencies and certain infrastructure relationships with individual components.

A manifest is still an architectural indicator. Its presence alone does not always establish that a directory represents an independently deployable application.

8. Analyzing Docker Compose Infrastructure

Application dependencies represent only part of the architecture.

An application may also depend on a database, a cache, a message broker, or other services.

Docker Compose files provide a particularly useful source of information.

The engine recognizes four conventional filenames:

docker-compose.yml docker-compose.yaml compose.yml compose.yaml 

For each detected service, it can extract information including its image, build configuration, ports, volumes, environment files, and dependencies between services.

Consider this educational example:

services: backend: build: ./backend ports: - "5000:5000" env_file: - .env depends_on: - database database: image: postgres:16 volumes: - postgres_data:/var/lib/postgresql/data volumes: postgres_data: 

Several elements can be observed in this file:

backend ├── build: ./backend ├── port: 5000:5000 ├── env_file: .env └── depends_on: database database ├── image: postgres:16 └── volume: postgres_data 

This information is obtained without starting the containers.

However, the declared configuration must remain distinct from the actual state of the infrastructure.

The presence of a port in Docker Compose does not demonstrate that it is currently accessible. Similarly, depends_on describes a declared relationship between services, but does not by itself guarantee that a dependency is ready to accept requests.

9. Enriching Services With Infrastructure Signatures

A Docker image can provide additional evidence about the purpose of a service.

For example:

postgres:16 → PostgreSQL redis:7 → Redis rabbitmq:3-management → RabbitMQ 

The engine includes signatures for identifying several known infrastructure components:

• PostgreSQL
• Redis
• MySQL
• MariaDB
• MongoDB
• RabbitMQ

These signatures enrich the discovered information.

They do not replace the analysis of Docker Compose files.

A service using an unknown image should remain in the inventory even when it cannot be associated with a recognized infrastructure component.

This distinction prevents discovery from being limited to technologies already present in a predefined catalog.

10. Analyzing Dockerfiles

A Dockerfile provides additional information about how a component is built and started.

The engine extracts instructions including:

Example:

FROM python:3.12-slim WORKDIR /app EXPOSE 5000 CMD ["gunicorn", "-w", "2", "-b", "0.0.0.0:5000", "app:app"] 

The analysis can identify Python as the base environment and Gunicorn as the declared startup command.

This information becomes particularly useful when correlated with Python dependencies.

Gunicorn can be used to start an application without appearing in an import gunicorn statement within the source code.

Source code analysis alone could therefore miss a reference that is present in the infrastructure configuration.

11. Extracting Environment Variables Without Exposing Their Values

Environment files often contain important information for understanding a project's integrations.

Project Inventory analyzes files including:

.env .env.example .env.sample 

However, these files may contain API keys, passwords, or other sensitive credentials.

The engine extracts only the variable names.

Example output:

OPENAI_API_KEY ENABLE_PERSISTENCE MEMORY_MAX_TURNS STRICT_MODE 

The values are not displayed.

An educational extraction example could look like this:

import re ENV_NAME = re.compile( r"^\s*(?:export\s+)?([A-Za-z_][A-Za-z0-9_]*)\s*=" ) def extract_env_names(content: str): names = set() for line in content.splitlines(): match = ENV_NAME.match(line) if match: names.add(match.group(1)) return sorted(names) 

This function illustrates basic variable name extraction without retaining the values in its output.

It is not a complete parser for every environment file format.

The analysis can therefore identify the existence of a configuration without directly exposing its sensitive contents.

This precaution must also apply to exports, error messages, and technical logs.

12. Identifying External Services From Evidence

Dependencies and environment variables can provide evidence about the external services integrated into a project.

The engine includes signatures for services such as OpenAI, Stripe, AWS, SendGrid, Twilio, and Sentry.

Documented example:

OpenAI package openai variable OPENAI_API_KEY 

Two separate observations associate the project with OpenAI.

A conceptual representation could look like this:

{ "service": "OpenAI", "evidence": [ { "type": "package", "value": "openai" }, { "type": "environment_variable", "value": "OPENAI_API_KEY" } ] } 

The result retains the evidence that led to the identification.

This approach is more informative than a simple list of external services because it explains why a service appears in the report.

An important limitation remains: the presence of a package and an environment variable does not prove that a connection to the service actually occurs.

The service may be optional, disabled, or simply anticipated in the configuration.

13. Analyzing Python Imports With the AST

Analyzing declared dependencies does not always establish which libraries are referenced in the source code.

For Python, the engine uses the AST, or Abstract Syntax Tree.

The AST represents the syntactic structure of a program without requiring its execution.

Consider this file:

import os import json from datetime import datetime from flask import Flask from openai import OpenAI from storage import save_record 

A text search might identify some imports, but it could also confuse source code with comments or strings.

The AST makes it possible to identify actual import statements within the program's syntax.

Educational example:

import ast def extract_imports(source: str): tree = ast.parse(source) imports = set() for node in ast.walk(tree): if isinstance(node, ast.Import): for alias in node.names: imports.add(alias.name) elif isinstance(node, ast.ImportFrom): if node.module: imports.add(node.module) return sorted(imports) 

This function analyzes source text without importing or executing the modules involved.

It could produce:

datetime flask json openai os storage 

A complete implementation must also account for relative imports, submodules, and files containing syntax errors.

Project Inventory classifies Python imports into three categories:

PYTHON STANDARD LIBRARY os json datetime EXTERNAL flask openai INTERNAL storage 

This classification distinguishes standard library modules, external dependencies, and modules belonging to the project itself.

14. Declared, Locked, and Referenced Dependencies

A dependency may be present in a manifest without appearing directly in the source code.

Conversely, an import may be detected even though no corresponding dependency is explicitly declared in the analyzed manifest.

Project Inventory correlates declared Python dependencies with references observed in source code and, where relevant, Docker configurations.

Example:

flask REFERENCED import flask gunicorn REFERENCED Docker ["gunicorn", "-w", "2", ...] 

These two packages are referenced in different ways.

Flask appears in the source code. Gunicorn appears in a Docker command.

The last status is deliberately different from "unused dependency."

The absence of a static reference can have several explanations:

• Dynamic imports
• Loading through a plugin system
• Indirect use by another library
• Commands executed outside Python source code
• Configuration outside the analyzed scope

There is also a distinction between the name of a distributed package and the name used to import it.

For example:

Package: python-dotenv Import: dotenv Package: Pillow Import: PIL 

A naive comparison between package names and imports can therefore produce incorrect results.

Dependency correlation must account for this distinction when the relationship can be established.

The inventory provides evidence for further analysis, but should not be used on its own to automatically remove dependencies.

15. Mapping Relationships Between Components

An inventory becomes more useful when it moves beyond a list of independent elements.

A project can be represented as a collection of components and relationships.

Backend subproject │ ├── declares → flask ├── declares → openai └── uses_external_service → OpenAI Backend Docker service │ ├── builds → Backend subproject ├── depends_on → Database service └── uses_env_file → .env Database Docker service │ └── mounts → postgres_data 

Project Inventory represents relationships including:

Each relationship retains its type, the types of its source and target components, and the evidence that led to its establishment.

A conceptual representation could look like this:

{ "type": "depends_on", "source": { "type": "docker_service", "id": "backend" }, "target": { "type": "docker_service", "id": "database" }, "evidence": [ { "file": "docker-compose.yml", "declaration": "services.backend.depends_on" } ] } 

This example illustrates a structured relationship. It does not reproduce the exact internal format of the engine.

The essential principle is that a relationship should only be created when it can be established from an observed element.

A technical map should not invent connections between components simply because they are present in the same project.

16. Separating Discovery From Enrichment

Discovery and enrichment address two different requirements.

Discovery identifies the elements actually present in the files.

Enrichment interprets certain elements using known signatures.

Project files │ ▼ Dynamic discovery │ ├── Packages ├── Variables ├── Services └── Imports │ ▼ Enrichment │ ├── External service signatures ├── Infrastructure signatures └── Observable relationships 

A scanner does not need to know every existing Python, Node.js, or Rust package in advance to inventory them.

It needs to know where to look and how to interpret the files that declare them.

A signature can then provide additional information.

OPENAI_API_KEY → OpenAI STRIPE_SECRET_KEY → Stripe SENTRY_DSN → Sentry 

An unknown package remains visible in the inventory even when no signature can enrich it.

This separation allows interpretation capabilities to expand progressively without restricting discovery to a predefined catalog.

17. Generating Multiple Representations of the Same Inventory

The engine produces several output formats from the analysis.

Consolidated results │ ├── Terminal │ ├── HTML │ ├── JSON │ └── CSV 

Each format serves a different purpose.

The HTML report presents the architecture and detected information in a readable format.

JSON retains detailed data in a structured representation.

CSV files provide specialized views of the inventory.

The documented output structure is:

output/ │ ├── project-inventory.json ├── project-inventory.html │ └── csv/ ├── dependencies.csv ├── dependency_usage.csv ├── dockerfiles.csv ├── environment_variables.csv ├── external_services.csv ├── imports.csv ├── infrastructure_components.csv ├── service_relationships.csv ├── services.csv ├── source_files.csv ├── subprojects.csv └── technical_files.csv 

Separating exporters makes it possible to produce different representations without reimplementing the entire analysis for each format.

When a tool processes data from external projects, report generation must also consider security.

A filename or extracted value must be properly escaped before being inserted into HTML. CSV exports intended for spreadsheet applications may also require protection against cells being interpreted as formulas.

These are general report generation considerations rather than a description of the engine's internal security controls.

18. Running Locally or on a Server

Project Inventory requires Python 3.11 or later.

Its documented entry point is:

python app.py 

The application then requests the path of the project to analyze.

Path of the project to analyze: 

The results are displayed in the terminal and automatically exported to the output directory.

The engine can run on Windows, Linux, or macOS, as well as on a server with direct access to the project files.

The analysis relies on the filesystem accessible to the application.

No source code or secrets need to be transmitted to an external service to generate the inventory.

The analyzed project is neither executed nor modified.

However, read-only analysis of the target project must remain distinct from report generation: the engine writes its own output files to its dedicated output directory.

19. The Limitations of Static Analysis

A technical inventory is not an exhaustive representation of a software application's actual behavior.

Some information may remain invisible when it depends on runtime execution.

Static analysis does not replace the observation of running processes, network traffic, or deployed environments.

Nor does it constitute, on its own, a security audit or vulnerability assessment.

Its purpose is to provide a structured foundation for understanding a project, identifying its components, and guiding further investigation.

Conclusion

Technical analysis of a software project involves more than listing the files present in a directory.

It requires recognizing formats, interpreting manifests, consolidating dependencies, identifying subprojects, and correlating information from source code and infrastructure configurations.

A modular Python architecture makes it possible to separate these responsibilities across scanners, detectors, analyzers, and exporters.

The distinction between discovery and enrichment allows the engine to dynamically inventory observed elements, then add information when signatures or relationships can be established.

But the most important principle remains the traceability of observations.

A declared dependency is not necessarily used. A detected variable does not prove a connection. A configured service is not necessarily running.

A reliable technical inventory must therefore distinguish what is discovered, declared, locked, referenced, or interpreted without turning a static observation into a claim about the actual runtime behavior of the software.

Technical Documentation

The engine architecture, supported ecosystems, static analysis mechanisms, and export formats are documented in the GitHub repository.

The repository provides technical documentation rather than downloadable source code or production files.

View the technical documentation on GitHub

https://palks-studio.com