STATERA Library · Series 08 · Volume 05 · Chapter 08 · EN
Source: https://github.com/Statera-Library/08_Vol_05_Robot_Middleware_and_ROS2/blob/main/Chapter_08_ROS2_Deployment_and_DevOps/Chapter_08_ROS2_Deployment_and_DevOps_EN.md
Volume 05 Robot Middleware and ROS2
08. ROS2 Deployment and DevOps
08.01 ROS2 Containerization: Dockerfile Best Practices [w/Code]
Containerization provides a controlled deployment boundary for ROS 2 software by packaging application binaries, runtime libraries, ROS distributions, and configuration dependencies into reproducible images. In robotics, this reduces differences between developer workstations, CI environments, simulation systems, edge computers, and production robots while preserving a consistent software baseline across deployments.
A ROS 2 Dockerfile should begin from the smallest trustworthy base image that still satisfies the application requirements. The selected image should explicitly match the intended ROS 2 distribution, operating-system release, and processor architecture. Production images should avoid unnecessary development utilities because every additional package increases image size, build time, dependency complexity, and the potential security attack surface.
Reproducibility requires dependency installation to be deterministic rather than dependent on the changing state of external repositories. Package versions should be constrained where practical, repository configuration should be explicit, and rosdep should resolve package dependencies from a well-defined workspace. Docker base images can also be pinned by digest when strict traceability is required for validated robot software releases.
Docker layer organization strongly affects build efficiency. Instructions that change infrequently, such as operating-system and ROS dependency installation, should appear before frequently modified application source code. Docker can then reuse cached dependency layers while rebuilding only the application-specific layers. This becomes particularly important for large ROS 2 workspaces containing perception, navigation, control, and AI packages.
Multi-stage builds provide an effective method for separating compilation from execution. A builder stage may contain compilers, colcon, development headers, testing utilities, and source code, while the final runtime stage receives only the installed ROS 2 artifacts and required runtime libraries. This approach produces smaller production images and prevents build tools and temporary files from unnecessarily becoming part of deployed robot software.
ROS 2 workspaces should normally be built using colcon inside the container so that compilation occurs against exactly the libraries used by the target runtime environment. The source workspace can be copied into a dedicated build stage, dependencies resolved through rosdep, and packages compiled with an appropriate build type. The resulting install directory then becomes the primary software artifact transferred into the runtime image.
Dockerfile instructions should minimize unnecessary filesystem state. Package-manager metadata and temporary files should be removed in the same layer in which they are created, and broad COPY operations should be avoided. A carefully designed .dockerignore file should exclude build, install, log, version-control, dataset, model-cache, and temporary directories unless they are explicitly required by the containerized application.
ROS 2 containers require deliberate handling of environment initialization. The container entrypoint should source the ROS distribution setup script and, when applicable, the workspace install/setup script before executing the requested process. This ensures that package paths, shared libraries, interface definitions, and ROS-specific environment variables are consistently available without requiring every command to repeat environment initialization logic.
Configuration should generally remain separate from immutable application binaries. Robot-specific parameters, calibration data, DDS configuration, security artifacts, maps, and deployment settings may be supplied through mounted volumes, secrets, or environment-specific configuration mechanisms. This separation allows the same tested image to operate across multiple robots without rebuilding an image simply because a robot identifier or deployment parameter changes.
ROS 2 networking requires additional consideration because DDS discovery and data transport interact with container networking. Multicast discovery, UDP traffic, interface selection, ROS_DOMAIN_ID, DDS implementation settings, and host networking behavior must be evaluated for the actual deployment environment. A container that operates correctly on a developer workstation may behave differently across robot computers, VLANs, Wi-Fi networks, or multi-robot systems.
Hardware access should follow the principle of minimum privilege. Cameras, LiDAR interfaces, serial devices, CAN adapters, GPUs, and other accelerators should expose only the devices and permissions required by the ROS 2 process. Running every robot container with privileged mode may simplify initial integration, but it weakens isolation and makes security auditing difficult. Explicit device mappings and narrowly defined Linux capabilities are preferable.
GPU-based ROS 2 workloads introduce another compatibility boundary involving the host driver, container runtime, CUDA libraries, and application framework. Perception and Physical AI containers should therefore separate hardware-dependent runtime requirements from application dependencies wherever possible. Image variants may be maintained for CPU, NVIDIA GPU, or Jetson targets while preserving common ROS 2 interfaces and application-level behavior.
Containers should not automatically imply that every ROS 2 node requires an independent image. Node composition, process boundaries, failure isolation, hardware ownership, communication overhead, update frequency, and lifecycle requirements should determine container boundaries. Closely coupled nodes may reasonably share a container, while perception, navigation, hardware interfaces, AI inference, diagnostics, and fleet gateways may require different deployment units.
Production images should execute ROS 2 applications as non-root users whenever hardware and operating requirements permit. File permissions, mounted volumes, device groups, Linux capabilities, read-only filesystems, and writable runtime directories should be designed explicitly. Container security should complement SROS2 and DDS security rather than replace them, because process isolation and communication authentication address different layers of the robot security architecture.
Health management should distinguish container availability from ROS 2 application readiness. A running container does not guarantee that DDS discovery succeeded, sensors initialized correctly, lifecycle nodes reached the active state, or required dependencies became available. Deployment systems should therefore combine container-level health checks with ROS 2 lifecycle state, diagnostics, topic activity, service readiness, and hardware status where operationally appropriate.
Logging should be designed for persistent operation rather than interactive development. ROS 2 console output, application logs, diagnostics, and container runtime logs need controlled retention and aggregation so that storage does not grow indefinitely on edge computers. Important operational information should remain accessible after container restart, while transient build files and debugging artifacts should not become permanent parts of production images.
Image tagging must support traceability. Mutable labels such as latest are convenient during development but insufficient for controlled fleet deployment. Production releases should identify the application version, ROS distribution, architecture, and relevant build revision through immutable tags, digests, or associated metadata. This allows operators to determine exactly which software image is running on each robot and enables reliable rollback after deployment failures.
CI/CD pipelines should build ROS 2 images from version-controlled Dockerfiles and verify them before fleet deployment. Typical validation can include dependency resolution, colcon compilation, unit tests, integration tests, security scanning, image metadata inspection, and simulation-based startup tests. The same validated artifact should progress through staging and production whenever possible instead of rebuilding nominally identical software at each deployment stage.
Containerization is therefore most effective when treated as part of the ROS 2 deployment architecture rather than merely as a packaging technique. A well-designed Dockerfile establishes reproducible dependencies, controlled runtime privileges, efficient builds, hardware-aware execution, observable health, and traceable releases. These properties create the foundation for subsequent Docker Compose, CI/CD, OTA, monitoring, and large-scale fleet deployment mechanisms.
08.02 Docker Compose-Based Robot SW Stack Deployment [w/Code]
Docker Compose provides a declarative method for deploying a complete ROS 2 robot software stack as a coordinated collection of containers. Instead of manually starting perception, navigation, control, diagnostics, and application processes, a Compose file describes their images, runtime parameters, networks, volumes, devices, and dependencies. This converts robot deployment into a version-controlled system configuration that can be reproduced across development and production platforms.
A robot software stack should be divided into services according to operational responsibility rather than simply assigning one container to every ROS 2 node. Typical boundaries may separate hardware interfaces, perception pipelines, localization, navigation, AI inference, diagnostics, and external gateways. Closely coupled ROS 2 components can remain within one service when composition improves performance, while independently updated or failure-sensitive functions benefit from separate containers.
The Compose YAML file becomes the deployment manifest describing how these services operate together. Each service can reference a prebuilt image or Dockerfile, define its command and entrypoint, specify environment variables, attach volumes, expose hardware devices, and configure restart behavior. Keeping this configuration in version control allows the deployed topology to evolve together with ROS 2 packages while preserving a traceable history of system-level changes.
Image versions should be explicitly controlled because a multi-container robot depends on compatibility among several software components. Mutable tags such as latest can cause different robots to obtain different implementations even when they use the same Compose file. Versioned tags or immutable image digests provide stronger reproducibility and make it possible to reconstruct a fleet configuration, verify a release, and roll back individual services when deployment problems occur.
ROS 2 communication introduces special networking requirements because DDS discovery and transport must operate across service boundaries. Compose networks can isolate application groups, but multicast, UDP communication, interface selection, ROS_DOMAIN_ID, and DDS configuration must remain compatible with the intended topology. Host networking can simplify some robot deployments, although it reduces network isolation and should therefore be selected according to operational requirements rather than used automatically.
Multi-robot systems require deliberate separation of communication domains and naming spaces. Environment variables and external configuration can assign ROS_DOMAIN_ID values, robot namespaces, DDS profiles, and unique robot identifiers without modifying container images. This allows a common software image set to be deployed across many robots while Compose configuration provides robot-specific identity and communication behavior appropriate to each physical platform.
Hardware-facing services require explicit access to physical resources. Compose can map cameras, serial ports, CAN interfaces, LiDAR devices, USB peripherals, GPUs, and other accelerators into selected containers rather than exposing all host hardware to the complete software stack. Device mappings, group permissions, runtime configuration, and Linux capabilities should follow minimum-privilege principles so that a failure in one service does not unnecessarily compromise other robot functions.
Persistent and configurable data should be separated from immutable container images through volumes and bind mounts. Calibration files, maps, ROS 2 parameter files, DDS profiles, security material, AI models, and operational data can therefore survive container replacement or be changed independently from application binaries. Writable directories should be deliberately defined because unrestricted persistence makes deployment behavior difficult to reproduce and complicates rollback between software versions.
Service dependencies require more than specifying a simple startup order. A container may have started while its ROS 2 node is still initializing, waiting for hardware, discovering DDS participants, loading a map, or transitioning through lifecycle states. Consequently, dependency management should combine Compose startup relationships with health checks, ROS 2 lifecycle orchestration, application readiness, and retry mechanisms instead of assuming that a running container represents a ready subsystem.
Health checks provide a bridge between container supervision and robot application state. Basic checks may verify that a process remains alive, while stronger checks can confirm expected services, device availability, application endpoints, or ROS 2 subsystem readiness. Safety-critical decisions should not depend solely on Docker health status, however, because container supervision does not replace dedicated robot diagnostics, safety controllers, watchdogs, or lifecycle-based safe-state mechanisms.
Restart policies can improve availability when processes terminate unexpectedly, but automatic restart must be designed according to subsystem behavior. Restarting a visualization or telemetry service may be straightforward, whereas repeatedly restarting a hardware driver or motion-related component could create undesirable state transitions. Recovery policies should therefore coordinate restart limits, lifecycle initialization, hardware state, dependency recovery, and escalation to a controlled robot safe state when automatic recovery fails.
Environment-specific configuration should be separated from the primary Compose definition whenever practical. A common base configuration can describe the standard ROS 2 stack, while environment files or Compose overrides adapt it for development computers, simulation, laboratory robots, GPU platforms, and production systems. This prevents duplicated deployment manifests from gradually diverging and supports a consistent architecture across heterogeneous robot computing environments.
Resource management becomes increasingly important when perception and Physical AI workloads share an edge computer with navigation and control services. CPU allocation, memory limits, GPU access, shared-memory capacity, process priorities, and storage consumption should be considered explicitly. Containers provide useful isolation boundaries, but inappropriate resource competition can still increase ROS 2 callback latency, degrade sensor processing, or cause AI workloads to interfere with time-sensitive robot functions.
Logging should be coordinated at the stack level because each service produces independent container and ROS 2 logs. Log rotation, maximum file size, persistent diagnostic storage, timestamps, and centralized collection should be configured so that long-running robots do not exhaust local storage. Service names, container versions, robot identifiers, and ROS 2 node information should remain correlated, allowing operators to reconstruct events across perception, navigation, hardware, and application services.
Secrets and security credentials should not be embedded directly into Compose files or container images. SROS2 keystores, certificates, registry credentials, API tokens, and other sensitive material should be injected through controlled secret-management mechanisms or protected mounts. Services should normally execute as non-root users, receive only required devices and capabilities, and expose only necessary network interfaces, reducing the security impact of a compromised container.
Profiles and optional services can make one Compose architecture usable for several operating modes. Development deployments may include visualization, debugging, tracing, and simulation services, while production robots activate only operational components. GPU inference, alternative sensor drivers, or maintenance tools can similarly be enabled selectively. This approach preserves a common deployment model without forcing every robot or development environment to run an identical set of containers.
The Compose deployment process should integrate with CI/CD rather than depend on manually built images. Each service image can be compiled, tested, scanned, versioned, and published to a registry before the Compose manifest references the validated release. Integration testing can then launch the complete stack in simulation or a hardware test environment, verifying communication, dependencies, lifecycle behavior, configuration compatibility, and startup sequencing before deployment to physical robots.
Rollback should be treated as a system-level capability because robot behavior depends on interactions among multiple services. Updating only one image may introduce interface, message, parameter, or behavioral incompatibilities with the remaining stack. A release manifest should therefore record the compatible image versions and configuration revisions forming a validated robot software baseline, enabling the entire deployment or selected compatible components to return to a known operational state.
Docker Compose is particularly suitable for single-robot computers, development systems, laboratory platforms, and moderately complex edge deployments where a lightweight orchestration mechanism is sufficient. As fleet scale and distributed infrastructure requirements increase, Kubernetes or K3s may provide stronger scheduling and cluster-management capabilities. Compose nevertheless remains valuable as a transparent deployment layer for defining and validating the service architecture of an individual ROS 2 robot.
A well-designed Docker Compose architecture therefore transforms ROS 2 deployment from a collection of startup commands into a reproducible robot software system. Container boundaries, networking, hardware mapping, persistent configuration, health supervision, security, resource control, versioning, CI/CD, and rollback become explicit parts of the deployment model. This creates a stable foundation for OTA updates, monitoring, fleet operations, and subsequent large-scale ROS 2 deployment mechanisms.
08.03 Kubernetes / K3s-Based Robot Deployment [w/Code]
Kubernetes provides a declarative orchestration layer for deploying, supervising, and updating containerized ROS 2 software across distributed robot and edge-computing systems. While Docker Compose is effective for managing a stack on an individual robot computer, Kubernetes introduces cluster-level scheduling, service management, recovery, configuration, and rollout mechanisms. K3s provides a lightweight Kubernetes distribution that makes these capabilities practical for resource-constrained robotics environments.
A robot deployment can model each physical robot, edge computer, or on-premise server as a Kubernetes node participating in a common cluster. Workloads are represented as Pods containing one or more closely related containers, while higher-level controllers maintain their desired operating state. This architecture separates application deployment intent from individual host configuration and allows robot software to be managed through a consistent cluster control plane.
K3s is particularly attractive for robotics because it reduces the operational footprint associated with conventional Kubernetes installations. It packages essential cluster components into a compact distribution while retaining standard Kubernetes APIs and deployment concepts. This allows laboratories, AMR fleets, inspection robots, and edge AI platforms to adopt Kubernetes-compatible orchestration without requiring the infrastructure normally associated with large cloud data centers.
The fundamental deployment unit is the Pod, but ROS 2 software boundaries should not automatically correspond to individual ROS 2 nodes. Closely coupled components requiring shared memory or synchronized execution may belong in the same Pod, whereas perception, navigation, AI inference, diagnostics, gateways, and supporting services can be separated according to lifecycle, resource, update, and fault-isolation requirements. Container boundaries should therefore reflect system architecture rather than arbitrary process granularity.
Deployments and other Kubernetes workload controllers describe the desired number and version of application instances. For stateless supporting services, Kubernetes can automatically replace failed Pods and maintain the requested replica count. Robot-specific components require greater caution because hardware drivers, motion controllers, and localization processes are associated with physical devices and state. Automatic rescheduling must respect the relationship between software workloads and the robot hardware they control.
Node labels, selectors, affinity rules, and taints can constrain workloads to appropriate computing platforms. A perception Pod may require an NVIDIA GPU, a hardware-interface Pod may need access to a specific robot computer, and an AI inference workload may be assigned to a high-performance edge server. These scheduling mechanisms allow heterogeneous CPU, GPU, Jetson, and server resources to coexist within a common deployment architecture while maintaining explicit placement policies.
ROS 2 communication over DDS introduces networking considerations that differ from typical web-oriented Kubernetes applications. DDS discovery may depend on multicast, UDP transport, specific network interfaces, and direct connectivity among participants. Overlay networks and network address translation can interfere with expected discovery behavior. Robotics deployments must therefore validate host networking, DDS discovery servers, interface configuration, ROS_DOMAIN_ID, and the selected DDS implementation against the actual cluster topology.
Multi-robot deployments require isolation and identity management beyond basic Kubernetes service discovery. Robot namespaces, ROS_DOMAIN_ID values, DDS profiles, network policies, and robot identifiers can be injected through environment variables and configuration resources. A common container image can therefore support multiple physical robots while deployment manifests determine each robot\'s communication domain, namespace, hardware association, and operational role without rebuilding application software.
ConfigMaps provide a mechanism for separating non-sensitive configuration from container images. ROS 2 parameter files, DDS profiles, application settings, robot identifiers, and environment-specific values can be supplied dynamically to Pods. Sensitive information such as credentials, certificates, API tokens, and SROS2 security material should instead be handled through Kubernetes Secrets or an external secret-management system, with access restricted to only the workloads that require it.
Persistent storage requires special treatment because robot workloads generate maps, logs, rosbag or MCAP recordings, calibration information, AI data, and diagnostic histories. Persistent Volumes and storage classes can separate this data from ephemeral Pod filesystems. Edge deployments must nevertheless account for intermittent connectivity and local storage constraints, ensuring that critical operational data remains available even when robots temporarily lose access to centralized storage infrastructure.
Hardware access is one of the most important differences between conventional Kubernetes workloads and robotic applications. Cameras, LiDAR sensors, CAN interfaces, serial devices, motor controllers, and GPUs are physically attached to particular machines. Device plugins, host device mappings, runtime classes, privileged capabilities, and node placement policies may therefore be required. Access should remain narrowly scoped so that container orchestration does not unnecessarily weaken the host security boundary.
Resource requests and limits allow Kubernetes to describe CPU and memory requirements for each workload, while specialized mechanisms can allocate GPU resources. This is useful when perception, navigation, simulation, diagnostics, and Physical AI inference share heterogeneous computing infrastructure. Resource policies should be validated carefully, however, because excessive CPU contention, memory pressure, or GPU competition can increase ROS 2 latency and degrade time-sensitive robot behavior.
Real-time control should remain separated from assumptions made by general-purpose cluster orchestration. Kubernetes can supervise deployment and availability, but it does not transform ordinary container scheduling into deterministic real-time execution. High-frequency motor control, safety loops, and strict timing functions should remain within appropriately designed real-time processes, controllers, or embedded systems. Kubernetes should primarily orchestrate higher-level ROS 2 workloads around these deterministic control boundaries.
Health management can combine Kubernetes probes with ROS 2 application-level diagnostics. Liveness probes determine whether a workload should be restarted, readiness probes determine whether it is prepared to receive operational traffic, and startup probes accommodate components with long initialization periods. For ROS 2 systems, these mechanisms can be supplemented with lifecycle state, sensor availability, DDS connectivity, diagnostics, and subsystem readiness rather than relying solely on process existence.
Kubernetes self-healing is useful but must be constrained by physical-system semantics. Automatically restarting a telemetry service is fundamentally different from restarting a motor interface or relocating a hardware-dependent Pod to another node. Recovery policies should distinguish stateless services from robot-bound workloads and coordinate restart behavior with lifecycle management, watchdogs, hardware initialization, fault diagnostics, and safe-state transitions when automatic recovery is inappropriate.
Rolling updates allow new software versions to be introduced gradually instead of replacing an entire deployment simultaneously. This is valuable for infrastructure and stateless services, while physical robot software requires additional operational safeguards. Deployment strategies should consider robot mission state, charging status, maintenance windows, hardware compatibility, and validated software baselines. Updates should be stoppable and reversible when health indicators or integration tests reveal unexpected behavior.
Namespaces and role-based access control provide organizational and security boundaries within the cluster. Development, simulation, staging, and production workloads can be separated, while service accounts and RBAC policies restrict which components may modify deployments or access configuration. Network policies can further limit communication paths. These mechanisms complement SROS2 and DDS security, which continue to protect ROS 2 identities, permissions, and communications at the middleware layer.
Observability becomes essential as deployment expands from one robot to a distributed fleet. Kubernetes exposes workload state, restart events, resource utilization, and scheduling information, while ROS 2 contributes node diagnostics, lifecycle states, topic behavior, sensor status, and application logs. Combining infrastructure-level and robot-level telemetry allows operators to distinguish container failures, network problems, resource exhaustion, DDS issues, hardware faults, and application-level degradation.
CI/CD pipelines can generate container images and Kubernetes manifests as version-controlled deployment artifacts. Images should be compiled, tested, security-scanned, tagged, and published to a registry before being referenced by validated manifests. Simulation and hardware-in-the-loop testing can then exercise complete deployments before release. GitOps-style practices can additionally maintain the desired fleet configuration as auditable repository state rather than relying on manual cluster modifications.
K3s can support hierarchical robot infrastructure in which individual robots execute local workloads while edge servers or on-premise computers provide heavier perception, AI inference, data aggregation, and fleet services. Placement policies can determine which computations remain on the robot and which execute on more powerful nodes. This creates a flexible foundation for combining edge intelligence, on-premise AI, and multi-robot coordination within a common deployment framework.
Connectivity loss must be treated as a normal robotics operating condition rather than an exceptional cluster failure. A mobile robot may temporarily lose Wi-Fi, private 5G, or access to the Kubernetes control plane while continuing its mission. Essential navigation, control, perception, and safety functions should therefore remain locally executable. Cluster orchestration should enhance management and recovery without making basic robot autonomy dependent on continuous infrastructure connectivity.
Kubernetes and K3s consequently extend ROS 2 deployment from container management on individual computers toward coordinated orchestration across robots, edge nodes, and shared infrastructure. Their value comes from declarative configuration, controlled scheduling, resource management, health supervision, security boundaries, observable operations, and managed updates. When combined with ROS 2 lifecycle management and robot-aware safety design, they provide a scalable deployment foundation for increasingly distributed robotic systems.
08.04 ROS2 CI/CD Pipeline: GitHub Actions [w/Code]
Continuous Integration and Continuous Delivery establish a repeatable mechanism for validating and releasing ROS 2 software whenever source code changes. GitHub Actions can automate this workflow directly from a repository by reacting to pushes, pull requests, tags, or manually triggered events. For robotics teams, the pipeline becomes a quality gate that verifies software before it reaches simulation, laboratory hardware, edge computers, or production robots.
A ROS 2 CI/CD workflow is normally defined as version-controlled YAML under the repository's GitHub Actions configuration. The workflow specifies triggers, jobs, execution environments, permissions, environment variables, and individual validation steps. Keeping pipeline definitions beside application source code allows build and test procedures to evolve with the software while making changes reviewable through the same pull-request process used for ROS 2 packages.
Pipeline triggers should reflect the development and release strategy. Pull requests can initiate fast compilation, linting, and unit tests before code is merged, while commits to protected branches can execute broader integration tests and container builds. Version tags or approved release events can initiate packaging and deployment stages. Separating validation from release triggers prevents ordinary development activity from unintentionally publishing or deploying production artifacts.
The build environment must reproduce the intended ROS 2 distribution, operating-system version, and architecture as closely as practical. GitHub-hosted runners can provide standardized environments for common builds, while containers can establish a more controlled ROS 2 toolchain. Self-hosted runners become useful when workflows require specialized GPUs, ARM or Jetson platforms, private networks, proprietary dependencies, or direct access to robotics test infrastructure.
Dependency installation should be automated and deterministic. A workflow can create the ROS 2 workspace, import required repositories, update rosdep information, resolve system dependencies, and install packages before invoking colcon. Dependency versions should be constrained where reproducibility matters because uncontrolled external updates can cause a previously successful pipeline to fail even when application source code has not changed.
The primary compilation stage should execute colcon build with options appropriate to the target workspace and development policy. Build failures, compiler warnings, missing dependencies, interface-generation problems, and package-ordering issues can therefore be detected before integration. Clean builds remain important even when caching is enabled because the pipeline must demonstrate that the repository can produce valid ROS 2 artifacts without relying on undeclared state from a developer workstation.
Caching can substantially reduce CI execution time for large ROS 2 projects, but cached data should be selected carefully. Package downloads, compiler caches, or stable dependency layers may provide useful acceleration, whereas blindly caching build and install directories can conceal dependency problems. Cache keys should incorporate relevant operating-system, ROS distribution, compiler, dependency, and source information so that incompatible artifacts are not reused across different build configurations.
Static analysis and formatting checks provide an early quality barrier before computationally expensive tests execute. ROS 2 packages can integrate linters, compiler diagnostics, formatting rules, copyright checks, and language-specific analysis through the ament ecosystem or dedicated tools. Running these checks consistently in CI reduces differences between developers and prevents style, maintainability, and common implementation defects from accumulating unnoticed.
Unit testing should execute automatically after successful compilation. C++ packages can use gtest through ament_cmake_gtest, while Python packages can use appropriate testing frameworks integrated with the ROS 2 build system. Test results generated by colcon should be collected and reported even when failures occur. This gives developers immediate information about which package or component failed rather than presenting the pipeline only as a binary success or failure indicator.
Integration testing verifies behavior that cannot be established from isolated packages. A workflow may launch multiple ROS 2 nodes and validate topic communication, services, actions, parameters, lifecycle transitions, or expected message flows. Test environments can also start Docker Compose stacks or temporary container networks. Timeouts and cleanup procedures are essential because distributed ROS 2 tests can otherwise leave processes running or cause CI jobs to wait indefinitely.
Simulation provides an important intermediate validation layer between software-only testing and physical robot deployment. Headless Gazebo, other simulators, or application-specific test environments can execute navigation, perception, control, and mission scenarios without requiring physical hardware for every commit. Simulation does not replace hardware testing, but it allows major integration failures and behavioral regressions to be detected earlier and at significantly greater execution frequency.
A matrix build can test the same ROS 2 source against multiple distributions, operating systems, middleware implementations, or build configurations. This is particularly useful for reusable libraries and robot platforms that must support heterogeneous deployment targets. Matrix coverage should remain intentional because every additional combination increases CI cost and maintenance effort. Critical production combinations deserve stronger coverage than experimental environments.
Container image generation can follow successful ROS 2 build and test stages. The pipeline can build Docker images using validated Dockerfiles, assign immutable version identifiers, and publish them to an authorized container registry. Multi-stage Docker builds help ensure that development dependencies remain outside production images. Image metadata should connect the deployed artifact to the source commit, release version, architecture, and ROS distribution from which it was produced.
Security checks should be incorporated before artifacts become eligible for deployment. Workflows can scan source dependencies, container images, credentials exposure, and known vulnerabilities while enforcing repository permissions and protected release processes. GitHub Actions permissions should follow least-privilege principles, and long-lived credentials should not be embedded in workflow files. Repository secrets or federated identity mechanisms should provide controlled access to registries and deployment infrastructure.
Artifacts provide traceability between CI execution and robot deployment. Test reports, coverage information, compiled packages, container metadata, configuration manifests, and selected diagnostic outputs can be retained according to project policy. A release should be reconstructable from its source revision and associated artifacts. This becomes especially important when diagnosing a robot failure months after deployment or demonstrating which software baseline was used during validation.
Self-hosted runners extend CI from cloud-based software validation into physical robotics infrastructure. A laboratory runner can access GPUs, Jetson devices, CAN interfaces, sensors, or complete robots that are unavailable to standard hosted environments. Such runners should be isolated and carefully secured because executing repository workflows on machines connected to physical hardware introduces substantially greater operational risk than ordinary software compilation.
Hardware-in-the-loop testing should be positioned after lower-cost validation stages have succeeded. The pipeline can deploy a candidate build to a controlled robot or test bench, initialize required ROS 2 subsystems, execute predefined scenarios, collect diagnostics, and evaluate expected outcomes. Hardware access should be serialized when necessary so that concurrent jobs do not attempt to control the same device, and failures should return the test system to a known safe state.
Deployment should promote previously validated artifacts rather than rebuild source code independently for each environment. A candidate container image can move from development validation to simulation, staging, laboratory robots, and eventually production while retaining the same immutable digest. Environment-specific parameters remain external to the image. This approach reduces uncertainty about whether the software tested in CI is actually identical to the software executing on the robot.
Production deployment requires stronger controls than ordinary CI execution. Protected environments, approval gates, release branches, signed artifacts, maintenance windows, and staged rollout policies can prevent unreviewed changes from reaching operational robots. Deployment workflows should record which software version was delivered to each target and preserve a rollback path to a validated baseline when post-deployment health checks reveal unexpected behavior.
For robot fleets, CI/CD should coordinate software versioning with configuration compatibility. A release may include ROS 2 packages, container images, parameter files, DDS profiles, lifecycle settings, and deployment manifests that have been validated together. Treating these elements as a coherent release baseline prevents partial updates from introducing incompatible interfaces or behavior across perception, navigation, control, AI inference, and fleet-management services.
Observability closes the loop between deployment and subsequent development. CI/CD can verify immediate deployment health, while operational monitoring provides evidence about crashes, resource consumption, ROS 2 diagnostics, communication failures, and application performance after release. These observations can become inputs for regression tests and future pipeline improvements, turning field experience into repeatable validation rather than leaving operational knowledge outside the development process.
A mature GitHub Actions pipeline therefore connects source control, reproducible builds, static analysis, automated testing, simulation, container generation, security validation, hardware testing, release management, and controlled deployment. For ROS 2 robotics, CI/CD is not simply an automation convenience; it becomes part of the software assurance architecture that establishes traceability from a source commit to the exact validated software operating on a physical robot.
08.05 ROS2 Package Distribution: bloom / ros.infrastructure
ROS 2 package distribution transforms source repositories into installable software that users can obtain through standard operating-system package managers. Rather than requiring every robot developer to clone repositories and compile workspaces manually, the ROS infrastructure provides a release process that generates binary packages for supported ROS distributions and platforms. This creates a consistent path from package development to reproducible installation and deployment across robotics systems.
A ROS 2 package normally begins in a source repository containing package.xml, build-system configuration, source code, interfaces, tests, and documentation. The package manifest defines metadata such as package name, version, maintainers, licenses, build dependencies, execution dependencies, and export information. Accurate metadata is essential because the release infrastructure uses this information to resolve dependencies and construct packages for downstream users.
Source development and package release are intentionally separated. Developers can continue normal development in the primary repository while a specific source revision is selected for an official release. This separation prevents the distribution process from depending directly on a changing development branch and provides a stable relationship between a released version, its source tag, generated packaging information, and resulting binary artifacts.
Bloom is the primary ROS release automation tool used to convert ROS package metadata into distribution-specific packaging information. It reads package manifests and repository configuration, processes dependency information, and generates the files required by the target packaging ecosystem. For Ubuntu-based ROS deployments, this typically means generating Debian packaging metadata that can later be built into binary .deb packages by the ROS build infrastructure.
The release process commonly uses a dedicated release repository generated and maintained through bloom. This repository is different from the normal source repository because it contains generated packaging branches and metadata corresponding to released versions. Keeping generated release information separate prevents packaging-specific files from cluttering application development while preserving a traceable history of how each ROS package version was prepared for distribution.
The ROS distribution index, commonly represented through rosdistro metadata, connects package repositories with a particular ROS distribution. Repository entries describe source, release, and documentation information together with version and status data. When maintainers prepare a package for a ROS distribution, the corresponding metadata must eventually be updated so that the central infrastructure recognizes the new release and can schedule the required build operations.
Dependency resolution is a central part of this process. Package dependencies declared in package.xml are mapped to platform-specific system packages through rosdep rules and distribution metadata. Bloom uses this information when producing packaging configuration, while build systems later install the required dependencies before compilation. Incorrect or incomplete dependency declarations can therefore cause failures even when the package builds successfully inside the maintainer\'s existing development workspace.
Package versioning should follow a disciplined release policy because versions become part of the public distribution interface. A release identifies a specific implementation that downstream packages and robot deployments may depend upon. Changing source code without changing the corresponding package version can undermine traceability, while uncontrolled version changes can complicate dependency management. Release tags and package.xml versions should therefore remain synchronized.
When a maintainer initiates a bloom release, the tool processes the selected source version, updates the release repository, and prepares changes required for the ROS distribution metadata. The resulting update is typically submitted for review before becoming part of the distribution index. This review step separates package-maintainer actions from modifications to the shared ROS distribution and provides an opportunity to detect metadata or release problems before automated builds begin.
After release metadata is accepted, ROS build infrastructure can generate platform-specific binary packages. Build jobs obtain the released source and packaging information, install dependencies in controlled environments, compile the software, execute applicable packaging steps, and produce repository artifacts. Building packages centrally reduces dependence on individual developer machines and helps ensure that binaries distributed to users correspond to known source and dependency configurations.
Binary repositories provide the user-facing endpoint of the ROS distribut...
![Robot Middleware and ROS2 — Ch.08: ROS2 Deployment and DevOps [EN]](https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F9z4fmoxmqqngiro7boaa.png)



