Introduction
Imagine you have written a program that performs a scientific calculation. You run it on your laptop, and everything works perfectly.
Now you move the same program to an HPC cluster. Instead of running on one CPU core, it runs across 32 CPU cores, multiple compute nodes and perhaps several GPUs.
Suddenly, the program crashes. Or worse, it completes successfully but produces an incorrect result.
Where would you start looking?
Would you check the CPU? The GPU? The memory? Or perhaps one of the MPI processes?
This is one of the challenges of High Performance Computing. As applications become more parallel, understanding and fixing bugs becomes more complicated.
In a traditional application, you might have one process executing instructions from beginning to end. In an HPC application, hundreds or thousands of processes and threads can execute different parts of the same calculation simultaneously.
A problem in just one process or GPU thread can affect the entire application.
This is where debugging tools such as Linaro DDT become useful. They allow developers to inspect what is happening inside an application, understand where something went wrong and investigate the reason behind the problem.
Let's understand the concepts step by step.
1. First, understand the different types of bugs
Before learning about debugging tools, we need to understand what kind of problems we are trying to solve.
Not every application problem looks like a crash. Sometimes an application runs perfectly from a technical perspective but produces the wrong answer.
There are four common categories of bugs in parallel applications.
Crashes
A crash happens when one or more processes in an application terminate unexpectedly.
For example, imagine an MPI application running across four processes:
Rank 0, Rank 1, Rank 2 and Rank 3.
If Rank 2 encounters an invalid memory access and terminates, the application may fail even though the other three processes are still running.
Think of four people working together on a project. If one person suddenly leaves without completing their task, the entire project might stop.
Crashes are generally among the easiest bugs to identify because the application visibly terminates.
Hangs
A hang happens when an application stops making useful progress.
There are two important concepts here.
Deadlock
Imagine two people waiting for each other.
Person A says, "I will continue when Person B gives me something."
Person B says exactly the same thing.
Neither person moves forward.
This is similar to a deadlock in an MPI application.
For example, two MPI processes might be waiting for messages from each other, but neither sends the message required by the other process.
Both remain stuck.
Livelock
A livelock is slightly different.
Imagine two people trying to pass through a narrow doorway. Both move left to avoid each other. Then both move right. They keep moving, but neither gets through.
They are doing something, but they are not making useful progress.
In an application, processes or threads may continue executing instructions but fail to make overall progress towards completing the calculation.
Race conditions
A race condition happens when multiple threads access shared data at the same time and the final result depends on the order in which their operations happen.
Let's take a very simple example.
Imagine two threads are updating the same variable.
The variable initially contains:
counter = 10
Both threads want to increase it by one.
You might expect the final value to be 12.
However, consider what happens:
- Thread 1 reads the value 10.
- Thread 2 also reads the value 10.
- Thread 1 increases its copy to 11.
- Thread 2 increases its copy to 11.
- Both write 11 back to memory.
The final value becomes 11 instead of 12.
Both threads executed their instructions, but the result is incorrect because their operations overlapped in an unsafe way.
This is a race condition.
Race conditions are particularly important in parallel programming because multiple threads can access shared data simultaneously.
Incorrect results
This is perhaps the most dangerous category.
An application starts successfully, performs its calculations and finishes without reporting an error.
However, the answer is wrong.
For example, imagine a scientific simulation calculating the temperature distribution across a material.
The application completes successfully, but some temperature values are incorrect because of a calculation error or incorrect memory access.
There might be no crash and no obvious warning.
This is why debugging is not just about finding out why an application stopped. It is also about understanding whether the application calculated the correct answer.
2. What is Linaro DDT?
Now that we understand the types of bugs, let's look at a tool designed to investigate them.
Linaro DDT is a graphical debugger designed for parallel applications and HPC environments.
It supports debugging applications written in languages such as C, C++, Fortran and Python.
It is part of the Linaro Forge toolset.
One important thing to understand is that DDT uses GDB underneath.
You might already be familiar with GDB if you have worked with Linux application debugging.
GDB allows developers to stop a program at a particular point, inspect variables, examine the call stack and execute instructions step by step.
DDT builds on this debugging foundation and provides additional capabilities for parallel and GPU applications.
Think about the difference:
With traditional GDB, you might investigate what is happening inside one process.
With DDT, you can investigate what is happening across multiple processes and threads at the same time.
For example, if you have an MPI application running across 64 processes, DDT can help you compare their execution states and identify which process is behaving differently.
This becomes extremely useful when debugging applications at HPC scale.
3. Why is debugging an MPI application different?
Let's understand this with a simple example.
Imagine you have an application that processes a large scientific dataset.
Instead of processing everything using one CPU core, you divide the work between four MPI processes.
Each process receives a portion of the dataset.
For example:
Rank 0 processes the first section.
Rank 1 processes the second section.
Rank 2 processes the third section.
Rank 3 processes the fourth section.
All four processes work together to complete the calculation.
Now imagine the application produces an incorrect result.
Which process caused the problem?
Perhaps Rank 0 is working correctly, Rank 1 is also correct, but Rank 2 has calculated an incorrect value.
Rank 3 might simply be receiving incorrect information from Rank 2.
If you only inspect the final result, identifying the original problem can be difficult.
This is where parallel debugging becomes useful.
DDT provides features that help developers:
Compare process execution: Examine what different processes are doing and identify differences between them.
Inspect parallel stacks: View the functions currently being executed by different processes and threads.
Identify MPI ranks: Investigate which particular MPI rank is experiencing a problem.
Compare variable values: Check whether a variable has the expected value across different processes.
Think of it as looking at several workers performing different tasks simultaneously rather than watching only one worker.
This ability to compare processes is one of the most important concepts in parallel debugging.
4. Understanding the DDT interface
The graphical interface of DDT contains several different windows and panels.
At first, the interface might look complicated. However, each panel has a specific purpose.
Here are some important areas.
Source Code View
This displays the actual application source code.
You can see the instructions being executed and identify the current execution point.
Variables
This panel displays the values of variables at the current execution point.
For example, if your application has variables called temperature, velocity and pressure, you can inspect their current values.
Evaluate Window
This allows you to evaluate expressions while debugging.
For example, instead of simply looking at the value of x, you might want to calculate x + y to understand what the program is producing.
Parallel Stack
This is particularly important in HPC.
It allows you to inspect the call stacks of multiple processes and threads.
Process Controls
These controls allow you to start, stop, pause and step through application execution.
Understanding these panels makes it easier to investigate problems without constantly switching between different debugging tools.
5. What is a call stack and why does it matter?
The call stack is one of the most important concepts in application debugging.
Let's understand it with a simple example.
Imagine your program contains three functions:
main()
calculate()
update()
The program starts inside main().
The main() function calls calculate().
Then calculate() calls update().
If the application encounters an error inside update(), the call stack helps you understand how the program reached that point.
It might look like this:
main() → calculate() → update()
This gives you the execution path that led to the problem.
Now imagine an MPI application with 64 processes.
Each process might be executing a different function or might be waiting at a different point.
Inspecting the call stack of every process individually would be time consuming.
DDT provides a Parallel Stack View that helps developers examine these execution states together.
It can also display GPU threads in the parallel stack view.
This is particularly useful when debugging applications that combine CPU execution with GPU computation.
6. Understanding GPU debugging
So far, we have mainly discussed CPU and MPI debugging.
But modern HPC applications increasingly use GPUs to accelerate scientific calculations.
GPU debugging introduces another level of complexity.
Let's first understand how GPU execution works.
A GPU contains many processing units that can execute a large number of threads simultaneously.
For example, imagine a scientific application that needs to perform the same mathematical operation on thousands of data elements.
Instead of processing each element sequentially on the CPU, the application can distribute the work across GPU threads.
This is extremely useful for computationally intensive workloads.
However, what happens if one of those GPU threads accesses an invalid memory location?
The application might encounter an illegal memory access error.
Finding the exact thread responsible for the problem can be difficult.
This is where GPU debugging tools become important.
Debugging AMD GPUs with DDT
DDT supports GPU debugging on AMD hardware using ROCm and ROCgdb.
It allows developers to debug CPU and GPU execution together.
This is important because GPU applications do not operate independently of the CPU.
The CPU typically manages the application and launches GPU kernels, while the GPU performs the computational work.
If something goes wrong, developers may need to inspect both sides.
Some important GPU debugging features include:
Kernel Progress View
This helps you understand the progress of GPU kernels.
A kernel is a function that executes on the GPU.
For example, a matrix multiplication application might launch a GPU kernel to calculate different portions of the result matrix.
GPU Thread Selector
This allows developers to select particular GPU threads for inspection.
GPU Device Pane
This provides information about the GPU device involved in the debugging session.
GPU Thread in Parallel Stack View
This helps developers inspect GPU thread execution alongside other execution information.
There is also an important detail about AMD GPU debugging.
DDT can step through a complete GPU wavefront of 64 threads at a time.
A wavefront is a group of GPU threads that execute together on AMD GPU hardware.
This is different from stepping through a single CPU instruction or thread.
For AMD GPU debugging, the application must be compiled with debugging information using -g, and ROCgdb must be available in the environment.
For the AMD MI300A GPU architecture used in the Hunter examples, the architecture can be specified using:
--offload-arch=gfx942
The important takeaway is that debugging GPU applications requires awareness of how GPU threads execute, how kernels are launched and how memory is accessed.
7. One of the most important GPU bugs: Illegal memory access
Let's understand a practical GPU debugging problem.
Imagine you have a three dimensional scientific dataset.
For simplicity, think about a box containing thousands of small cells.
Each cell has neighbouring cells around it.
A GPU application processes these cells and performs calculations using their neighbouring values.
However, the application needs to calculate the correct memory location for each neighbour.
What happens if it calculates the wrong memory address?
It might attempt to access memory outside the allocated array.
This is called an out of bounds memory access.
A simple example
Imagine you have an array containing ten elements.
The valid indexes are:
0, 1, 2, 3, 4, 5, 6, 7, 8, 9
Now imagine your program attempts to access:
array[10]
That location is outside the array.
The program is trying to access memory that does not belong to the allocated array.
This can result in an illegal memory access, a crash or incorrect behaviour.
A practical example: Marching Tetrahedra
Consider a GPU implementation of an algorithm called Marching Tetrahedra.
This algorithm is used to extract an approximate three dimensional surface from a scalar field.
A scalar field is simply a dataset in which every point has a numerical value.
For example, imagine a three dimensional space in which every point represents a temperature or pressure measurement.
To perform its calculations, the algorithm needs to access the eight neighbouring points around each position.
The developers converted the three dimensional dataset into a one dimensional array to make it easier to process on the GPU.
However, calculating the memory positions of neighbouring points became more complicated.
The problem appeared when GPU threads processed points close to the boundaries of the dataset.
For example, when a thread processed a point at the edge of the three dimensional domain, its neighbour calculations could produce an invalid memory address.
This resulted in illegal memory access errors.
The underlying issue was incorrect boundary checking.
Using Linaro DDT, developers could investigate the problematic GPU threads and identify the out of bounds memory accesses.
This is a very important lesson for anyone learning GPU programming.
Your application might work correctly for most data elements but fail when processing elements at the edges of an array.
Always think about array boundaries and whether every memory access is valid.
8. Understanding the Multi Dimensional Array Viewer
Another useful feature of DDT is the Multi Dimensional Array Viewer.
Let's understand why this is useful.
Imagine your application contains a two dimensional array representing temperature values across a surface.
The array might look like this:
10 12 14 16 11 13 15 17 12 14 16 18 13 15 17 19 You can understand the data by looking at these numbers.
But imagine the array contains thousands or millions of elements.
Looking at every value individually becomes difficult.
The Multi Dimensional Array Viewer allows developers to inspect arrays visually.
It supports viewing arrays on a single process or distributed across multiple MPI ranks.
You can also specify which portions of an array you want to inspect.
For example, you might want to view only rows 0 to 10 and columns 0 to 10.
The viewer also supports filtering.
Imagine you only want to see elements with values greater than zero.
You can use the condition:
$value > 0
The tool can also visualise array data as a three dimensional representation.
Why is this useful?
Because sometimes a numerical problem becomes much easier to understand when you can actually see the data rather than simply reading individual values.
For scientific applications working with matrices, grids and multidimensional datasets, this can be particularly helpful.
9. What is offline debugging?
Until now, we have discussed debugging through a graphical interface.
But imagine you have an application that runs for several hours on a large HPC cluster.
You cannot always sit in front of the debugger and watch every execution step.
This is where offline debugging becomes useful.
DDT can run in a non interactive mode.
Instead of manually opening the debugger and inspecting the application, you can execute the application under DDT and generate a debugging report.
For example:
ddt --offline --output=report.html mpiexec -n 4 ./simple Let's understand the command.
ddt launches the DDT debugger.
--offline enables non interactive debugging.
--output=report.html specifies the name of the generated report.
mpiexec -n 4 launches the application using four MPI processes.
./simple is the application being debugged.
You can also specify a particular source code location where you want the debugger to stop.
For example:
ddt --offline -o offline-debugging.html \ --break-at=simple.c:43 \ --evaluate="recvbuf; sendbuf" \ mpiexec -n 4 ./simple Here, the debugger is instructed to stop at line 43 of simple.c and evaluate the variables recvbuf and sendbuf.
The resulting report can help developers understand what happened during execution.
The --mem-debug option can also be used to enable memory debugging and memory leak detection.
This approach is particularly useful for long running applications, large scale debugging and automated testing or continuous integration workflows.
10. How does debugging work on an actual HPC cluster?
Debugging an HPC application is not necessarily something you do directly on your laptop.
The application might be running on remote compute nodes, while you use a graphical interface on your local machine.
Consider a remote debugging workflow using Linaro DDT.
First, the Linaro Forge remote client is installed on the developer's laptop.
The developer configures a remote connection to the HPC system by providing the login details and the Forge installation directory on the cluster.
After authentication, including the required one time authentication code, the debugging job is launched on the HPC system.
The job starts DDT in connect mode and waits for the remote client.
The developer then accepts the Reverse Connect request from the local client.
This establishes the debugging session.
The important concept is that the application executes on the HPC system, while the developer interacts with it through the graphical debugging client.
This is particularly useful when working with large HPC systems where applications need access to specific compute nodes and GPU resources.
What about Slurm?
In an HPC environment managed by Slurm, the same general principle applies.
The debugging application must run within an appropriate compute allocation with the required CPU, memory and GPU resources.
For example, a user might request an interactive allocation or submit a batch job, depending on the cluster configuration and debugging workflow.
The exact commands depend on how the HPC system and debugging software are configured.
The important concept is that debugging jobs must run within the HPC cluster's allocated resources and execution environment.
11. What should beginners take away from all this?
If you are new to HPC, you do not need to remember every DDT feature or command immediately.
Instead, focus on understanding the concepts.
First, debugging parallel applications is different from debugging ordinary serial applications because multiple processes and threads execute simultaneously.
Second, MPI debugging requires you to understand what individual ranks are doing and whether they are communicating correctly.
Third, GPU debugging introduces additional complexity because thousands of GPU threads may execute different parts of a calculation.
Fourth, memory errors are particularly important. An incorrect array index or boundary condition can cause an application to access memory outside its allocated region.
Finally, debugging is not just about finding where an application crashes. It is about understanding why the application behaves incorrectly.
Tools such as Linaro DDT make this investigation easier by providing visibility into processes, threads, variables, call stacks, arrays and GPU execution.
Conclusion
As HPC applications become more complex, debugging becomes an essential part of scientific computing.
An application that works correctly on a single CPU core might behave differently when distributed across multiple MPI processes or executed on GPUs.
Understanding how to inspect processes, investigate memory access, compare variables and analyse execution states is therefore an important skill for HPC developers and engineers.
The biggest lesson is simple: Do not just look at the error message. Understand what the application was doing when the error occurred.
That is the foundation of effective debugging in HPC.
If you are learning parallel programming, start by understanding serial debugging with GDB, then move towards MPI debugging and finally GPU debugging.
Once you understand these fundamentals, tools like Linaro DDT become much easier to learn and use.
