[BUG] likwid-mpirun : Unable to read the version numbers of MPI

[nadir-ogd]

Hi,

I’m trying to use likwid-mpirun in my SLURM job script, which includes the following command:
likwid-mpirun -np 64 -g ENERGY -o ./results/energy.csv -m --debug ./executable

However, I keep encountering the following error:
likwid/5.4.1/bin/likwid-mpirun:1866: table index is nil
stack traceback:
likwid/5.4.1/bin/likwid-mpirun:1866: in local 'parseMarkerOutputFile'
likwid/5.4.1/bin/likwid-mpirun:2655: in main chunk
[C]: in ?

I am using OpenMPI as my MPI implementation, as indicated in the debug output:
DEBUG: Using MPI implementation openmpi

For reference, here is the output of mpiexec --version:
mpiexec (OpenRTE) 4.1.1

After searching for similar issues, I found a suggestion to set a thread skip mask using -s 0x0, so I tried:
likwid-mpirun -np 64 -g ENERGY -o ./results/energy.csv -m --debug -s 0x0 ./executable

Unfortunately, this did not resolve the issue.

Do you have any insights into what might be causing this error ? Is there a known workaround or configuration adjustment that I should try?

Thanks in advance for your help!

Best,
Nadir

[TomTheBear]

Does your application contain markers? Have you built it with -DLIKWID_PERFMON?

Redirecting the output to a file is currently not supported for likwid-mpirun. The option exists but it will print a warning

Please provide the output of a run with -d/--debug. Is it LIKWID 5.4.1 or some older version?

The skip mask is probably not the reason for your issues.

[nadir-ogd]

Yes, my application includes markers, and it is built with -DLIKWID_PERFMON 👍.

Here is the output file generated with the debug option: likwid-mpirun -np 64 -g ENERGY --debug -m ./executable

debug.out.txt
Note: I have removed the full absolute paths for confidentiality.

The error persists in this case:

likwid/5.4.1/bin/likwid-mpirun:1866: table index is nil
stack traceback:
likwid/5.4.1/bin/likwid-mpirun:1866: in local 'parseMarkerOutputFile'
likwid/5.4.1/bin/likwid-mpirun:2655: in main chunk
[C]: in ?

However, I noticed that if I remove the -m option and execute the following command:
likwid-mpirun -np 64 -g ENERGY --debug ./executable

there is no error. I get the same debug file, but at the end, I see LIKWID's output, though the energy value is 0, like this:

[TomTheBear]

The debug logs show one issue that might cause the error. likwid-mpirun does not tell to measure the ENERGY group itself to the MPI processes but it splits up the eventset based on the topology of the node itself runs on. Only one MPI process per socket should read the L3 energy counters and only one SMT thread (if active) of a core should read the core energy counters. This fails for your system. No energy event is measured by any MPI process:

DEBUG: Process [0-63] measures with event set: ACTUAL_CPU_CLOCK:FIXC1,MAX_CPU_CLOCK:FIXC2,RETIRED_INSTRUCTIONS:PMC0,CPU_CLOCKS_UNHALTED:PMC1

A guess would be that likwid-mpirun tries to evaluate the energy counters but the MarkerAPI tables returned by the processes are too short. The responsible function to split the eventset is splitUncoreEvents().

You can add another condition for the energy counters but that solves it only partly. The core energy counters are not covered by this. There is currently no code for "per CPU core" in likwid-mpirun (I know, there is core in splitUncoreEvents() but it is really bad naming for hwthread). So more changes are required. I take a look.

diff --git a/src/applications/likwid-mpirun.lua b/src/applications/likwid-mpirun.lua
index 5a651d9d..4dcd7b39 100644
--- a/src/applications/likwid-mpirun.lua
+++ b/src/applications/likwid-mpirun.lua
@@ -1241,6 +1241,8 @@ local function splitUncoreEvents(groupdata)
                         table.insert(llc, event)
                     elseif e["Counter"]:match("^UPMC%d") then
                         table.insert(socket, event)
+                    elseif e["Counter"]:match("^PWR2") then
+                        table.insert(llc, event)
                     elseif e["Counter"]:match("^DFC%d") then
                         if cpuinfo["family"] == 0x17 then
                             table.insert(numa, event)

[nadir-ogd]

Hello,

Thank you for this first solution.

I’ve added the condition as requested, and I can now see RAPL_L3_energy:PWR2 appearing in some processes (with IDs of the form 8*k, k ≥ 0) :

However, as supposed, the error is still present and not fully resolved — further modifications are likely needed.
Looking forward to your feedback on the remaining modifications.

Thanks again!
Nadir

debug.txt

[TomTheBear]

Does it report energy data for the L3 events now in the non-MarkerAPI case?

I worked on it already but the situation got complicated the deeper you go. The general changes to support per-core and per-L3 events in likwid-mpirun is not the big issue but that the availability of energy events change with the backend. With accessdaemon and direct mode, the per-core and per-L3 events exist but with perf_event, you only get per-socket energy data. This means a change of the event list and potential performance groups based on the backend. While the event list change can be done, selecting a different performance group or changing its eventset and metrics is not available at the moment. The user could select different ones like ENERGY and PERF_ENERGY but that's not handy and most users do not know which backend the administrators have selected.

[nadir-ogd]

i can see eenrgy data for L3 events without the MarkerAPI option :

In my case, the LIKWID installed on the cluster probably uses accessdaemon as backend, so I'm not sure why per-core energy data isn't available.

also if I understood correctly, does that mean it's currently not possible to access energy data through the MarkerAPI?

[TomTheBear]

It is available with accessdaemon, just not through likwid-mpirun at the moment.

And you didn't understand corretly. You can access the energy data through the MarkerAPI but again not through likwid-mpirun at the moment.

I will do a fix for likwid-mpirun with accessdaemon mode. For the problematic perf_event case, I need to find a different solution.

[TomTheBear]

Please test the patch provided in the above commit. It should enable the proper scheduling of the AMD RAPL events for all AMD Zen versions LIKWID supports in accessdaemon mode.

[nadir-ogd]

Hello,

Thank you for your proposed solution. I was waiting for the cluster administrator to apply it.

Unfortunately, I'm still unable to run likwid-mpirun successfully, even after the modifications. It's true that I'm no longer getting the following error:

likwid/5.4.1/bin/likwid-mpirun:1866: table index is nil
stack traceback:
likwid/5.4.1/bin/likwid-mpirun:1866: in local 'parseMarkerOutputFile'
likwid/5.4.1/bin/likwid-mpirun:2655: in main chunk
[C]: in ?

And I can now see all the PER_CORE_ENERGY entries in the debug output file debug.txt

However, I'm still encountering multiple MPI-related errors in my job.err file job.txt

I'm using the --mpi slurm option because it forces SLURM integration, and when I try using --mpi openmpi, it suggests switching back to SLURM — I'm not sure why this is happening.

Also, I’m not seeing any results tables generated in either mode — with or without the Marker API.

Do you have any idea what might be causing these issues?

btw im using this script to run : script.txt

Thanks again for your support,