It will be impossible to root cause this issue without the source code for the apps you are running.
Can you point to where I can download and instructions to build the source ?

Hi Tom.  Yes.  Einstein@Home has their source code here:

https://einsteinathome.org/application-source-code-and-license

They run multiple "sub-apps", so it's not just one code base, but that page is where you start.  I'm sorry for having to bring this up.  I've run BOINC for 25 years, now, and in the past, AMD has worked this out within the first year of new architectures being released.  RDNA3 has been an exception.  Fedora is unsupported, and I understand that, but past experience tells me that support usually comes around in a reasonable time.  I was told by ROCm devs on GH that they believed Fedora wasn't running the same OSS code as Ubuntu.

Good news, though. ROCm 6.2 is working every well for BRP7 Meerkat work units!  This is a big improvement.  However, Gravitational Wave (GW) O3AS work units are not working, as I reported originally in this bug.  This is a regression, as O3AS was working fine on Fedora before ROCm 6.2 at the same time Meerkat was crashing.  Strange change.

Let me know what else I can do to help.  While I can't jump on issues, immediately, I put a lot of time and money into this, on the margin, so I'd be happy if Fedora could benefit from those investments, too.

I spoke too soon.  Things were pretty good for a about 4 days, but I've had three crashes in the last 48 hours on MRP7 Meerkat.  It seems to be pretty WU dependent.  I'll crank out a few hundred successful WUs between crashes.


Name:
Ter5_1_dns_cfbf00097_segment_6_dms_200_40000_72_9750000_0
Workunit ID:
854961295
Created:
18 Dec 2024 1:50:12 UTC
Sent:
18 Dec 2024 3:52:01 UTC
Report deadline:
1 Jan 2025 3:52:01 UTC
Received:
18 Dec 2024 7:38:44 UTC
Server state:
Over
Outcome:
Computation error
Client state:
Compute error
Exit status:
6 (0x00000006) Unknown error code
Computer:
12191882
Run time (sec):
48.59
CPU time (sec):
19.92
Peak working set size (MB):
607.75
Peak swap size (MB):
6256.72
Peak disk usage (MB):
0.02
Validation state:
Invalid
Granted credit:
0
Application:
Binary Radio Pulsar Search (MeerKAT) v0.17 (BRP7-opencl-ati)
x86_64-pc-linux-gnu
Stderr output

<core_client_version>8.0.3</core_client_version>
<![CDATA[
<message>
process exited with code 6 (0x6, -250)</message>
<stderr_txt>
[23:35:06][2967312][INFO ] Application startup - thank you for supporting Einstein@Home!
[23:35:06][2967312][INFO ] Starting data processing...
[23:35:06][2967312][INFO ] Using OpenCL platform provided by: Advanced Micro Devices, Inc.
[23:35:06][2967312][INFO ] Using OpenCL device "gfx1100" by: Advanced Micro Devices, Inc.
[23:35:10][2967312][INFO ] Number of generated templates to be used: 50000
[23:35:10][2967312][INFO ] Checkpoint file unavailable: Ter5_1_dns_cfbf00097_segment_6_dms_200_72.cpt (No such file or directory).
------> Starting from scratch...
[23:35:10][2967312][INFO ] Header contents:
------> Original WAPP file: /atlas/data/TRAPUM_GC/Terzan5/epoch1/30min_segments/dedispersed_files/cfbf00097/Ter5_1_dns_cfbf00097_segment_6_dms_200_DM237.20
------> Sample time in microseconds: 153.121
------> Observation time in seconds: 2568.9524
------> Time stamp (MJD): 59097.776208934454
------> Number of samples/record: 0
------> Center freq in MHz: 857.5673828
------> Channel band in MHz: 3.34375
------> Number of channels/record: 256
------> Nifs: 1
------> RA (J2000): 174807.67
------> DEC (J2000): -244650.099998
------> Galactic l: 0
------> Galactic b: 0
------> Name: J1748-2446N
------> Lagformat: 0
------> Sum: 1
------> Level: 3
------> AZ at start: 0
------> ZA at start: 0
------> AST at start: 0
------> LST at start: 0
------> Project ID: --
------> Observers: --
------> File size (bytes): 0
------> Data size (bytes): 0
------> Number of samples: 16777216
------> Trial dispersion measure: 237.2 cm^-3 pc
------> Scale factor: 1.06933
[23:35:11][2967312][INFO ] Seed for random number generator is -1034112314.
[23:35:14][2967312][INFO ] Derived global search parameters:
------> f_A probability = 0.04
------> single bin prob(P_noise > P_thr) = 3.24424e-09
------> thr1 = 19.5464
------> thr2 = 22.7124
------> thr4 = 27.844
------> thr8 = 36.3869
------> thr16 = 50.9469
Memory access fault by GPU node-1 (Agent handle: 0x186cd920) on address 0x7f1e22287000. Reason: Page not present or supervisor privilege.

[23:35:48][2967312][ERROR] Application caught signal 6.

------> Obtained 10 stack frames for this thread.
------> Backtrace:
BFD: DWARF error: could not find variable specification at offset 685
BFD: DWARF error: could not find variable specification at offset 6fb
BFD: DWARF error: could not find variable specification at offset 2726
BFD: DWARF error: could not find variable specification at offset 2774
BFD: DWARF error: could not find variable specification at offset 277f
BFD: DWARF error: could not find variable specification at offset 2827
BFD: DWARF error: could not find variable specification at offset 2869
BFD: DWARF error: could not find variable specification at offset 28ab
Frame 10:
	Binary file: ../../projects/einstein.phys.uwm.edu/einsteinbinary_BRP7_0.17_x86_64-pc-linux-gnu__BRP7-opencl-ati (0x4b821f)
	Source file: erp_boinc_wrapper.cpp (Function: y&#137;&#235;&#241; / Line: 159)
BFD: /lib64/libc.so.6: unknown type [0x13] section `.relr.dyn'
Frame 9:
	Binary file: /lib64/libc.so.6 (0x7f1fcae25dd0)
	Offset info: +0x19dd0
BFD: /lib64/libc.so.6: unknown type [0x13] section `.relr.dyn'
Frame 8:
	Binary file: /lib64/libc.so.6 (0x7f1fcae7ec94)
	Offset info: +0x72c94
BFD: /lib64/libc.so.6: unknown type [0x13] section `.relr.dyn'
Frame 7:
	Binary file: /lib64/libc.so.6 (0x7f1fcae25d1e)
	Offset info: gsignal+0x1e
BFD: /lib64/libc.so.6: unknown type [0x13] section `.relr.dyn'
Frame 6:
	Binary file: /lib64/libc.so.6 (0x7f1fcae0d942)
	Offset info: abort+0xdf
BFD: /lib64/libhsa-runtime64.so.1: unknown type [0x13] section `.relr.dyn'
Frame 5:
	Binary file: /lib64/libhsa-runtime64.so.1 (0x7f1fc9e5a7ef)
	Offset info: +0x5a7ef
BFD: /lib64/libhsa-runtime64.so.1: unknown type [0x13] section `.relr.dyn'
Frame 4:
	Binary file: /lib64/libhsa-runtime64.so.1 (0x7f1fc9e6223e)
	Offset info: +0x6223e
BFD: /lib64/libhsa-runtime64.so.1: unknown type [0x13] section `.relr.dyn'
Frame 3:
	Binary file: /lib64/libhsa-runtime64.so.1 (0x7f1fc9e1c46c)
	Offset info: +0x1c46c
BFD: /lib64/libc.so.6: unknown type [0x13] section `.relr.dyn'
Frame 2:
	Binary file: /lib64/libc.so.6 (0x7f1fcae7ccd7)
	Offset info: +0x70cd7
BFD: /lib64/libc.so.6: unknown type [0x13] section `.relr.dyn'
Frame 1:
	Binary file: /lib64/libc.so.6 (0x7f1fcaf00c8c)
	Offset info: +0xf4c8c
------> End of backtrace

23:35:53 (2967312): called boinc_finish(6)

</stderr_txt>
]]>

More information.  I realized now that the last few crashes all happened while watching video.  In fact, I think each time it was while starting a youtube video.  It's possible it's related to that, and not Meerkat.

An update.  I now have kernel 6.12.6 and, for whatever reason, stability has improved.  I haven't watched many videos, so perhaps that is is.

However, I turned GW O3AS back on to test, and it is still failing regularly.  So strange.  Here is the output:

Task 1705019600
Name:
h1_0167.80_O3aLC01Cl1In0__O3ASBu_168.00Hz_24438_0
Workunit ID:
858908301
Created:
29 Dec 2024 8:31:03 UTC
Sent:
29 Dec 2024 8:31:04 UTC
Report deadline:
5 Jan 2025 8:31:04 UTC
Received:
29 Dec 2024 9:32:30 UTC
Server state:
Over
Outcome:
Computation error
Client state:
Compute error
Exit status:
255 (0x000000FF) Unknown error code
Computer:
12191882
Run time (sec):
15.28
CPU time (sec):
13.11
Peak working set size (MB):
595.93
Peak swap size (MB):
6470.41
Peak disk usage (MB):
0.03
Validation state:
Invalid
Granted credit:
0
Application:
All-Sky Gravitational Wave search on O3 v1.07 (GW-opencl-ati-2)
x86_64-pc-linux-gnu
Stderr output

<core_client_version>8.0.3</core_client_version>
<![CDATA[
<message>
process exited with code 255 (0xff, -1)</message>
<stderr_txt>
putenv 'LAL_DEBUG_LEVEL=3'
2024-12-29 00:32:01.3116 (1177456) [normal]: This program is published under the GNU General Public License, version 2
2024-12-29 00:32:01.3116 (1177456) [normal]: For details see http://einstein.phys.uwm.edu/license.php
2024-12-29 00:32:01.3116 (1177456) [normal]: This Einstein@home App was built at: Nov  9 2023 13:12:46

2024-12-29 00:32:01.3116 (1177456) [normal]: Start of BOINC application '../../projects/einstein.phys.uwm.edu/einstein_O3AS_1.07_x86_64-pc-linux-gnu__GW-opencl-ati-2'.
[DEBUG} GPU type: 2
[DEBUG} got GPU info from BOINC
[DEBUG} got VendorID 4098
2024-12-29 00:32:01.3873 (1177456) [debug]: Flags: LAL_DEBUG, OPTIMIZE, HS_OPTIMIZATION, GC_SSE2_OPT, X64, SSE, SSE2, GNUC X86 GNUX86
2024-12-29 00:32:01.3874 (1177456) [debug]: glibc version/release: 2.40/stable
2024-12-29 00:32:01.387408 - mytime()
2024-12-29 00:32:01.3875 (1177456) [debug]: Set up communication with graphics process.
2024-12-29 00:32:01.3878 (1177456) [normal]: Parsed user input successfully
Code-version: %% LAL: 7.1.4.1 (CLEAN )
%% LALPulsar: 3.1.0.1 (CLEAN )
%% LALApps: 7.3.0.1 (CLEAN )

2024-12-29 00:32:01.3878 (1177456) [normal]: Initialise compartments with freqWidth = 0.05 and candidates per compartment = 10000.
2024-12-29 00:32:01.3880 (1177456) [debug]: Opening temp output file '../../projects/einstein.phys.uwm.edu/h1_0167.80_O3aLC01Cl1In0__O3ASBu_168.00Hz_24438_0_0.tmp' for writing ...
2024-12-29 00:32:01.5335 (1177456) [normal]: Reading input data ... 
2024-12-29 00:32:01.5336 (1177456) [normal]: Loading SFTs matching '../../projects/einstein.phys.uwm.edu/h1_0167.80_O3aLC01Cl1In0;../../projects/einstein.phys.uwm.edu/l1_0167.80_O3aLC01Cl1In0;../../projects/einstein.phys.uwm.edu/h1_0168.00_O3aLC01Cl1In0;../../projects/einstein.phys.uwm.edu/l1_0168.00_O3aLC01Cl1In0;../../projects/einstein.phys.uwm.edu/h1_0168.20_O3aLC01Cl1In0;../../projects/einstein.phys.uwm.edu/l1_0168.20_O3aLC01Cl1In0;../../projects/einstein.phys.uwm.edu/h1_0168.40_O3aLC01Cl1In0;../../projects/einstein.phys.uwm.edu/l1_0168.40_O3aLC01Cl1In0;../../projects/einstein.phys.uwm.edu/h1_0168.60_O3aLC01Cl1In0;../../projects/einstein.phys.uwm.edu/l1_0168.60_O3aLC01Cl1In0;../../projects/einstein.phys.uwm.edu/h1_0168.80_O3aLC01Cl1In0;../../projects/einstein.phys.uwm.edu/l1_0168.80_O3aLC01Cl1In0;../../projects/einstein.phys.uwm.edu/h1_0169.00_O3aLC01Cl1In0;../../projects/einstein.phys.uwm.edu/l1_0169.00_O3aLC01Cl1In0;../../projects/einstein.phys.uwm.edu/h1_0169.20_O3aLC01Cl1In0;../../projects/einstein.phys.uwm.edu/l1_0169.20_O3aLC01Cl1In0' into catalog ...2024-12-29 00:32:01.6548 (1177456) [normal]: done.
2024-12-29 00:32:01.6549 (1177456) [normal]: Validating SFTs (detectors: H1, L1, ) ... success.
2024-12-29 00:32:04.5617 (1177456) [normal]: Search FstatMethod used: 'ResampGPU'
2024-12-29 00:32:04.5617 (1177456) [normal]: Recalc FstatMethod used: 'DemodSSE'
2024-12-29 00:32:04.5617 (1177456) [normal]: GPU Device used for Search/Recalc and/or semi coherent step: 'gfx1100 ( Platform: AMD Accelerated Parallel Processing )'
2024-12-29 00:32:04.5618 (1177456) [normal]: GPU Backend used for Search/Recalc and/or semi coherent step: 'OpenCL'
2024-12-29 00:32:04.5618 (1177456) [normal]: GPU version is used for the semi-coherent step!
2024-12-29 00:32:14.4083 (1177456) [normal]: Number of segments: 18, total number of SFTs in segments: 5748
2024-12-29 00:32:14.4109 (1177456) [normal]: Finished reading input data.
2024-12-29 00:32:14.4109 (1177456) [normal]: GPS reference time = 1246070525.0000 ,  GPS data mid time = 1246070525.0000
2024-12-29 00:32:14.4109 (1177456) [normal]: dFreqStack = 1.000000e-06, df1dot = 7.500000e-11, df2dot = 0.000000e+00, df3dot = 0.000000e+00
2024-12-29 00:32:14.4110 (1177456) [normal]: Setup, N = 18, T = 878400 s, Tobs = 15809012 s, gammaRefine = 250, gamma2Refine = 1098, gamma3Refine = 1
Failed OpenCL buildlog: 
ld.lld: error: undefined symbol: __printf_alloc
>>> referenced by /tmp/comgr-2471db/input/linked.bc.o:(XLALLoopOverCoarseGridFrequencyBins)
>>> referenced by /tmp/comgr-2471db/input/linked.bc.o:(XLALLoopOverCoarseGridFrequencyBins)
Error: Creating the executable from LLVM IRs failed.
XLAL Error - XLALOpenCLGetProgramFromSource (/home/jenkins/workspace/workspace/EaH-GW-OpenCL-Testing/SLAVE/LIBC215/TARGET/linux-x86_64/EinsteinAtHome/source/lalsuite/lalpulsar/lib/GPUUtils/OpenCLUtils.c:705): clBuildProgram failed with OpenCL error: CL_BUILD_PROGRAM_FAILURE
XLAL Error - XLALOpenCLGetProgramFromSource (/home/jenkins/workspace/workspace/EaH-GW-OpenCL-Testing/SLAVE/LIBC215/TARGET/linux-x86_64/EinsteinAtHome/source/lalsuite/lalpulsar/lib/GPUUtils/OpenCLUtils.c:705): Generic failure
XLAL Error - XLALGCTOpenCLKernelsSetup (/home/jenkins/workspace/workspace/EaH-GW-OpenCL-Testing/SLAVE/LIBC215/TARGET/linux-x86_64/EinsteinAtHome/source/lalsuite/lalapps/src/pulsar/GCT/HierarchSearchGCT_OpenCL.c:212): Check failed: XLALOpenCLGetProgramFromSource ( source, &(GCTOpenCLKernels.HierarchSearchGCTProgramm) ) == XLAL_SUCCESS
XLAL Error - XLALGCTOpenCLKernelsSetup (/home/jenkins/workspace/workspace/EaH-GW-OpenCL-Testing/SLAVE/LIBC215/TARGET/linux-x86_64/EinsteinAtHome/source/lalsuite/lalapps/src/pulsar/GCT/HierarchSearchGCT_OpenCL.c:212): Internal function call failed: Generic failure
XLAL Error - MAIN (/home/jenkins/workspace/workspace/EaH-GW-OpenCL-Testing/SLAVE/LIBC215/TARGET/linux-x86_64/EinsteinAtHome/source/lalsuite/lalapps/src/pulsar/GCT/HierarchSearchGCT.c:1394): Check failed: XLALGCTOpenCLKernelsSetup( uvar->SortToplist, uvar->getMaxFperSeg, uvar->computeBSGL, detectorIDs, usefulParams.BSGLsetup ) == XLAL_SUCCESS
XLAL Error - MAIN (/home/jenkins/workspace/workspace/EaH-GW-OpenCL-Testing/SLAVE/LIBC215/TARGET/linux-x86_64/EinsteinAtHome/source/lalsuite/lalapps/src/pulsar/GCT/HierarchSearchGCT.c:1394): Internal function call failed: Generic failure
2024-12-29 00:32:14.6019 (1177456) [CRITICAL]: ERROR: MAIN() returned with error '-1'
Code-version: %% LAL: 7.1.4.1 (CLEAN )
%% LALPulsar: 3.1.0.1 (CLEAN )
%% LALApps: 7.3.0.1 (CLEAN )

FPU status flags:  COND_0
2024-12-29 00:32:14.6021 (1177456) [debug]: worker done. return(-1) to caller
2024-12-29 00:32:14.6021 (1177456) [normal]: done. calling boinc_finish(-1).
00:32:14 (1177456): called boinc_finish
Warning:  Program terminating, but clFFT resources not freed.
Please consider explicitly calling clfftTeardown( ).

</stderr_txt>
]]>

But, I should say that, despite a large number of recent AMDGPU exceptions, which I normally associate with system crashes, the system is recovering from these exceptions by itself.  Normally, I see messages to the effect that AMDGPU couldn't "evict" an error-ed process.  But, I haven't see those messages associated with these most recent page faults.  Another interesting development.

I looked at building the source and could not figure it out.
Is there a build.sh or a reduced testcase you could provide ?

Hmm, I haven't done it before, either.  I can look into it, though.  More soon...

Shoot, I didn't mean to clear the needinfo flag.

Okay, so I reached out to help from E@H:

https://einsteinathome.org/content/o3as-broken-rdna3-linux-opencl-ati-worked-few-months-ago-need-help-building-test-env

They sent me back here for the reason I opened this ticket in the first place: upstream says it's a problem with Fedora drivers.  Seems to work fine on Ubuntu, so I cannot get any help upstream.

E@H also suggested I contact Oliver Behnke.  Does that name mean anything to you?

Just to clarify, the applicable text in the output is:

ld.lld: error: undefined symbol: __printf_alloc

which doesn't really make sense to me.

I do not know what to make of that either.
I do not know who Oliver is either.
Is there a container or something that you can share that i could reproduce the problem with ?
I have no knowledge of the E@H project, something turnkey would be best.

Okay, I'll see what I can do.  I know the code is open.  I don't know about turnkey, but I'll see if I can get there.

*** Bug 2218022 has been marked as a duplicate of this bug. ***

(From closed bug)
> I assume it's a duplicate of 2330958? Also Fedora 38 is long past EOL and "rocm-opencl" was replaced with "rocclr". Feel free to reopen against rocclr if 2330958 doesn't capture the issue. I believe Tom Rix was helping out there.

>Note that we've sort of redone a lot of the packaging to be more aligned with upstream (we originally used a different compiler).

Always pleased to hear about improvements.  But, I'm confused about rocclr, though.  There is no rocclr in F42.  I still have rocm-opencl.  So, I'm sure I don't understand what you mean.

I haven't had time to do what I promised.  Sorry.

This message is a reminder that Fedora Linux 41 is nearing its end of life.
Fedora will stop maintaining and issuing updates for Fedora Linux 41 on 2025-12-15.
It is Fedora's policy to close all bug reports from releases that are no longer
maintained. At that time this bug will be closed as EOL if it remains open with a
'version' of '41'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, change the 'version' 
to a later Fedora Linux version. Note that the version field may be hidden.
Click the "Show advanced fields" button if you do not see it.

Thank you for reporting this issue and we are sorry that we were not 
able to fix it before Fedora Linux 41 is end of life. If you would still like 
to see this bug fixed and are able to reproduce it against a later version 
of Fedora Linux, you are encouraged to change the 'version' to a later version
prior to this bug being closed.

Fedora Linux 41 entered end-of-life (EOL) status on 2025-12-15.

Fedora Linux 41 is no longer maintained, which means that it
will not receive any further security or bug fix updates. As a result we
are closing this bug.

If you can reproduce this bug against a currently maintained version of Fedora Linux
please feel free to reopen this bug against that version. Note that the version
field may be hidden. Click the "Show advanced fields" button if you do not see
the version field.

If you are unable to reopen this bug, please file a new report against an
active release.

Thank you for reporting this bug and we are sorry it could not be fixed.