Bug 2337089

Summary: dlib: tests crash on aarch64
Product: [Fedora] Fedora Reporter: Ben Beasley <code>
Component: dlibAssignee: Onuralp SEZER <thunderbirdtr>
Status: CLOSED ERRATA QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 42CC: arthur, gui1ty, neuro-sig, thunderbirdtr
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: dlib-19.24.8-3.fc42 dlib-19.24.8-3.fc40 dlib-19.24.8-3.fc41 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2025-03-24 01:32:47 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Ben Beasley 2025-01-11 12:11:32 UTC
Description of problem:

The dtest executable crashes on aarch64.

Version-Release number of selected component (if applicable): dlib-19.24.6-3.fc42


How reproducible:

Build the package on aarch64.


Actual results:

+ ./dtest --runall --no_test_ffmpeg
Running test_matrix3 \ double free or corruption (!prev)
qemu: uncaught target signal 6 (Aborted) - core dumped

(The above is in emulation; this happens on real hardware, too.)


Expected results:

All tests pass.


Additional info:

First failing Koschei build: https://koschei.fedoraproject.org/build/19123027

Comment 1 Ben Beasley 2025-01-11 12:13:02 UTC
(gdb) run
Starting program: /builddir/build/BUILD/dlib-19.24.6-build/dlib-19.24.6/dlib/test/redhat-linux-build/dtest --runall --no_test_ffmpeg
warning: Could not trace the inferior process.
warning: ptrace: Function not implemented
During startup program exited with code 127.

It looks like I can only debug this on real hardware.

Comment 2 Sandro 2025-01-11 12:28:12 UTC
No issues building `dlib` in Copr. The hardware according to `hw_info.log` differs slightly (stepping, flags).

We are also tweaking the parallelism in the spec based on memory available. On the Koji builders this results in `%global _smp_mflags "-j5"`, whereas in Copr it's `-j1`. I could try just setting it to `-j1` for `aarch64` regardless of memory available. But I'm not sure it's actually related.

Comment 3 Sandro 2025-01-11 21:48:18 UTC
The good news, if you want to call it that, is: only `test_matrix3` fails on aarch64 and it fails consistently. When I run only that test with full logging enabled, the log shows the following entries before the test explodes:

   31 TRACE [0] test.matrix3:  
big double
   33 TRACE [0] test.matrix3: 1.1
   41 TRACE [0] test.matrix3: 1.2
   45 TRACE [0] test.matrix3: 1.3
   49 TRACE [0] test.matrix3: 1.4
   52 TRACE [0] test.matrix3: 1.5
   58 TRACE [0] test.matrix3: 1.5.1
   67 TRACE [0] test.matrix3: 1.5.2
   73 TRACE [0] test.matrix3: 1.5.3
   83 TRACE [0] test.matrix3: 1.6
   88 TRACE [0] test.matrix3: 1.6.1
   97 TRACE [0] test.matrix3: 1.6.2
  102 TRACE [0] test.matrix3: 1.6.2.1
  107 TRACE [0] test.matrix3: 1.6.3
  118 TRACE [0] test.matrix3: 1.7

The bad news: the output is not very helpful. I looked at the code of the test. Right after "big double" is logged, it calls `test_blas<double>(120,131)`. That function mentions something with regards to matrices and references `matrix.h`[1] and after "1.7" is printed I see two function calls [2]. That's where my expertise ends. Maybe that helps. But a proper backtrace would of course be better.

[1] https://github.com/davisking/dlib/blob/master/dlib/matrix/matrix.h
[2] https://github.com/davisking/dlib/blob/daefd41b9fff274ca9e70e9b5f65d25a9799ca96/dlib/test/matrix3.cpp#L211C1-L212C118

Comment 4 Aoife Moloney 2025-02-26 13:22:32 UTC
This bug appears to have been reported against 'rawhide' during the Fedora Linux 42 development cycle.
Changing version to 42.

Comment 5 Fedora Update System 2025-03-19 09:38:40 UTC
FEDORA-2025-118d5710f8 (dlib-19.24.8-3.fc41) has been submitted as an update to Fedora 41.
https://bodhi.fedoraproject.org/updates/FEDORA-2025-118d5710f8

Comment 6 Fedora Update System 2025-03-19 09:38:53 UTC
FEDORA-2025-3ac216fc54 (dlib-19.24.8-3.fc40) has been submitted as an update to Fedora 40.
https://bodhi.fedoraproject.org/updates/FEDORA-2025-3ac216fc54

Comment 7 Fedora Update System 2025-03-19 09:39:05 UTC
FEDORA-2025-f045e7d0ee (dlib-19.24.8-3.fc42) has been submitted as an update to Fedora 42.
https://bodhi.fedoraproject.org/updates/FEDORA-2025-f045e7d0ee

Comment 8 Fedora Update System 2025-03-20 06:12:36 UTC
FEDORA-2025-f045e7d0ee has been pushed to the Fedora 42 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --refresh --advisory=FEDORA-2025-f045e7d0ee`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2025-f045e7d0ee

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 9 Fedora Update System 2025-03-20 06:31:41 UTC
FEDORA-2025-118d5710f8 has been pushed to the Fedora 41 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --refresh --advisory=FEDORA-2025-118d5710f8`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2025-118d5710f8

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 10 Fedora Update System 2025-03-20 06:50:18 UTC
FEDORA-2025-3ac216fc54 has been pushed to the Fedora 40 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --refresh --advisory=FEDORA-2025-3ac216fc54`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2025-3ac216fc54

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 11 Fedora Update System 2025-03-24 01:32:47 UTC
FEDORA-2025-f045e7d0ee (dlib-19.24.8-3.fc42) has been pushed to the Fedora 42 stable repository.
If problem still persists, please make note of it in this bug report.

Comment 12 Fedora Update System 2025-03-28 14:48:29 UTC
FEDORA-2025-3ac216fc54 (dlib-19.24.8-3.fc40) has been pushed to the Fedora 40 stable repository.
If problem still persists, please make note of it in this bug report.

Comment 13 Fedora Update System 2025-03-28 15:05:11 UTC
FEDORA-2025-118d5710f8 (dlib-19.24.8-3.fc41) has been pushed to the Fedora 41 stable repository.
If problem still persists, please make note of it in this bug report.