Bug 1902663

Summary: compiling doctests fails during %cargo_test on aarch64 only
Product: [Fedora] Fedora Reporter: Fabio Valentini <decathorpe>
Component: rustAssignee: Rust SIG <rust-sig>
Status: CLOSED CURRENTRELEASE QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 34CC: amulhern, eclipseo, igor.raits, jistone, rust-sig, TicoTimo
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2021-06-16 23:24:54 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
stack traces with missing annotations, gdb hates me none

Description Fabio Valentini 2020-11-30 10:31:24 UTC
I have now found two packages that fail to build on aarch64 only during compilation of doctests: clap and slog

clap: https://koji.fedoraproject.org/koji/taskinfo?taskID=56241039
slog: https://koji.fedoraproject.org/koji/taskinfo?taskID=56431109

There is no output for the failed tests other than:
"Couldn't compile the test."

Comment 1 Fabio Valentini 2021-01-25 16:24:12 UTC
Also affects nom 5:
https://koji.fedoraproject.org/koji/taskinfo?taskID=60471169

Comment 2 Josh Stone 2021-01-25 19:27:28 UTC
Is this a consistent failure in those packages, or only occasional?

Comment 3 Fabio Valentini 2021-01-25 20:20:46 UTC
The failures happens consistently every time on aarch64 for the affected packages.

Though nom seems to be the worst of the bunch, with *all* non-ignored doctests failing to compile.

Looks like rustdoc is crashing ...

Comment 4 Josh Stone 2021-01-25 21:04:44 UTC
aarch64-unknown-linux-gnu is now considered a tier-1 target, so if this is reproducible with the upstream toolchain, I expect it will get a lot of attention.

Comment 5 Robert-André Mauchin 🐧 2021-01-29 15:13:21 UTC
same with rust-slice-deque

Comment 6 Fabio Valentini 2021-01-29 18:22:11 UTC
I'll be getting aarch64 hardware on Monday, so I hope to be able to debug this better soon :)

Comment 7 Fabio Valentini 2021-02-04 21:21:10 UTC
Alright. I have reproduced this natively on aarch64 hardware.

Cloned the clap git repository, ran "cargo test --release".
With cargo + rust from Fedora 33 repos I have these crashes:

Process 20893 (rustc) crashed in getDefSrcRegIgnoringCopies(llvm::Register, llvm::MachineRegisterInfo const&)()

 rustc killed by SIGSEGV
 
 #1 [libLLVM-11.so] getDefSrcRegIgnoringCopies(llvm::Register, llvm::MachineRegisterInfo const&)
 #2 [libLLVM-11.so] llvm::getDefIgnoringCopies(llvm::Register, llvm::MachineRegisterInfo const&)
 #3 [libLLVM-11.so] llvm::getOpcodeDef(unsigned int, llvm::Register, llvm::MachineRegisterInfo const&)
 #4 [libLLVM-11.so] llvm::CombinerHelper::matchUndefStore(llvm::MachineInstr&)
 #5 [libLLVM-11.so] (anonymous namespace)::AArch64GenPreLegalizerCombinerHelper::tryCombineAll(llvm::GISelChangeObserver&, llvm::MachineInstr&, llvm::MachineIRBuilder&) const [clone .constprop.0]

I'll try to reproduce with stable rust installed via rustup next.

Comment 8 Fabio Valentini 2021-02-04 23:34:51 UTC
I cannot reproduce the crashes with "official" Rust binaries installed with rustup:

stable-aarch64-unknown-linux-gnu (default)
rustc 1.49.0 (e1884a8e3 2020-12-29)

So the crashes are specific to the rust / cargo packages installed from Fedora repos:

rust-1.49.0-1.fc33.aarch64
cargo-1.49.0-1.fc33.aarch64

(or rather, LLVM, looking at the stack trace in the last comment?)

Comment 9 Josh Stone 2021-02-08 19:27:06 UTC
Can you install the corresponding llvm-libs-debuginfo so we get inlines and file:line in the backtrace?

There are a few backports in upstream rust's llvm that *might* be relevant, but their bugs don't match your backtrace info AFAICS.

Comment 10 Fabio Valentini 2021-02-08 23:38:36 UTC
Interesting. gdb crashed while generating a backtrace from the coredump ...
This is the only thing I got from it:

Program terminated with signal SIGSEGV, Segmentation fault.
#0  llvm::MachineInstr::getOperand (i=0, this=0x0) at ../include/llvm/CodeGen/MachineInstr.h:472
472	    return Operands[i];

Comment 11 Fabio Valentini 2021-02-09 00:40:04 UTC
Something very weird is going on.

debuginfo for all involved packages is already installed:
Going to install 0 debuginfo packages

But gdb can't find any of them:
Can't find packages for 16 debuginfo files

Missing debuginfo file: /usr/lib/debug/.build-id/c1/d93c73ef25ac17c8fc246842d95e835bfe6920.debug
Missing debuginfo file: /usr/lib/.build-id/6d/5c6c52f2523cfecd1f850120f3cda4c1fd7ff8.debug
Missing debuginfo file: /usr/lib/.build-id/bb/70237d48dc3f8c59e4a24f7957149334f3a476.debug
Missing debuginfo file: /usr/lib/.build-id/f8/b6f038f29adf3d706c50fd31ff5d1a1c9e2a0b.debug
Missing debuginfo file: /usr/lib/.build-id/d8/8efbcde5da4516f55939527314406d6acc5fa4.debug
Missing debuginfo file: /usr/lib/.build-id/c1/d93c73ef25ac17c8fc246842d95e835bfe6920.debug
Missing debuginfo file: /usr/lib/.build-id/e5/428466c972ce3253ee98c1bf4e0bd8a3e46450.debug
Missing debuginfo file: /usr/lib/.build-id/b0/409727b2ea2ace7d340b21f4c22a49c8586238.debug
Missing debuginfo file: /usr/lib/.build-id/d8/f5acb829dca2b1fc56f8c9370914f1dae7214a.debug
Missing debuginfo file: /usr/lib/.build-id/b9/d5a34ebc331bf181da2a96a3f061928afcfda3.debug
Missing debuginfo file: /usr/lib/.build-id/d1/db9e285cb7c68f98d2becbcf4a74cf4dc3505a.debug
Missing debuginfo file: /usr/lib/.build-id/e3/1ef7b1bb28ff8e35ca83ef10d5ff5b2c8b4e50.debug
Missing debuginfo file: /usr/lib/.build-id/de/6207efea2eed6cb4dbd7f72f20d2fe8f07811a.debug
Missing debuginfo file: /usr/lib/.build-id/19/1b98dee8807a42746c5ae1fc59f0919e297781.debug
Missing debuginfo file: /usr/lib/.build-id/3c/954829f1ba61a250a43048cbcdd43bd3ff767d.debug
Missing debuginfo file: /usr/lib/.build-id/74/4e3bac88fb0888aa37a5c322974ae8f672a829.debug

And indeed, those files do not exist, only their non-debug-suffix symlink counterparts.


I have attached the stack traces I could wrangle out from coredumpctl, since gdb apparently cannot generate any annotated backtraces for me without crashing.

Comment 12 Fabio Valentini 2021-02-09 00:40:49 UTC
Created attachment 1755819 [details]
stack traces with missing annotations, gdb hates me

Comment 13 Ben Cotton 2021-02-09 15:29:09 UTC
This bug appears to have been reported against 'rawhide' during the Fedora 34 development cycle.
Changing version to 34.

Comment 14 Fabio Valentini 2021-06-16 23:24:54 UTC
As far as I can tell, this has been fixed for rawhide and f34 since Rust switched to LLVM 12.