Bug 2367960

Summary: rustc 1.87.0 panics while compiling rust-hyper-util on ppc64le
Product: [Fedora] Fedora Reporter: Ben Beasley <code>
Component: rustAssignee: Josh Stone <jistone>
Status: CLOSED RAWHIDE QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: medium Docs Contact:
Priority: unspecified    
Version: rawhideCC: amulhern, decathorpe, fedora, igor.raits, jchecahi, jistone, paumurph, rust-sig, TicoTimo
Target Milestone: ---Flags: jistone: mirror+
Target Release: ---   
Hardware: Unspecified   
OS: Linux   
Whiteboard:
Fixed In Version: rust-1.89.0-4.fc44 Doc Type: ---
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2025-08-18 22:21:12 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 2366643    

Description Ben Beasley 2025-05-22 10:43:46 UTC 
In Rawhide, with rust-1.87.0-1.fc43, rustc panics while compiling rust-hyper-util on ppc64le. This worked with rust 1.86; see the PR[1] and associated scratch build[2].

[1] https://src.fedoraproject.org/rpms/rust-hyper-util/pull-request/1
[2] https://koji.fedoraproject.org/koji/taskinfo?taskID=133014799

Reproducible: Always

Steps to Reproduce:
fedpkg co rust-hyper-util
cd rust-hyper-util
fedpkg build
Actual Results:
thread 'rustc' panicked at compiler/rustc_infer/src/infer/outlives/obligations.rs:124:40:
already borrowed: BorrowMutError
stack backtrace:
   0:     0x7fff929d0de8 - <std::sys::backtrace::BacktraceLock::print::DisplayBacktrace as core::fmt::Display>::fmt::h358da5a7e33a66e2
   1:     0x7fff9215b838 - core::fmt::write::h6bc2db31dd03e7df
   2:     0x7fff929bec44 - <unknown>
   3:     0x7fff929d0be4 - <unknown>
   4:     0x7fff929d41a4 - <unknown>
   5:     0x7fff929d3b50 - std::panicking::default_hook::hc8100c38be18401f
   6:     0x7fff8f5a0b6c - <unknown>
   7:     0x7fff8f571530 - <unknown>
   8:     0x7fff929d50a8 - std::panicking::rust_panic_with_hook::hb454b0dcbf5fc939
   9:     0x7fff929d4b38 - <unknown>
  10:     0x7fff929d14b0 - std::sys::backtrace::__rust_end_short_backtrace::h03d44392342f1184
  11:     0x7fff929d4584 - __rustc[c2610c6383308acc]::rust_begin_unwind
  12:     0x7fff8edb9414 - core::panicking::panic_fmt::hf33b6b3da6dfe6bc
  13:     0x7fff8edb9278 - core::cell::panic_already_borrowed::h59a581a7b2415564
  14:     0x7fff90d7f280 - <rustc_borrowck[ff17395deed2ec8e]::type_check::TypeChecker as rustc_middle[687e2f65837ea9de]::mir::visit::Visitor>::visit_body
  15:     0x7fff90ce3f5c - <unknown>
  16:     0x7fff90cadac0 - <unknown>
  17:     0x7fff90ddbf10 - <unknown>
  18:     0x7fff91d2d7a8 - <unknown>
  19:     0x7fff91cc9adc - rustc_query_system[9f00eb678419a2a2]::query::plumbing::try_execute_query::<rustc_query_impl[4578bb995ede35ef]::DynamicConfig<rustc_data_structures[7e0816562978da6d]::vec_cache::VecCache<rustc_span[c8935130265359c6]::def_id::LocalDefId, rustc_middle[687e2f65837ea9de]::query::erase::Erased<[u8; 8usize]>, rustc_query_system[9f00eb678419a2a2]::dep_graph::graph::DepNodeIndex>, false, false, false>, rustc_query_impl[4578bb995ede35ef]::plumbing::QueryCtxt, false>
  20:     0x7fff91d50d04 - <unknown>
  21:     0x7fff914a7690 - <unknown>
  22:     0x7fff9251feac - rustc_interface[537fa850fc24b132]::passes::analysis
  23:     0x7fff927bf39c - <unknown>
  24:     0x7fff9270d844 - rustc_query_system[9f00eb678419a2a2]::query::plumbing::try_execute_query::<rustc_query_impl[4578bb995ede35ef]::DynamicConfig<rustc_query_system[9f00eb678419a2a2]::query::caches::SingleCache<rustc_middle[687e2f65837ea9de]::query::erase::Erased<[u8; 0usize]>>, false, false, false>, rustc_query_impl[4578bb995ede35ef]::plumbing::QueryCtxt, false>
  25:     0x7fff927c6238 - <unknown>
  26:     0x7fff923b43ac - <unknown>
  27:     0x7fff923b0594 - <unknown>
  28:     0x7fff923ac588 - std[8587342a5b72dfb4]::sys::backtrace::__rust_begin_short_backtrace::<rustc_interface[537fa850fc24b132]::util::run_in_thread_with_globals<rustc_interface[537fa850fc24b132]::util::run_in_thread_pool_with_globals<rustc_interface[537fa850fc24b132]::interface::run_compiler<(), rustc_driver_impl[d9b811d677f6ce88]::run_compiler::{closure#0}>::{closure#1}, ()>::{closure#0}, ()>::{closure#0}::{closure#0}, ()>
  29:     0x7fff923b7c54 - <unknown>
  30:     0x7fff929dc5c8 - <unknown>
  31:     0x7fff8e2b693c - start_thread
  32:     0x7fff8e363908 - __clone3
error: the compiler unexpectedly panicked. this is a bug.
note: we would appreciate a bug report: https://github.com/rust-lang/rust/issues/new?labels=C-bug%2C+I-ICE%2C+T-compiler&template=ice.md
note: please make sure that you have updated to the latest nightly
warning: the ICE couldn't be written to `/usr/share/cargo/registry/hyper-1.6.0/rustc-ice-2025-05-22T02_05_45-332.txt`: Permission denied (os error 13)
note: rustc 1.87.0 (17067e9ac 2025-05-09) (Fedora 1.87.0-1.fc43) running on powerpc64le-unknown-linux-gnu
note: compiler flags: --crate-type lib -C opt-level=3 -C embed-bitcode=no -C codegen-units=1 -C debuginfo=2 -C opt-level=3 -C debuginfo=2 -C codegen-units=1 -C strip=none -C link-arg=-specs=/usr/lib/rpm/redhat/redhat-package-notes
note: some of the compiler flags provided by cargo are hidden
query stack during panic:
#0 [mir_borrowck] borrow-checking `client::conn::http1::<impl at /usr/share/cargo/registry/hyper-1.6.0/src/client/conn/http1.rs:279:1: 284:53>::poll`
#1 [analysis] running analysis passes on this crate
end of query stack
thread 'rustc' panicked at compiler/rustc_infer/src/infer/outlives/obligations.rs:124:40:
already borrowed: BorrowMutError
stack backtrace:
   0:     0x7fff929d0de8 - <std::sys::backtrace::BacktraceLock::print::DisplayBacktrace as core::fmt::Display>::fmt::h358da5a7e33a66e2
   1:     0x7fff9215b838 - core::fmt::write::h6bc2db31dd03e7df
   2:     0x7fff929bec44 - <unknown>
   3:     0x7fff929d0be4 - <unknown>
   4:     0x7fff929d41a4 - <unknown>
   5:     0x7fff929d3b50 - std::panicking::default_hook::hc8100c38be18401f
   6:     0x7fff8f5a0b6c - <unknown>
   7:     0x7fff8f571530 - <unknown>
   8:     0x7fff929d50a8 - std::panicking::rust_panic_with_hook::hb454b0dcbf5fc939
   9:     0x7fff929d4b38 - <unknown>
  10:     0x7fff929d14b0 - std::sys::backtrace::__rust_end_short_backtrace::h03d44392342f1184
  11:     0x7fff929d4584 - __rustc[c2610c6383308acc]::rust_begin_unwind
  12:     0x7fff8edb9414 - core::panicking::panic_fmt::hf33b6b3da6dfe6bc
  13:     0x7fff8edb9278 - core::cell::panic_already_borrowed::h59a581a7b2415564
  14:     0x7fff90d7f280 - <rustc_borrowck[ff17395deed2ec8e]::type_check::TypeChecker as rustc_middle[687e2f65837ea9de]::mir::visit::Visitor>::visit_body
  15:     0x7fff90ce3f5c - <unknown>
  16:     0x7fff90cadac0 - <unknown>
  17:     0x7fff90ddbf10 - <unknown>
  18:     0x7fff91d2d7a8 - <unknown>
  19:     0x7fff91cc9adc - rustc_query_system[9f00eb678419a2a2]::query::plumbing::try_execute_query::<rustc_query_impl[4578bb995ede35ef]::DynamicConfig<rustc_data_structures[7e0816562978da6d]::vec_cache::VecCache<rustc_span[c8935130265359c6]::def_id::LocalDefId, rustc_middle[687e2f65837ea9de]::query::erase::Erased<[u8; 8usize]>, rustc_query_system[9f00eb678419a2a2]::dep_graph::graph::DepNodeIndex>, false, false, false>, rustc_query_impl[4578bb995ede35ef]::plumbing::QueryCtxt, false>
  20:     0x7fff91d50d04 - <unknown>
  21:     0x7fff914a7690 - <unknown>
  22:     0x7fff9251feac - rustc_interface[537fa850fc24b132]::passes::analysis
  23:     0x7fff927bf39c - <unknown>
  24:     0x7fff9270d844 - rustc_query_system[9f00eb678419a2a2]::query::plumbing::try_execute_query::<rustc_query_impl[4578bb995ede35ef]::DynamicConfig<rustc_query_system[9f00eb678419a2a2]::query::caches::SingleCache<rustc_middle[687e2f65837ea9de]::query::erase::Erased<[u8; 0usize]>>, false, false, false>, rustc_query_impl[4578bb995ede35ef]::plumbing::QueryCtxt, false>
  25:     0x7fff927c6238 - <unknown>
  26:     0x7fff923b43ac - <unknown>
  27:     0x7fff923b0594 - <unknown>
  28:     0x7fff923ac588 - std[8587342a5b72dfb4]::sys::backtrace::__rust_begin_short_backtrace::<rustc_interface[537fa850fc24b132]::util::run_in_thread_with_globals<rustc_interface[537fa850fc24b132]::util::run_in_thread_pool_with_globals<rustc_interface[537fa850fc24b132]::interface::run_compiler<(), rustc_driver_impl[d9b811d677f6ce88]::run_compiler::{closure#0}>::{closure#1}, ()>::{closure#0}, ()>::{closure#0}::{closure#0}, ()>
  29:     0x7fff923b7c54 - <unknown>
  30:     0x7fff929dc5c8 - <unknown>
  31:     0x7fff8e2b693c - start_thread
  32:     0x7fff8e363908 - __clone3
error: the compiler unexpectedly panicked. this is a bug.
note: we would appreciate a bug report: https://github.com/rust-lang/rust/issues/new?labels=C-bug%2C+I-ICE%2C+T-compiler&template=ice.md
note: please make sure that you have updated to the latest nightly
warning: the ICE couldn't be written to `/usr/share/cargo/registry/hyper-1.6.0/rustc-ice-2025-05-22T02_05_45-332.txt`: Permission denied (os error 13)
note: rustc 1.87.0 (17067e9ac 2025-05-09) (Fedora 1.87.0-1.fc43) running on powerpc64le-unknown-linux-gnu
note: compiler flags: --crate-type lib -C opt-level=3 -C embed-bitcode=no -C codegen-units=1 -C debuginfo=2 -C opt-level=3 -C debuginfo=2 -C codegen-units=1 -C strip=none -C link-arg=-specs=/usr/lib/rpm/redhat/redhat-package-notes
note: some of the compiler flags provided by cargo are hidden
query stack during panic:
#0 [mir_borrowck] borrow-checking `client::conn::http1::upgrades::<impl at /usr/share/cargo/registry/hyper-1.6.0/src/client/conn/http1.rs:590:5: 595:57>::poll`
#1 [analysis] running analysis passes on this crate
end of query stack

Expected Results:
Successful build
Comment 1 Ben Beasley 2025-05-22 10:57:05 UTC 
A very similar-looking ppc64le-only panic was spotted in Koschei for rust-metrics-exporter-prometheus: https://koschei.fedoraproject.org/build/20568549
Comment 2 Fabio Valentini 2025-05-22 13:24:11 UTC 
There's not only panics, but also straight up segfaults in librust_driver on ppc64le:

error: rustc interrupted by SIGSEGV, printing backtrace
/lib64/librustc_driver-bbe9c34a8516c373.so(+0xfa6bac) [0x7fff7f1a6bac]
linux-vdso64.so.1(__kernel_sigtramp_rt64+0x0) [0x7fff83500484]
/lib64/librustc_driver-bbe9c34a8516c373.so(_RNvXs1_NtCslTQcjdwlyaw_14rustc_borrowck10type_checkNtB5_11TypeCheckerNtNtNtCs8YddYaRJ3vC_12rustc_middle3mir5visit7Visitor10visit_body+0xbf00) [0x7fff8095ca60]
(...)
note: we would appreciate a report at https://github.com/rust-lang/rust
help: you can increase rustc's stack size by setting RUST_MIN_STACK=16777216
note: backtrace dumped due to SIGSEGV! resuming signal
error: could not compile `tokio-stream` (lib)

from https://koschei.fedoraproject.org/build/20573346
Comment 3 Josh Stone 2025-05-22 22:06:42 UTC 
1.87.0-1 has been untagged from rawhide and eln, and we hadn't submitted f41 and f42 to bodhi at all yet.

Jesus also tried a scratch build with ppc64le PGO disabled, and that build looks better at first glance, but I will investigate more before we try to push out a new build.
Comment 4 Josh Stone 2025-05-28 16:13:38 UTC 
I disabled ppc64le PGO in 1.87.0-2.fc43, and so far it looks good in koschei, as well as my own scratch builds for these particular packages:

rust-hyper-util: https://koji.fedoraproject.org/koji/taskinfo?taskID=133281625
rust-tokio: https://koji.fedoraproject.org/koji/taskinfo?taskID=133281626

I'm going to leave this open though, because we haven't actually found the underlying bug yet.
Comment 5 Paul Murphy 2025-07-22 19:46:23 UTC 
I investigated this and found it to be an llvm codegen bug already reported as https://github.com/llvm/llvm-project/issues/138208, and fixed upstream.
Comment 6 Josh Stone 2025-08-12 15:45:01 UTC 
The LLVM fix has been backported -- I'll try re-enabling ppc64le PGO and see how these go.
Comment 7 Josh Stone 2025-08-18 22:21:12 UTC 
PGO is back on in rust-1.89.0-4.fc44, and these scratch builds passed:

rust-hyper-util: https://koji.fedoraproject.org/koji/taskinfo?taskID=136214151
rust-tokio: https://koji.fedoraproject.org/koji/taskinfo?taskID=136214170