In Rawhide, with rust-1.87.0-1.fc43, rustc panics while compiling rust-hyper-util on ppc64le. This worked with rust 1.86; see the PR[1] and associated scratch build[2]. [1] https://src.fedoraproject.org/rpms/rust-hyper-util/pull-request/1 [2] https://koji.fedoraproject.org/koji/taskinfo?taskID=133014799 Reproducible: Always Steps to Reproduce: fedpkg co rust-hyper-util cd rust-hyper-util fedpkg build Actual Results: thread 'rustc' panicked at compiler/rustc_infer/src/infer/outlives/obligations.rs:124:40: already borrowed: BorrowMutError stack backtrace: 0: 0x7fff929d0de8 - <std::sys::backtrace::BacktraceLock::print::DisplayBacktrace as core::fmt::Display>::fmt::h358da5a7e33a66e2 1: 0x7fff9215b838 - core::fmt::write::h6bc2db31dd03e7df 2: 0x7fff929bec44 - <unknown> 3: 0x7fff929d0be4 - <unknown> 4: 0x7fff929d41a4 - <unknown> 5: 0x7fff929d3b50 - std::panicking::default_hook::hc8100c38be18401f 6: 0x7fff8f5a0b6c - <unknown> 7: 0x7fff8f571530 - <unknown> 8: 0x7fff929d50a8 - std::panicking::rust_panic_with_hook::hb454b0dcbf5fc939 9: 0x7fff929d4b38 - <unknown> 10: 0x7fff929d14b0 - std::sys::backtrace::__rust_end_short_backtrace::h03d44392342f1184 11: 0x7fff929d4584 - __rustc[c2610c6383308acc]::rust_begin_unwind 12: 0x7fff8edb9414 - core::panicking::panic_fmt::hf33b6b3da6dfe6bc 13: 0x7fff8edb9278 - core::cell::panic_already_borrowed::h59a581a7b2415564 14: 0x7fff90d7f280 - <rustc_borrowck[ff17395deed2ec8e]::type_check::TypeChecker as rustc_middle[687e2f65837ea9de]::mir::visit::Visitor>::visit_body 15: 0x7fff90ce3f5c - <unknown> 16: 0x7fff90cadac0 - <unknown> 17: 0x7fff90ddbf10 - <unknown> 18: 0x7fff91d2d7a8 - <unknown> 19: 0x7fff91cc9adc - rustc_query_system[9f00eb678419a2a2]::query::plumbing::try_execute_query::<rustc_query_impl[4578bb995ede35ef]::DynamicConfig<rustc_data_structures[7e0816562978da6d]::vec_cache::VecCache<rustc_span[c8935130265359c6]::def_id::LocalDefId, rustc_middle[687e2f65837ea9de]::query::erase::Erased<[u8; 8usize]>, rustc_query_system[9f00eb678419a2a2]::dep_graph::graph::DepNodeIndex>, false, false, false>, rustc_query_impl[4578bb995ede35ef]::plumbing::QueryCtxt, false> 20: 0x7fff91d50d04 - <unknown> 21: 0x7fff914a7690 - <unknown> 22: 0x7fff9251feac - rustc_interface[537fa850fc24b132]::passes::analysis 23: 0x7fff927bf39c - <unknown> 24: 0x7fff9270d844 - rustc_query_system[9f00eb678419a2a2]::query::plumbing::try_execute_query::<rustc_query_impl[4578bb995ede35ef]::DynamicConfig<rustc_query_system[9f00eb678419a2a2]::query::caches::SingleCache<rustc_middle[687e2f65837ea9de]::query::erase::Erased<[u8; 0usize]>>, false, false, false>, rustc_query_impl[4578bb995ede35ef]::plumbing::QueryCtxt, false> 25: 0x7fff927c6238 - <unknown> 26: 0x7fff923b43ac - <unknown> 27: 0x7fff923b0594 - <unknown> 28: 0x7fff923ac588 - std[8587342a5b72dfb4]::sys::backtrace::__rust_begin_short_backtrace::<rustc_interface[537fa850fc24b132]::util::run_in_thread_with_globals<rustc_interface[537fa850fc24b132]::util::run_in_thread_pool_with_globals<rustc_interface[537fa850fc24b132]::interface::run_compiler<(), rustc_driver_impl[d9b811d677f6ce88]::run_compiler::{closure#0}>::{closure#1}, ()>::{closure#0}, ()>::{closure#0}::{closure#0}, ()> 29: 0x7fff923b7c54 - <unknown> 30: 0x7fff929dc5c8 - <unknown> 31: 0x7fff8e2b693c - start_thread 32: 0x7fff8e363908 - __clone3 error: the compiler unexpectedly panicked. this is a bug. note: we would appreciate a bug report: https://github.com/rust-lang/rust/issues/new?labels=C-bug%2C+I-ICE%2C+T-compiler&template=ice.md note: please make sure that you have updated to the latest nightly warning: the ICE couldn't be written to `/usr/share/cargo/registry/hyper-1.6.0/rustc-ice-2025-05-22T02_05_45-332.txt`: Permission denied (os error 13) note: rustc 1.87.0 (17067e9ac 2025-05-09) (Fedora 1.87.0-1.fc43) running on powerpc64le-unknown-linux-gnu note: compiler flags: --crate-type lib -C opt-level=3 -C embed-bitcode=no -C codegen-units=1 -C debuginfo=2 -C opt-level=3 -C debuginfo=2 -C codegen-units=1 -C strip=none -C link-arg=-specs=/usr/lib/rpm/redhat/redhat-package-notes note: some of the compiler flags provided by cargo are hidden query stack during panic: #0 [mir_borrowck] borrow-checking `client::conn::http1::<impl at /usr/share/cargo/registry/hyper-1.6.0/src/client/conn/http1.rs:279:1: 284:53>::poll` #1 [analysis] running analysis passes on this crate end of query stack thread 'rustc' panicked at compiler/rustc_infer/src/infer/outlives/obligations.rs:124:40: already borrowed: BorrowMutError stack backtrace: 0: 0x7fff929d0de8 - <std::sys::backtrace::BacktraceLock::print::DisplayBacktrace as core::fmt::Display>::fmt::h358da5a7e33a66e2 1: 0x7fff9215b838 - core::fmt::write::h6bc2db31dd03e7df 2: 0x7fff929bec44 - <unknown> 3: 0x7fff929d0be4 - <unknown> 4: 0x7fff929d41a4 - <unknown> 5: 0x7fff929d3b50 - std::panicking::default_hook::hc8100c38be18401f 6: 0x7fff8f5a0b6c - <unknown> 7: 0x7fff8f571530 - <unknown> 8: 0x7fff929d50a8 - std::panicking::rust_panic_with_hook::hb454b0dcbf5fc939 9: 0x7fff929d4b38 - <unknown> 10: 0x7fff929d14b0 - std::sys::backtrace::__rust_end_short_backtrace::h03d44392342f1184 11: 0x7fff929d4584 - __rustc[c2610c6383308acc]::rust_begin_unwind 12: 0x7fff8edb9414 - core::panicking::panic_fmt::hf33b6b3da6dfe6bc 13: 0x7fff8edb9278 - core::cell::panic_already_borrowed::h59a581a7b2415564 14: 0x7fff90d7f280 - <rustc_borrowck[ff17395deed2ec8e]::type_check::TypeChecker as rustc_middle[687e2f65837ea9de]::mir::visit::Visitor>::visit_body 15: 0x7fff90ce3f5c - <unknown> 16: 0x7fff90cadac0 - <unknown> 17: 0x7fff90ddbf10 - <unknown> 18: 0x7fff91d2d7a8 - <unknown> 19: 0x7fff91cc9adc - rustc_query_system[9f00eb678419a2a2]::query::plumbing::try_execute_query::<rustc_query_impl[4578bb995ede35ef]::DynamicConfig<rustc_data_structures[7e0816562978da6d]::vec_cache::VecCache<rustc_span[c8935130265359c6]::def_id::LocalDefId, rustc_middle[687e2f65837ea9de]::query::erase::Erased<[u8; 8usize]>, rustc_query_system[9f00eb678419a2a2]::dep_graph::graph::DepNodeIndex>, false, false, false>, rustc_query_impl[4578bb995ede35ef]::plumbing::QueryCtxt, false> 20: 0x7fff91d50d04 - <unknown> 21: 0x7fff914a7690 - <unknown> 22: 0x7fff9251feac - rustc_interface[537fa850fc24b132]::passes::analysis 23: 0x7fff927bf39c - <unknown> 24: 0x7fff9270d844 - rustc_query_system[9f00eb678419a2a2]::query::plumbing::try_execute_query::<rustc_query_impl[4578bb995ede35ef]::DynamicConfig<rustc_query_system[9f00eb678419a2a2]::query::caches::SingleCache<rustc_middle[687e2f65837ea9de]::query::erase::Erased<[u8; 0usize]>>, false, false, false>, rustc_query_impl[4578bb995ede35ef]::plumbing::QueryCtxt, false> 25: 0x7fff927c6238 - <unknown> 26: 0x7fff923b43ac - <unknown> 27: 0x7fff923b0594 - <unknown> 28: 0x7fff923ac588 - std[8587342a5b72dfb4]::sys::backtrace::__rust_begin_short_backtrace::<rustc_interface[537fa850fc24b132]::util::run_in_thread_with_globals<rustc_interface[537fa850fc24b132]::util::run_in_thread_pool_with_globals<rustc_interface[537fa850fc24b132]::interface::run_compiler<(), rustc_driver_impl[d9b811d677f6ce88]::run_compiler::{closure#0}>::{closure#1}, ()>::{closure#0}, ()>::{closure#0}::{closure#0}, ()> 29: 0x7fff923b7c54 - <unknown> 30: 0x7fff929dc5c8 - <unknown> 31: 0x7fff8e2b693c - start_thread 32: 0x7fff8e363908 - __clone3 error: the compiler unexpectedly panicked. this is a bug. note: we would appreciate a bug report: https://github.com/rust-lang/rust/issues/new?labels=C-bug%2C+I-ICE%2C+T-compiler&template=ice.md note: please make sure that you have updated to the latest nightly warning: the ICE couldn't be written to `/usr/share/cargo/registry/hyper-1.6.0/rustc-ice-2025-05-22T02_05_45-332.txt`: Permission denied (os error 13) note: rustc 1.87.0 (17067e9ac 2025-05-09) (Fedora 1.87.0-1.fc43) running on powerpc64le-unknown-linux-gnu note: compiler flags: --crate-type lib -C opt-level=3 -C embed-bitcode=no -C codegen-units=1 -C debuginfo=2 -C opt-level=3 -C debuginfo=2 -C codegen-units=1 -C strip=none -C link-arg=-specs=/usr/lib/rpm/redhat/redhat-package-notes note: some of the compiler flags provided by cargo are hidden query stack during panic: #0 [mir_borrowck] borrow-checking `client::conn::http1::upgrades::<impl at /usr/share/cargo/registry/hyper-1.6.0/src/client/conn/http1.rs:590:5: 595:57>::poll` #1 [analysis] running analysis passes on this crate end of query stack Expected Results: Successful build
A very similar-looking ppc64le-only panic was spotted in Koschei for rust-metrics-exporter-prometheus: https://koschei.fedoraproject.org/build/20568549
There's not only panics, but also straight up segfaults in librust_driver on ppc64le: error: rustc interrupted by SIGSEGV, printing backtrace /lib64/librustc_driver-bbe9c34a8516c373.so(+0xfa6bac) [0x7fff7f1a6bac] linux-vdso64.so.1(__kernel_sigtramp_rt64+0x0) [0x7fff83500484] /lib64/librustc_driver-bbe9c34a8516c373.so(_RNvXs1_NtCslTQcjdwlyaw_14rustc_borrowck10type_checkNtB5_11TypeCheckerNtNtNtCs8YddYaRJ3vC_12rustc_middle3mir5visit7Visitor10visit_body+0xbf00) [0x7fff8095ca60] (...) note: we would appreciate a report at https://github.com/rust-lang/rust help: you can increase rustc's stack size by setting RUST_MIN_STACK=16777216 note: backtrace dumped due to SIGSEGV! resuming signal error: could not compile `tokio-stream` (lib) from https://koschei.fedoraproject.org/build/20573346
1.87.0-1 has been untagged from rawhide and eln, and we hadn't submitted f41 and f42 to bodhi at all yet. Jesus also tried a scratch build with ppc64le PGO disabled, and that build looks better at first glance, but I will investigate more before we try to push out a new build.
I disabled ppc64le PGO in 1.87.0-2.fc43, and so far it looks good in koschei, as well as my own scratch builds for these particular packages: rust-hyper-util: https://koji.fedoraproject.org/koji/taskinfo?taskID=133281625 rust-tokio: https://koji.fedoraproject.org/koji/taskinfo?taskID=133281626 I'm going to leave this open though, because we haven't actually found the underlying bug yet.
I investigated this and found it to be an llvm codegen bug already reported as https://github.com/llvm/llvm-project/issues/138208, and fixed upstream.
The LLVM fix has been backported -- I'll try re-enabling ppc64le PGO and see how these go.
PGO is back on in rust-1.89.0-4.fc44, and these scratch builds passed: rust-hyper-util: https://koji.fedoraproject.org/koji/taskinfo?taskID=136214151 rust-tokio: https://koji.fedoraproject.org/koji/taskinfo?taskID=136214170