Created attachment 1572160 [details] build.log Description of problem: Trying to build Rust code on ppc64le in Koji now gives: ``` libtool: link: gcc -shared -fPIC -DPIC src/lib/.libs/librpmostree_1_la-rpmostree.o src/lib/.libs/librpmostree_1_la-rpmostree-db.o src/lib/.libs/librpmostree_1_la-rpmostree-package.o -Wl,--whole-archive ./.libs/librpmostreepriv.a -Wl,--no-whole-archive -L/builddir/build/BUILD/rpm-ostree-2019.4/libdnf-build/libdnf -ldnf -ljson-glib-1.0 -lostree-1 -lsystemd -lpolkit-gobject-1 -lrpm -lrpmio -lrepo -lsolv -larchive -lgio-2.0 -lgobject-2.0 -lglib-2.0 -lcap /builddir/build/BUILD/rpm-ostree-2019.4/target/release/librpmostree_rust.a -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -pthread -O2 -g -fstack-protector-strong -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mcpu=power8 -mtune=power8 -Wl,-rpath=/usr/lib64/rpm-ostree -Wl,-z -Wl,relro -Wl,--as-needed -Wl,-z -Wl,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -Wl,--push-state -Wl,--no-as-needed -Wl,-ldl -Wl,-lm -Wl,--pop-state -pthread -Wl,-soname -Wl,librpmostree-1.so.1 -o .libs/librpmostree-1.so.1.0.0 BUILDSTDERR: /usr/bin/ld: /usr/lib/gcc/ppc64le-redhat-linux/9/libgcc.a(ppc64-fp.o): in function `__fixdfdi': BUILDSTDERR: (.text+0x510): multiple definition of `__fixdfdi'; /builddir/build/BUILD/rpm-ostree-2019.4/target/release/librpmostree_rust.a(compiler_builtins-cae9ea46cd4dabd1.compiler_builtins.qr7h34vq-cgu.0.rcgu.o):(.text.__fixdfdi+0x0): first defined here BUILDSTDERR: /usr/bin/ld: /usr/lib/gcc/ppc64le-redhat-linux/9/libgcc.a(ppc64-fp.o): in function `__fixsfdi': BUILDSTDERR: (.text+0x5c0): multiple definition of `__fixsfdi'; /builddir/build/BUILD/rpm-ostree-2019.4/target/release/librpmostree_rust.a(compiler_builtins-cae9ea46cd4dabd1.compiler_builtins.qr7h34vq-cgu.0.rcgu.o):(.text.__fixsfdi+0x0): first defined here BUILDSTDERR: /usr/bin/ld: /usr/lib/gcc/ppc64le-redhat-linux/9/libgcc.a(ppc64-fp.o): in function `__fixunsdfsi': BUILDSTDERR: (.text+0x680): multiple definition of `__fixunsdfsi'; /builddir/build/BUILD/rpm-ostree-2019.4/target/release/librpmostree_rust.a(compiler_builtins-cae9ea46cd4dabd1.compiler_builtins.qr7h34vq-cgu.0.rcgu.o):(.text.__fixunsdfsi+0x0): first defined here BUILDSTDERR: /usr/bin/ld: /usr/lib/gcc/ppc64le-redhat-linux/9/libgcc.a(ppc64-fp.o): in function `__fixunssfsi': BUILDSTDERR: (.text+0x6e0): multiple definition of `__fixunssfsi'; /builddir/build/BUILD/rpm-ostree-2019.4/target/release/librpmostree_rust.a(compiler_builtins-cae9ea46cd4dabd1.compiler_builtins.qr7h34vq-cgu.0.rcgu.o):(.text.__fixunssfsi+0x0): first defined here BUILDSTDERR: /usr/bin/ld: /usr/lib/gcc/ppc64le-redhat-linux/9/libgcc.a(ppc64-fp.o): in function `__floatdidf': BUILDSTDERR: (.text+0x800): multiple definition of `__floatdidf'; /builddir/build/BUILD/rpm-ostree-2019.4/target/release/librpmostree_rust.a(compiler_builtins-cae9ea46cd4dabd1.compiler_builtins.qr7h34vq-cgu.0.rcgu.o):(.text.__floatdidf+0x0): first defined here BUILDSTDERR: /usr/bin/ld: /usr/lib/gcc/ppc64le-redhat-linux/9/libgcc.a(ppc64-fp.o): in function `__floatundidf': BUILDSTDERR: (.text+0x840): multiple definition of `__floatundidf'; /builddir/build/BUILD/rpm-ostree-2019.4/target/release/librpmostree_rust.a(compiler_builtins-cae9ea46cd4dabd1.compiler_builtins.qr7h34vq-cgu.0.rcgu.o):(.text.__floatundidf+0x0): first defined here BUILDSTDERR: /usr/bin/ld: /usr/lib/gcc/ppc64le-redhat-linux/9/libgcc.a(ppc64-fp.o): in function `__floatdisf': BUILDSTDERR: (.text+0x880): multiple definition of `__floatdisf'; /builddir/build/BUILD/rpm-ostree-2019.4/target/release/librpmostree_rust.a(compiler_builtins-cae9ea46cd4dabd1.compiler_builtins.qr7h34vq-cgu.0.rcgu.o):(.text.__floatdisf+0x0): first defined here BUILDSTDERR: /usr/bin/ld: /usr/lib/gcc/ppc64le-redhat-linux/9/libgcc.a(ppc64-fp.o): in function `__floatundisf': BUILDSTDERR: (.text+0x900): multiple definition of `__floatundisf'; /builddir/build/BUILD/rpm-ostree-2019.4/target/release/librpmostree_rust.a(compiler_builtins-cae9ea46cd4dabd1.compiler_builtins.qr7h34vq-cgu.0.rcgu.o):(.text.__floatundisf+0x0): first defined here BUILDSTDERR: collect2: error: ld returned 1 exit status ``` Ref: https://kojipkgs.fedoraproject.org//work/tasks/4168/34984168/build.log (Also added as attachment in case the above gets pruned). Version-Release number of selected component (if applicable): gcc-9.1.1-1.fc30.ppc64le rust-1.34.2-1.fc30.ppc64le How reproducible: Always Steps to Reproduce: 1. Try to build rpm-ostree in Koji on ppc64le Actual results: Build failure Expected results: No failure Additional info: Filed this against rust, though it might be a gcc issue. I don't *think* this is an issue in rpm-ostree and is instead likely a regression in the buildroot itself. I actually hit this around two/three weeks ago too, but only on rawhide and not on f30. Unfortunately, it seems like the rawhide build logs have already been pruned, but the f30 one (which succeeded) had: gcc-9.0.1-0.10.fc30.ppc64le rust-1.34.0-1.fc30.ppc64le Ref: https://kojipkgs.fedoraproject.org//packages/rpm-ostree/2019.3.5.g0da9f997/2.fc30/data/logs/ppc64le/root.log So the most obvious difference here is the gcc-9.0 -> gcc-9.1 bump.
A few data points... I can reproduce this on f29, which has older gcc and llvm, so that probably rules those out. rust-1.34.2-1.fc29.ppc64le llvm-libs-7.0.1-4.fc29.ppc64le gcc-8.3.1-2.fc29.ppc64le binutils-2.31.1-24.fc29.ppc64le I have a beta build of 1.35 (release tomorrow!) and that also fails: https://koji.fedoraproject.org/koji/taskinfo?taskID=34876444 rust-1.35.0-0.1.beta.7.fc29.ppc64le However, any of these work: rust-1.34.1-1.fc29.ppc64le rust-1.34.0-1.fc29.ppc64le rust-1.33.0-1.fc29.ppc64le 1.34.1 vs 1.34.2 is really strange, because the *only* difference is the destabilization of Error::type_id. This should make *zero* difference to codegen. I can also reproduce this with upstream rustc, from the current nightly back to 1.32.0 -- but then it works with upstream 1.31.1! Note, upstream ppc64 cargo is broken before 1.35, so I replaced that with our /usr/bin/cargo in each case. (ref: https://github.com/rust-lang/rust/pull/58986) Anyway, it's really baffling that the apparent regression point is different between fedora and upstream.
One thing that might be related to this is that rpm-ostree builds both an executable `/usr/bin/rpm-ostree` as well as a shared library `/usr/lib64/librpmostree-1.so.1.0.0` - and its internal Rust code is built as a static library and linked into both, and the main executable links to the shared library. Now...clearly, we should avoid duplicating the Rust code. It feels like the only solution to this is basically to have the executable do double duty as a shared library; I think that's possible in theory? Another thing that may be related is that we're building the C code with gcc, but Rust obviously uses llvm.
> 1.34.1 vs 1.34.2 is really strange, because the *only* difference is [...] I was wrong about this -- I also started configuring with --set rust.codegen-units-std=1 in that build. I missed this before because I was looking at the rawhide spec differences, where I had already made that change in an earlier build. But this didn't land on f29 until I updated to 1.34.2... Upstream, the official builds added and started using this option in commit b8f977a8a7ab9, which git calls 1.32.0~186^2~2^2~2, so 1.31.1 wouldn't have had this. So, at least there is a smoking gun...
Here are the symbol tables for the objects containing __fixdfdi: gcc-8.3.1-2.fc29.ppc64le/libgcc.a(ppc64-fp.o): Symbol table [17] '.symtab' contains 29 entries: 13 local symbols String table: [18] '.strtab' Num: Value Size Type Bind Vis Ndx Name [0..=12 LOCAL ...] 13: 0000000000000000 0 NOTYPE GLOBAL DEFAULT UNDEF __gcc_qmul 14: 0000000000000000 0 NOTYPE GLOBAL DEFAULT UNDEF __gcc_qsub 15: 00000000000007b0 96 FUNC GLOBAL HIDDEN 1 __floatunditf 16: 0000000000000000 1308 FUNC GLOBAL HIDDEN 1 __fixtfdi 17: 0000000000000000 0 NOTYPE GLOBAL DEFAULT UNDEF .TOC. 18: 0000000000000520 176 FUNC GLOBAL HIDDEN 1 __fixdfdi 19: 00000000000005d0 184 FUNC GLOBAL HIDDEN 1 __fixsfdi 20: 0000000000000690 84 FUNC GLOBAL HIDDEN 1 __fixunsdfsi 21: 00000000000006f0 84 FUNC GLOBAL HIDDEN 1 __fixunssfsi 22: 0000000000000000 0 NOTYPE GLOBAL DEFAULT UNDEF __gcc_qadd 23: 0000000000000750 96 FUNC GLOBAL HIDDEN 1 __floatditf 24: 0000000000000810 56 FUNC GLOBAL HIDDEN 1 __floatdidf 25: 0000000000000850 56 FUNC GLOBAL HIDDEN 1 __floatundidf 26: 0000000000000890 116 FUNC GLOBAL HIDDEN 1 __floatdisf 27: 0000000000000910 92 FUNC GLOBAL HIDDEN 1 __floatundisf 28: 0000000000000970 692 FUNC GLOBAL HIDDEN 1 __fixunstfdi rust-1.34.1-1.fc29.ppc64le/librpmostree_rust.a(compiler_builtins-b350ac5e3f9487d9.compiler_builtins.5cerm65h-cgu.0.rcgu.o): Symbol table [27] '.symtab' contains 51 entries: 27 local symbols String table: [28] '.strtab' Num: Value Size Type Bind Vis Ndx Name [0..=26 LOCAL ...] 27: 0000000000000000 128 FUNC GLOBAL HIDDEN 18 __fixdfdi 28: 0000000000000000 116 FUNC GLOBAL HIDDEN 17 __fixdfsi 29: 0000000000000000 176 FUNC GLOBAL HIDDEN 19 __fixdfti 30: 0000000000000000 140 FUNC GLOBAL HIDDEN 15 __fixsfdi 31: 0000000000000000 144 FUNC GLOBAL HIDDEN 14 __fixsfsi 32: 0000000000000000 188 FUNC GLOBAL HIDDEN 16 __fixsfti 33: 0000000000000000 108 FUNC GLOBAL HIDDEN 24 __fixunsdfdi 34: 0000000000000000 92 FUNC GLOBAL HIDDEN 23 __fixunsdfsi 35: 0000000000000000 152 FUNC GLOBAL HIDDEN 25 __fixunsdfti 36: 0000000000000000 136 FUNC GLOBAL HIDDEN 21 __fixunssfdi 37: 0000000000000000 140 FUNC GLOBAL HIDDEN 20 __fixunssfsi 38: 0000000000000000 168 FUNC GLOBAL HIDDEN 22 __fixunssfti 39: 0000000000000000 204 FUNC GLOBAL HIDDEN 5 __floatdidf 40: 0000000000000000 212 FUNC GLOBAL HIDDEN 4 __floatdisf 41: 0000000000000000 96 FUNC GLOBAL HIDDEN 3 __floatsidf 42: 0000000000000000 208 FUNC GLOBAL HIDDEN 2 __floatsisf 43: 0000000000000000 336 FUNC GLOBAL HIDDEN 7 __floattidf 44: 0000000000000000 344 FUNC GLOBAL HIDDEN 6 __floattisf 45: 0000000000000000 188 FUNC GLOBAL HIDDEN 11 __floatundidf 46: 0000000000000000 196 FUNC GLOBAL HIDDEN 10 __floatundisf 47: 0000000000000000 64 FUNC GLOBAL HIDDEN 9 __floatunsidf 48: 0000000000000000 196 FUNC GLOBAL HIDDEN 8 __floatunsisf 49: 0000000000000000 316 FUNC GLOBAL HIDDEN 13 __floatuntidf 50: 0000000000000000 324 FUNC GLOBAL HIDDEN 12 __floatuntisf rust-1.34.2-1.fc29.ppc64le/librpmostree_rust.a(compiler_builtins-4c38d522dbf7ff8b.compiler_builtins.f061to8e-cgu.0.rcgu.o): Symbol table [368] '.symtab' contains 553 entries: 278 local symbols String table: [369] '.strtab' Num: Value Size Type Bind Vis Ndx Name [0..=277 LOCAL, then a lot more ...] 489: 0000000000000000 136 FUNC GLOBAL HIDDEN 218 __eqdf2 490: 0000000000000000 156 FUNC GLOBAL HIDDEN 215 __eqsf2 491: 0000000000000000 156 FUNC GLOBAL HIDDEN 240 __extendsfdf2 492: 0000000000000000 128 FUNC GLOBAL HIDDEN 205 __fixdfdi 493: 0000000000000000 116 FUNC GLOBAL HIDDEN 204 __fixdfsi 494: 0000000000000000 176 FUNC GLOBAL HIDDEN 206 __fixdfti 495: 0000000000000000 140 FUNC GLOBAL HIDDEN 202 __fixsfdi 496: 0000000000000000 144 FUNC GLOBAL HIDDEN 201 __fixsfsi 497: 0000000000000000 188 FUNC GLOBAL HIDDEN 203 __fixsfti 498: 0000000000000000 108 FUNC GLOBAL HIDDEN 211 __fixunsdfdi 499: 0000000000000000 92 FUNC GLOBAL HIDDEN 210 __fixunsdfsi 500: 0000000000000000 152 FUNC GLOBAL HIDDEN 212 __fixunsdfti 501: 0000000000000000 136 FUNC GLOBAL HIDDEN 208 __fixunssfdi 502: 0000000000000000 140 FUNC GLOBAL HIDDEN 207 __fixunssfsi 503: 0000000000000000 168 FUNC GLOBAL HIDDEN 209 __fixunssfti 504: 0000000000000000 204 FUNC GLOBAL HIDDEN 192 __floatdidf 505: 0000000000000000 212 FUNC GLOBAL HIDDEN 191 __floatdisf 506: 0000000000000000 96 FUNC GLOBAL HIDDEN 190 __floatsidf 507: 0000000000000000 208 FUNC GLOBAL HIDDEN 189 __floatsisf 508: 0000000000000000 336 FUNC GLOBAL HIDDEN 194 __floattidf 509: 0000000000000000 344 FUNC GLOBAL HIDDEN 193 __floattisf 510: 0000000000000000 188 FUNC GLOBAL HIDDEN 198 __floatundidf 511: 0000000000000000 196 FUNC GLOBAL HIDDEN 197 __floatundisf 512: 0000000000000000 64 FUNC GLOBAL HIDDEN 196 __floatunsidf 513: 0000000000000000 196 FUNC GLOBAL HIDDEN 195 __floatunsisf 514: 0000000000000000 316 FUNC GLOBAL HIDDEN 200 __floatuntidf 515: 0000000000000000 324 FUNC GLOBAL HIDDEN 199 __floatuntisf 516: 0000000000000000 128 FUNC GLOBAL HIDDEN 216 __gedf2 517: 0000000000000000 148 FUNC GLOBAL HIDDEN 213 __gesf2 518: 0000000000000000 28 FUNC GLOBAL HIDDEN 331 __gtdf2 [... up to 552]
------- Comment From hannsj_uhl.com 2019-05-24 04:15 EDT------- .
In rust-1.35.0-1.fc31, I filtered rust.codegen-units-std=1 for only non-%power64 arches, and a scratch build of rpm-ostree now succeeds: https://koji.fedoraproject.org/koji/taskinfo?taskID=35032930 Rust 1.35 builds are underway for f30, f29, and epel7 too. I'm going to leave this bug open for tracking purposes though, because I feel there's still a larger issue looming with having these compiler_builtins in a single object, regardless of architecture.
Rust pr70846 will help this a lot, keeping codegen-units for compiler-builtins separated at the module level. There might still need to be changes in compiler-builtins itself to move more symbols into their own module.
Upstream pr73136 changed compiler-builtins to use a huge number of codegen-units, regardless of the setting for the rest of std. This landed in 1.46.0, but I hadn't gotten around to verifying this in our environment yet. In rust-1.52.1-2.fc35 I removed the previous workaround for %power64, and it looks like rpm-ostree still builds fine: https://koji.fedoraproject.org/koji/taskinfo?taskID=69249123