mozjs128-128.11.0-7.fc44 seems to have major problems with missing symbols on (only) aarch64. The tests in https://koji.fedoraproject.org/koji/taskinfo?taskID=136538233 (a gnome-maps build) failed with multiple occurrences of: /usr/bin/gjs: symbol lookup error: /lib64/libgjs.so.0: undefined symbol: __aarch64_ldadd4_acq_rel, version mozjs_128 nirik confirmed that on his aarch64 Rawhide system, after updating to mozjs128-128.11.0-7.fc44 , GNOME won't start. And openQA GNOME tests also all failed in today's Rawhide (they worked in yesterday's, which had older mozjs128). We don't know why this is happening, but it's very similar to a bug from 2020 which was apparently caused by a change in gcc: https://bugzilla.redhat.com/show_bug.cgi?id=1830472 . We noticed a new llvm in Koji and got excited about that for a minute, but that's not it - that llvm is on a side tag. mozjs128-128.11.0-7.fc44 was built with llvm 20.1.8-18.fc44 . the previous working build - mozjs128-128.11.0-5.fc43 - was built with 20.1.8-3.fc43 . I'll take a look at the differences between those llvm builds, for now.
Happy to gather more info if any can help...
Obvious F44 Beta blocker as GNOME doesn't start.
Before anyone digs too deep into this: GNOME doesn't depend on mozjs128 since 49 beta. The reason it does in Fedora is https://bugzilla.redhat.com/show_bug.cgi?id=2390023 (selinux).
Yes, but the problem may well be wider than just mozjs128, if it's a toolchain issue.
I do not understand these packages well, so put a grain of salt on my interpretation here. I do see that libgjs.so.0 requires __aarch64_ldadd4_acq_rel: $ llvm-readelf -aW /var/lib/mock/fedora-rawhide-aarch64/root/usr/lib64/libgjs.so.0.0.0 | grep __aarch64_ldadd4_acq_rel 00000000001be2f8 000000bd00000402 R_AARCH64_JUMP_SLOT 0000000000000000 __aarch64_ldadd4_acq_rel@mozjs_128 + 0 189: 0000000000000000 0 FUNC GLOBAL DEFAULT UND __aarch64_ldadd4_acq_rel@mozjs_128 The version part @mozjs_128 looks suspicious, but I don't see the same dependency on this function in libmozjs-128.so. In my mock environment these are the only libraries with references to this function: $ for f in /var/lib/mock/fedora-rawhide-aarch64/root/usr/lib64/lib*; do if test -f $f && llvm-readelf -aW $f | grep __aarch64_ldadd4_acq_rel > /dev/null; then echo $f; fi; done /var/lib/mock/fedora-rawhide-aarch64/root/usr/lib64/libc.so.6 /var/lib/mock/fedora-rawhide-aarch64/root/usr/lib64/libgjs.so.0 /var/lib/mock/fedora-rawhide-aarch64/root/usr/lib64/libgjs.so.0.0.0 I do agree with Adam this might be a toolchain issue. It might be worth investigating if the libgcc linker script is indeed working as expected.
The bug is definitely in the mozjs128 package. gjs did not change recently. With mozjs128-128.11.0-5.fc43 , things work. With mozjs128-128.11.0-7.fc44 , things are broken. Interestingly, I think mozjs128-128.11.0-7.fc43 is also not broken, so it might be useful to check the differences in the buildroot between the 7.fc43 and 7.fc44 builds.
llvm and gcc builds seem to be identical NV (but different R) between the 7.fc43 and 7.fc44 of mozjs128. glibc is different in Rawhide, though. The fc44 build was with glibc 2.42.9000-3.fc44 . The fc43 build was with 2.42-4.fc43 . That might matter.
Please check https://bodhi.fedoraproject.org/updates/FEDORA-2025-d308a81810 and do not let it go stable if it's also affected.
Trying with: koji tag f43-build-side-117930 mozjs128-128.11.0-7.fc43 && \ koji wait-repo f43-build-side-117930 --build mozjs128-128.11.0-7.fc43 --request && \ koji build f43-build-side-117930 --scratch 'git+https://src.fedoraproject.org/rpms/gnome-maps.git#64dc20e9522ba20fa70bf8337f1c60a8d2f8103c'
https://koji.fedoraproject.org/koji/taskinfo?taskID=136647653 completed successfully
yeah, I mentioned in #c6 that the fc43 build seems OK.
mozjs140 is broken too: /usr/bin/ld: /usr/lib64/libgjs.so: undefined reference to `__aarch64_ldadd8_acq_rel@mozjs_140' /usr/bin/ld: /usr/lib64/libgjs.so: undefined reference to `__aarch64_ldadd4_acq_rel@mozjs_140'
No clue what's wrong here, but as Adam has already mentioned above, the last time we had a similar bug it turned out to be a toolchain problem (bug #1830472). So I'm going to go ahead and guess that llvm is a reasonable component to assign this to.
Untagging is not an option because mozjs140-140.2.0-1.fc44 is the only build for rawhide ever. I had been working on a GNOME megaupdate, but due to interdependencies between packages, I won't be able to submit anything for rawhide until we figure out what's wrong here.
CCing glibc folks for the possible glibc angle (see comment 7).
Untag ticket for mozjs140 - https://pagure.io/releng/issue/12939 . And I checked and yes, the broken mozjs140 build was done with glibc 2.42.9000-3.fc44 , so it's still a viable suspect. If I can find a minute I'll test building glibc 2.42.9000-3 for f43 and then building a mozjs against that glibc, and see if it causes the bug to show up...
(In reply to Michael Catanzaro from comment #14) > I had been working on a GNOME megaupdate, but due to interdependencies > between packages, I won't be able to submit anything for rawhide until we > figure out what's wrong here. Adam has pointed out that, while there's no previous .fc44 build, the last .fc43 build should be perfectly fine, so this is not actually a blocker for anything other than updating mozjs128/140.
(In reply to Adam Williamson from comment #7) > llvm and gcc builds seem to be identical NV (but different R) between the > 7.fc43 and 7.fc44 of mozjs128. glibc is different in Rawhide, though. The > fc44 build was with glibc 2.42.9000-3.fc44 . The fc43 build was with > 2.42-4.fc43 . That might matter. The 2.42.9000-3.fc44 build is the first one that has the %c conformance changes for scanf (“stdio-common: Reject insufficient character data in scanf [BZ #12701]”). But it doesn't look like binutils uses scanf, so I don't see why this would cause symbols to go away. In general, a missing __aarch64_ldadd4_acq_rel@mozjs_128 symbol suggests that something incorrectly applies a * wildcard in a project-specific symbol version file to a compiler helper function. But both libgcc and compiler-rt mark this function as hidden, so it should not be possible to apply a symbol version to it. And it should not be exported from any shared object. However, this seems to be somewhat common: fweimer=> SELECT DISTINCT symboldb.nevra(package), file.name, ed.version FROM symboldb.elf_definition ed JOIn symboldb.file USING (contents_id) JOIN symboldb.package USING (package_id) JOIN symboldb.package_set_member USING (package_id) WHERE ed.name = '__aarch64_ldadd4_acq_rel' ORDER BY 1, 2; nevra | name | version -------------------------------------------------+---------------------------------------------------------+----------- 389-ds-base-2.7.0-5.el9.aarch64 | /usr/lib64/dirsrv/plugins/libentryuuid-plugin.so | 389-ds-base-2.7.0-5.el9.aarch64 | /usr/lib64/dirsrv/plugins/libentryuuid-syntax-plugin.so | 389-ds-base-2.7.0-5.el9.aarch64 | /usr/lib64/dirsrv/plugins/libpwdchan-plugin.so | 389-ds-base-3.1.3-2.fc42.aarch64 | /usr/lib64/dirsrv/plugins/libentryuuid-plugin.so | 389-ds-base-3.1.3-2.fc42.aarch64 | /usr/lib64/dirsrv/plugins/libentryuuid-syntax-plugin.so | 389-ds-base-3.1.3-2.fc42.aarch64 | /usr/lib64/dirsrv/plugins/libpwdchan-plugin.so | 389-ds-base-3.1.3-4.el10.aarch64 | /usr/lib64/dirsrv/plugins/libentryuuid-plugin.so | 389-ds-base-3.1.3-4.el10.aarch64 | /usr/lib64/dirsrv/plugins/libentryuuid-syntax-plugin.so | 389-ds-base-3.1.3-4.el10.aarch64 | /usr/lib64/dirsrv/plugins/libpwdchan-plugin.so | 389-ds-base-3.1.3-8.fc43.aarch64 | /usr/lib64/dirsrv/plugins/libentryuuid-plugin.so | 389-ds-base-3.1.3-8.fc43.aarch64 | /usr/lib64/dirsrv/plugins/libentryuuid-syntax-plugin.so | 389-ds-base-3.1.3-8.fc43.aarch64 | /usr/lib64/dirsrv/plugins/libpwdchan-plugin.so | chromedriver-139.0.7258.138-1.fc43.aarch64 | /usr/lib64/chromium-browser/chromedriver | chromedriver-139.0.7258.154-1.fc42.aarch64 | /usr/lib64/chromium-browser/chromedriver | chromium-139.0.7258.138-1.fc43.aarch64 | /usr/lib64/chromium-browser/chrome_crashpad_handler | chromium-139.0.7258.154-1.fc42.aarch64 | /usr/lib64/chromium-browser/chrome_crashpad_handler | chromium-headless-139.0.7258.138-1.fc43.aarch64 | /usr/lib64/chromium-browser/headless_shell | chromium-headless-139.0.7258.154-1.fc42.aarch64 | /usr/lib64/chromium-browser/headless_shell | kdepim-addons-25.08.0-1.fc42.aarch64 | /usr/lib64/libadblockplugin.so.6.5.0 | kdepim-addons-25.08.0-1.fc43.aarch64 | /usr/lib64/libadblockplugin.so.6.5.0 | libblkio-1.5.0-4.fc43.aarch64 | /usr/lib64/libblkio.so.1.5.0 | mesa-vulkan-drivers-25.0.7-4.el10.aarch64 | /usr/lib64/libvulkan_nouveau.so | mesa-vulkan-drivers-25.1.4-2.fc43.aarch64 | /usr/lib64/libvulkan_nouveau.so | mesa-vulkan-drivers-25.1.7-1.fc42.aarch64 | /usr/lib64/libvulkan_nouveau.so | mozjs115-115.26.0-1.fc42.aarch64 | /usr/lib64/libmozjs-115.so.0.0.0 | mozjs_115 mozjs115-115.26.0-5.fc43.aarch64 | /usr/lib64/libmozjs-115.so.0.0.0 | mozjs_115 mozjs128-128.11.0-1.fc42.aarch64 | /usr/lib64/libmozjs-128.so.0.0.0 | mozjs_128 mozjs128-128.11.0-5.fc43.aarch64 | /usr/lib64/libmozjs-128.so.0.0.0 | mozjs_128 mozjs140-140.1.0-2.fc43.aarch64 | /usr/lib64/libmozjs-140.so.0.0.0 | mozjs_140 mozjs140-140.2.0-1.fc43.aarch64 | /usr/lib64/libmozjs-140.so.0.0.0 | mozjs_140 pdns-recursor-5.2.0-3.fc43.aarch64 | /usr/bin/pdns_recursor | pdns-recursor-5.2.0-3.fc43.aarch64 | /usr/bin/rec_control | suricata-7.0.11-1.fc42.aarch64 | /usr/bin/suricata | suricata-7.0.11-1.fc43.aarch64 | /usr/bin/suricata | (34 rows) This suggests that there is some linker bug.
That seems plausible: binutils also differs between Rawhide and F43. Rawhide has 2.45.50 now and also had 2.45; F43 is on 2.44. The affected mozjs140 and mozjs128 were both built with 2.45-3.fc44. CCing Nick. Nick, the tl;dr: F44 builds of mozjs128 and mozjs144 are missing a bunch of symbols on aarch64 only. Equivalent builds for F43 do not have the same problem. The compiler versions don't differ between the F43 and F44 builds, but glibc and binutils versions do.
Rawhide side tag with binutils-2.45-3.fc44 tagged: mozjs128 scratch build: https://koji.fedoraproject.org/koji/taskinfo?taskID=136784925 mozjs140 scratch build: https://koji.fedoraproject.org/koji/taskinfo?taskID=136784926 At least the mozjs128-128.11.0-7.fc44.aarch64 build does not seem to have __aarch64_ldadd4_acq_rel@@mozjs_128 (checked via nm -D /usr/lib64/libmozjs-128.so.0.0.0)
Huh? What's the point of that? We already know that Rawhide builds with binutils-2.45-3.fc44 are broken: those are exactly the broken builds that the bug is about. It might be interesting to test an *F43* build with binutils-2.45-3.fc44 . I guess it might also be interesting to do a Rawhide build with 2.45.50 in case that happens to fix it.
mea culpa, I assumed 2.45.50 was the potentially faulty one :/
new scratchbuilds in current rawhide (i.e. with 2.45.50): https://koji.fedoraproject.org/koji/taskinfo?taskID=136787172 https://koji.fedoraproject.org/koji/taskinfo?taskID=136787179 new scratchbuilds with binutils-2.44.90-4.fc43 tagged will follow (waitrepo in progress)
new scratchbuilds with binutils-2.44.90-4.fc43 tagged: https://koji.fedoraproject.org/koji/taskinfo?taskID=136787378 https://koji.fedoraproject.org/koji/taskinfo?taskID=136787380
If my nm'ing is correct, I don't see the __aarch64_ldadd4_acq_rel symbol in either one of the mozjs128 builds.
Hmm, and now I look at it, mozjs128-128.11.0-7.fc43 - which I think we've got down as "known good"? - was built with 2.45-1.fc43 . So...maybe it's not binutils? glibc seems to be the thing that really differs between the two, so far. I still need to find a minute to try an F43 build with F44 glibc...
Hmm. So - all in local mock - I built glibc-2.42.9000-3 for F43, then built mozjs128 against it, then built gnome-maps (49~beta-1) against that mozjs128. The build succeeded. So I *think* that indicates the bug isn't caused by glibc-2.42-9000, unless I got something wrong...
Any update here? Firefox ESR is already up to 140.5.0, but mozjs140 hasn't been updated for months.
Turns out this is still broken, and we re-introduced the bug by building and tagging mozjs140-140.6.0-3.fc44 as part of the F44 mass rebuild. Can we please get this fixed so we don't have the footgun lying around any more?
I just did a little test; I reverted the 140.2.0 and 140.6.0 commits and rebuilt with current Rawhide build chain. So it's basically the same repo state as 140.1.0-2.fc43 , but built with current toolchain. That build works fine. So the issue here really was introduced between 140.1.0 and 140.2.0 somehow, it's not a toolchain issue, it seems. I'm going to commit those reversions to dist-git so the footgun is gone, leaving dist-git in a state where, if a build happens, it'll be a working one. Please only re-commit 140.6.0 or later once it's actually fixed.
The revert also reverted the package to use Python 3.13. I opened https://src.fedoraproject.org/rpms/mozjs140/pull-request/2 to use the current Python version instead, but I'd like to make sure that this isn't the cause of the problem before we ship it.
I'd be surprised, but I can test a scratch build with that PR and see.
Huh. It *did* fail. https://src.fedoraproject.org/rpms/mozjs140/pull-request/2#comment-303219
(In reply to Adam Williamson from comment #30) > I just did a little test; I reverted the 140.2.0 and 140.6.0 commits and > rebuilt with current Rawhide build chain. So it's basically the same repo > state as 140.1.0-2.fc43 , but built with current toolchain. That build works > fine. So the issue here really was introduced between 140.1.0 and 140.2.0 > somehow, it's not a toolchain issue, it seems. Interestingly, in your scratchbuild, as well as in a fresh rawhide scratchbuild of mozjs140 rawhide branch, I cannot see the symbol: $ nm -D usr/lib64/libmozjs-140.so.0 | grep __aarch64_ (is empty)
Adam, please note that f43 had mozjs140-140.2.0-1.fc43 for quite a few months, without any issues, so the troubles can't solely come from 140.1 > 140.2.
So, it seems the revert made no difference: https://src.fedoraproject.org/rpms/mozjs140/pull-request/2#comment-303344
Yes, sorry, I think I did a bad test and injected some confusion. I forgot openQA staging was checked out on a special branch for jforbes to do a kernel bisect. On that branch I'd disabled the "make sure the test actually installed the updated packages" check, for...reasons (it requires networking and networking was broken on some of the kernel builds). It looks like the tests I did of my scratch build, that passed, didn't actually install the scratch build packages, so should have failed that check, but since it was turned off, I didn't notice :( My bad, sorry.
I see this is using clang + ld.bfd. Nick, are you aware of any issues in ld.bfd that could cause a missing dependency on __aarch64_ldadd4_acq_rel? See Florian's comment 18 for more details.
So if I take the mozjs140 build from the mass rebuild (without the symbols) and rebuild gjs aginst that, it no longer uses those symbols. I did that in a side tag. $ koji tag f44-build-side-126897 mozjs140-140.6.0-3.fc44 $ koji wait-repo f44-build-side-126897 --build mozjs140-140.6.0-3.fc44 --request && koji build f44-build-side-126897 --fail-fast --nowait --scratch 'git+https://src.fedoraproject.org/rpms/gjs.git#HEAD' --arch-override=aarch64 ... https://koji.fedoraproject.org/koji/taskinfo?taskID=141447119 Before (old mozjs140 with the symbols, old gjs rebuilt against that): <mock-chroot> sh-5.3# rpm -q gjs mozjs140 gjs-1.86.0-1.fc44.aarch64 mozjs140-140.1.0-2.fc43.aarch64 <mock-chroot> sh-5.3# gjs gjs> undefined ^D $ llvm-readelf -aW /var/lib/mock/fedora-rawhide-aarch64/root/usr/lib64/libgjs.so.0.0.0 | grep __aarch64_ldadd4 00000000001bdde0 0000003600000402 R_AARCH64_JUMP_SLOT 0000000000000000 __aarch64_ldadd4_acq_rel@mozjs_140 + 0 54: 0000000000000000 0 FUNC GLOBAL DEFAULT UND __aarch64_ldadd4_acq_rel@mozjs_140 $ nm -D /var/lib/mock/fedora-rawhide-aarch64/root/usr/lib64/libmozjs-140.so.0 | grep __aarch64_ldadd4 0000000000d8b040 W __aarch64_ldadd4_acq_rel@@mozjs_140 0000000000d8ac6c W __aarch64_ldadd4_rel@@mozjs_140 0000000000d8b150 W __aarch64_ldadd4_relax@@mozjs_140 With only mozjs140 rebuilt and updated: <mock-chroot> sh-5.3# rpm -q gjs mozjs140 gjs-1.86.0-1.fc44.aarch64 mozjs140-140.6.0-3.fc44.aarch64 <mock-chroot> sh-5.3# gjs /usr/bin/gjs: symbol lookup error: /lib64/libgjs.so.0: undefined symbol: __aarch64_ldadd4_acq_rel, version mozjs_140 $ nm -D /var/lib/mock/fedora-rawhide-aarch64/root/usr/lib64/libmozjs-140.so.0 | grep __aarch64_ldadd4 (empty) And after installing the gjs form my side tag: <mock-chroot> sh-5.3# rpm -q gjs mozjs140 gjs-1.87.1-1.fc44.aarch64 mozjs140-140.6.0-3.fc44.aarch64 <mock-chroot> sh-5.3# gjs gjs> undefined ^D $ llvm-readelf -aW /var/lib/mock/fedora-rawhide-aarch64/root/usr/lib64/libgjs.so.0.0.0 | grep __aarch64_ldadd4 (empty) $ nm -D /var/lib/mock/fedora-rawhide-aarch64/root/usr/lib64/libmozjs-140.so.0 | grep __aarch64_ldadd4 (empty)
(I noticed I updated gjs from 1.86.0 to 1.87.1 while testing this, but retesting with the actual build of 1.87.1 from Koji, no difference.)
I think 1.87.1 is part of a pending GNOME 50 mass update, for testing in openQA / on real systems (Kevin has an aarch64 GNOME system he can reproduce the issue on) we probably need a 1.86.0 build...I can do that, though.
gjs 1.86.0 fails to build (likely due to gcc 16), so we would need to revert the update to 1.87.1 in dist git, apply a fix for 1.86.0 and then build that one. I don't think it's worth it. Either we tag the new mozjs140 build from the mass rebuild to the gnome side tag and let them rebuild gjs once more, or we wait for them to ship their update and deal with this after. I believe we should not mix 2 things together, and this is certainly not urgent, so I'd wait. I am also assigning this back to mozjs140, as this is where this will be dealt with, not in llvm or binutils.
Running side tag chainbuilds of mozj140 + gjs on f45, f44, eln. If the f44 one succeeds, I'll fast-forward the f44 branch to match rawhide. ---- Not sure if it makes sense to touch mozj128. cjs depends on it and might need a rebuild as well :/
FEDORA-2026-8474c8ca5e (gjs-1.87.2-3.fc45 and mozjs140-140.6.0-4.fc45) has been submitted as an update to Fedora 45. https://bodhi.fedoraproject.org/updates/FEDORA-2026-8474c8ca5e
FEDORA-2026-f8a6ad4578 (gjs-1.87.2-3.eln155 and mozjs140-140.6.0-4.eln155) has been submitted as an update to Fedora ELN. https://bodhi.fedoraproject.org/updates/FEDORA-2026-f8a6ad4578
FEDORA-2026-c7e2662ab3 (gjs-1.87.2-3.fc44 and mozjs140-140.6.0-4.fc44) has been submitted as an update to Fedora 44. https://bodhi.fedoraproject.org/updates/FEDORA-2026-c7e2662ab3
FEDORA-2026-f8a6ad4578 (gjs-1.87.2-3.eln155 and mozjs140-140.6.0-4.eln155) has been pushed to the Fedora ELN stable repository. If problem still persists, please make note of it in this bug report.
FEDORA-2026-8474c8ca5e (gjs-1.87.2-3.fc45 and mozjs140-140.6.0-4.fc45) has been pushed to the Fedora 45 stable repository. If problem still persists, please make note of it in this bug report.
FEDORA-2026-c7e2662ab3 (gjs-1.87.2-3.fc44 and mozjs140-140.6.0-4.fc44) has been pushed to the Fedora 44 stable repository. If problem still persists, please make note of it in this bug report.
This seems to have reached F42 now: https://bodhi.fedoraproject.org/updates/FEDORA-2026-0f9776bbb8 I'll see if I can fix it up.