Hide Forgot
I don't know if this is a anaconda issue, a systemd issue, a python issue, a ctypes issue, or something deeper, but lets start here. Since 20211218, rawhide composes have been failing. The Minimal armv7 image fails. For example: https://koji.fedoraproject.org/koji/taskinfo?taskID=80210789 I did a scratch image build and looked more and found that anaconda wasn't starting due to the following traceback: Traceback (most recent call last): constraints.py", line File "/sbin/anaconda", line 183, in <module> from pyanaconda.core import util, constants TYPE_PARTITION, \ File "/usr/lib/python3.10/site-packages/pyanaconda/core/util.py", line 43, in in <module> <module> from pyanaconda.core.configuration.anaconda import conf File "/usr/lib/python3.10/site-packages/pyanaconda/core/configuration/anaconda .py", line 30, in <module> from pyanaconda.core.configuration.storage_constraints import StorageConstra ints in load_ctypes_librar File "/usr/lib/python3.10/site-packages/pyanaconda/core/configuration/storage_constraints.py", line 22, in <module> from pyanaconda.core.storage import DEVICE_TYPE_LVM, DEVICE_TYPE_MD, DEVICE_ TYPE_PARTITION, \ File "/usr/lib/python3.10/site-packages/pyanaconda/core/storage.py", line 22, in <module> from blivet import udev File "/usr/lib/python3.10/site-packages/blivet/udev.py", line 39, in <module> global_udev = pyudev.Context() File "/usr/lib/python3.10/site-packages/pyudev/core.py", line 61, in __init__ self._libudev = load_ctypes_library('udev', SIGNATURES, ERROR_CHECKERS) File "/usr/lib/python3.10/site-packages/pyudev/_ctypeslib/utils.py", line 55, in load_ctypes_library raise ImportError('No library named %s' % name) ImportError: No library named udev Exception ignored in: <function Context.__del__ at 0xb29a67c0> Traceback (most recent call last): File "/usr/lib/python3.10/site-packages/pyudev/core.py", line 65, in __del__ self._libudev.udev_unref(self) AttributeError: 'Context' object has no attribute '_libudev' This is only happening on armv7/32 bit arm as far as I can tell.
Hmm, so the code it's ostensibly failing on just does: from ctypes.util import find_library find_library("udev") So it'd be interesting to know if just doing that on a Rawhide armv7 box fails. The fact that it's apparently happening during some sort of teardown, though, does mean it could be something ickier and...Python-ier. Maybe we should pull in Miro for that angle, CCing. Marking as an accepted blocker as this hits the automatic blocker policy: "Bugs which entirely prevent the composition of one or more of the release-blocking images required to be built for a currently-pending (pre-)release" https://fedoraproject.org/wiki/QA:SOP_blocker_bug_process#Automatic_blockers
So this broke between 20211217.n.1 and 20211219.n.0 it seems (17.n.1 completed, 19.n.0 failed, there doesn't appear to have been an 18.n.0 compose at all). libudev is in systemd-libs, and there was a new systemd build on 2021-12-18, though at first blush the changes don't seem very relevant... https://koji.fedoraproject.org/koji/buildinfo?buildID=1868311
Yeah, they really don't. ;( I suppose it could be a compiler or tool chain issue? I could try untagging all the systemd's since the one in 20211217.n.1 and re-run? It would be nice to know if that fixed it first tho.
Adding zbyszek
I'm not aware of any changes that could be relevant. Upstream we did some tweaks to the build system, but nothing that would touch libudev directly. Downstream also nothing relevant. And abidiff says that there have been no ABI changes in systemd-250. In mock on armv7-test01.fp.o the results are the same, and commands from comment #1 above work just fine. 'from blivet import udev' also works. Hmm, how would one go about doing a "scratch image build" ?
huh, so I tried another scratch image build with: koji image-build Fedora-Minimal-Rawhide 20211225.n.0.kevin f36 "https://kojipkgs.fedoraproject.org/compose/rawhide/Fedora-Rawhide-20211225.n.0/compose/Everything/armhfp/os" --disk-size 4 --distro Fedora-30 --format raw-xz --ksurl "git+https://pagure.io/fedora-kickstarts.git?#1bfe603576516ee1396d89e3e0f4f9113fa5e20a" --kickstart fedora-disk-minimal.ks --release 20211225.n.0.kevin --scratch armhfp But I think we restricted all image builds to admins. ;( also, in order to get the vm console you have to go to the builder and virsh console, which needs root on builders. ;( Anyhow, now the image fails sooner. It can't start dbus and errors out. [Wed Dec 22 23:59:59 2021] systemd[1]: Detected virtualization kvm. [Wed Dec 22 23:59:59 2021] systemd[1]: Detected architecture arm. [Wed Dec 22 23:59:59 2021] systemd[1]: Running in initial RAM disk. [Wed Dec 22 23:59:59 2021] systemd[1]: No hostname configured, using default hostname. [Wed Dec 22 23:59:59 2021] systemd[1]: Hostname set to <fedora>. [Wed Dec 22 23:59:59 2021] systemd[1]: Initializing machine ID from VM UUID. [Wed Dec 22 23:59:59 2021] hrtimer: interrupt took 10982525 ns [Wed Dec 22 23:59:59 2021] systemd-gpt-auto-generator[213]: EFI loader partition unknown, exiting. [Wed Dec 22 23:59:59 2021] systemd-gpt-auto-generator[213]: (The boot loader did not set EFI variable LoaderDevicePartUUID.) [Thu Dec 23 00:00:00 2021] systemd[1]: Queued start job for default target initrd.target. [Thu Dec 23 00:00:00 2021] systemd[1]: Reached target initrd-root-device.target - Initrd Root Device. [Thu Dec 23 00:00:00 2021] systemd[1]: Reached target initrd-usr-fs.target - Initrd /usr File System. [Thu Dec 23 00:00:00 2021] systemd[1]: Reached target slices.target - Slice Units. [Thu Dec 23 00:00:00 2021] systemd[1]: Reached target swap.target - Swaps. [Thu Dec 23 00:00:00 2021] systemd[1]: Reached target timers.target - Timer Units. [Thu Dec 23 00:00:00 2021] systemd[1]: Listening on dbus.socket - D-Bus System Message Bus Socket. [Thu Dec 23 00:00:00 2021] systemd[1]: Listening on iscsid.socket - Open-iSCSI iscsid Socket. [Thu Dec 23 00:00:00 2021] systemd[1]: Listening on iscsiuio.socket - Open-iSCSI iscsiuio Socket. [Thu Dec 23 00:00:00 2021] systemd[1]: Listening on systemd-journald-audit.socket - Journal Audit Soc ket. [Thu Dec 23 00:00:00 2021] systemd[1]: Listening on systemd-journald-dev-log.socket - Journal Socket (/dev/log). [Thu Dec 23 00:00:00 2021] systemd[1]: Listening on systemd-journald.socket - Journal Socket. [Thu Dec 23 00:00:00 2021] systemd[1]: Listening on systemd-udevd-control.socket - udev Control Socke t. [Thu Dec 23 00:00:00 2021] systemd[1]: Listening on systemd-udevd-kernel.socket - udev Kernel Socket. [Thu Dec 23 00:00:00 2021] systemd[1]: Reached target sockets.target - Socket Units. [Thu Dec 23 00:00:00 2021] systemd[1]: Starting kmod-static-nodes.service - Create List of Static Dev ice Nodes... [Thu Dec 23 00:00:00 2021] systemd[1]: Started rngd.service - Hardware RNG Entropy Gatherer Daemon. [Thu Dec 23 00:00:00 2021] systemd[1]: Starting systemd-journald.service - Journal Service... [Thu Dec 23 00:00:00 2021] systemd[1]: systemd-modules-load.service - Load Kernel Modules was skipped because all trigger condition checks failed. [Thu Dec 23 00:00:00 2021] systemd[1]: Starting systemd-sysctl.service - Apply Kernel Variables... [Thu Dec 23 00:00:00 2021] systemd[1]: systemd-sysusers.service - Create System Users was skipped bec ause of a failed condition check (ConditionNeedsUpdate=/etc). [Thu Dec 23 00:00:00 2021] systemd[1]: Starting systemd-vconsole-setup.service - Setup Virtual Consol e... [Thu Dec 23 00:00:00 2021] systemd[1]: Finished kmod-static-nodes.service - Create List of Static Dev ice Nodes. [Thu Dec 23 00:00:00 2021] systemd[1]: systemd-journald.service: Main process exited, code=exited, st atus=127/n/a [Thu Dec 23 00:00:00 2021] systemd[1]: systemd-journald.service: Failed with result 'exit-code'. [Thu Dec 23 00:00:00 2021] systemd[1]: Failed to start systemd-journald.service - Journal Service. ... [Thu Dec 23 00:17:28 2021] systemd[1]: Listening on dbus.socket - D-Bus System Message Bus Socket. [Thu Dec 23 00:17:28 2021] systemd[1]: Starting dbus-broker.service - D-Bus System Message Bus... [Thu Dec 23 00:17:28 2021] systemd[1]: dbus-broker.service: Main process exited, code=exited, status=1/FAILURE [Thu Dec 23 00:17:28 2021] systemd[1]: dbus-broker.service: Failed with result 'exit-code'. [Thu Dec 23 00:17:28 2021] systemd[1]: Failed to start dbus-broker.service - D-Bus System Message Bus. Could this be https://github.com/systemd/systemd/issues/21665 ? perhaps this landed after the initial anaconda/udev thing, but we likely need to fix it before we can see the initial problem again. And I don't understand why it only happens on armv7... ;( Happy to try and gather more debug info...
In the snippet, systemd-journald.service fails first. It runs as root and I don't think it has any dependence on non-root users. So I think this must be something else. dbus-broker also runs as root, and shouldn't really care about sysusers. So I don't think 21665 is related. (Also, dracut didn't install sysusers binary afaik, so for our initrd's those changes are not relevant.) status=127 is strange. I'd say that failure to start systemd-journald and dbus-broker looks like it could be selinux. (N.B.: isn't the new output format nicer to read?)
Exception ignored in: <function Context.__del__ at 0xb29a67c0> Traceback (most recent call last): File "/usr/lib/python3.10/site-packages/pyudev/core.py", line 65, in __del__ self._libudev.udev_unref(self) AttributeError: 'Context' object has no attribute '_libudev' That part is https://github.com/pyudev/pyudev/issues/421 The main issue seems like https://github.com/pyudev/pyudev/issues/83 to me.
https://docs.python.org/3/library/ctypes.html#finding-shared-libraries ays that: > On Linux, find_library() tries to run external programs (/sbin/ldconfig, gcc, objdump and ld) to find the library file. Is it possible that started to happen when one or more tools that find_library() uses under the hood stopped being available and hence it falbacks to another one (or the other way around) and hence hitting the ARM issue as described in https://bugs.python.org/issue13508 ?
So, I think we are dealing with 2 issues here now. The orig pyudev thing and a new systemd-journald doesn't start issue. I filed https://bugzilla.redhat.com/show_bug.cgi?id=2035812 on the systemd-journald / dbus starting issue. Lets keep this for the pyudev issue.
So, on the udev thing... it seems like ldconfig isn't working right: [anaconda root@fedora ~]# python3 Python 3.10.1 (main, Dec 7 2021, 00:00:00) [GCC 11.2.1 20211203 (Red Hat 11.2.1-7)] on linux Type "help", "copyright", "credits" or "license" for more information. >>> from ctypes.util import find_library >>> find_library("udev") >>> [anaconda root@fedora ~]# ldconfig -p [anaconda root@fedora ~]# There's a cache file, but no output from ldconfig -p.... none of gcc, objdump or ld is installed.
Moving to glibc for comment on the ldconfig -p on 32bit arm not working...
I'm trying a new compose with glibc-2.34.9000-33.fc36 untagged to see if thats indeed the issue.
I see you're now stuck in a game of Compose Dependency Whack-A-Mole?
So, untagging zstd-1.5.1-1.fc36 gets us past that bug and systemd boots things fine again. Now we are left with the orig bug here which I think is due to ldconfig -p on armv7 not working, causing anaconda to not be able to import the libudev library via pyudev. I cannot easily untag glibc-2.34.9000-33.fc36 because a number of things have been built with it and they seem to have >= requirements for it. (Althought perhaps thats only firefox?)
Does ldconfig -p show some output if your run ldconfig yourself?
(In reply to Florian Weimer from comment #16) > Does ldconfig -p show some output if your run ldconfig yourself? Nope. If I spin a scratch koji image build, go to the builder, virsh console into the install, go to the second tty: [anaconda root@fedora ~]# ldconfig -p [anaconda root@fedora ~]# [anaconda root@fedora ~]# ldconfig -V ldconfig (GNU libc) 2.34.9000 Copyright (C) 2021 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. Written by Andreas Jaeger.
What about ldconfig -v? Does it find some objects? And does ldconfig -p show anything afterwards? Thanks.
glibc-2.34.9000-32.fc36.armv7hl works fine, glibc-2.34.9000-33.fc36.armv7hl breaks. Tested in fedora-rawhide-armhfp mock on armv7-test02.fedorainfracloud.org )you can do that as well). <mock-chroot> sh-5.1$ ldconfig -p <mock-chroot> sh-5.1$ ldconfig -v ldconfig: Path `/usr/lib' given more than once (from <builtin>:0 and <builtin>:0) /lib: (from <builtin>:0) libxxhash.so.0 -> libxxhash.so.0.8.1 librpmbuild.so.9 -> librpmbuild.so.9.2.0 libopcodes-2.37-22.fc36.so -> libopcodes-2.37-22.fc36.so libctf.so.0 -> libctf.so.0.0.0 libctf-nobfd.so.0 -> libctf-nobfd.so.0.0.0 libbfd-2.37-22.fc36.so -> libbfd-2.37-22.fc36.so libdw.so.1 -> libdw-0.186.so libasm.so.1 -> libasm-0.186.so libdebuginfod.so.1 -> libdebuginfod-0.186.so libdevmapper.so.1.02 -> libdevmapper.so.1.02 libcryptsetup.so.12 -> libcryptsetup.so.12.7.0 libauthselect.so.3 -> libauthselect.so.3.1.2 libpamc.so.0 -> libpamc.so.0.82.1 libpam_misc.so.0 -> libpam_misc.so.0.82.1 libpam.so.0 -> libpam.so.0.85.1 libpwquality.so.1 -> libpwquality.so.1.0.2 librpmio.so.9 -> librpmio.so.9.2.0 libnghttp2.so.14 -> libnghttp2.so.14.21.1 libkmod.so.2 -> libkmod.so.2.3.7 libgcrypt.so.20 -> libgcrypt.so.20.3.4 libverto.so.1 -> libverto.so.1.0.0 libgpg-error.so.0 -> libgpg-error.so.0.32.1 librpm.so.9 -> librpm.so.9.2.0 libnss_resolve.so.2 -> libnss_resolve.so.2 libgomp.so.1 -> libgomp.so.1.0.0 libp11-kit.so.0 -> libp11-kit.so.0.3.0 libgssrpc.so.4 -> libgssrpc.so.4.2 libffi.so.6 -> libffi.so.6.0.2 libeconf.so.0 -> libeconf.so.0.4.0 liblber-2.4.so.2 -> liblber-2.4.so.2.11.7 libnss_myhostname.so.2 -> libnss_myhostname.so.2 libdb-5.3.so -> libdb-5.3.so libkdb5.so.10 -> libkdb5.so.10.0 libbrotlienc.so.1 -> libbrotlienc.so.1.0.9 libbrotlidec.so.1 -> libbrotlidec.so.1.0.9 libbrotlicommon.so.1 -> libbrotlicommon.so.1.0.9 libcrypto.so.3 -> libcrypto.so.3.0.0 libpcreposix.so.0 -> libpcreposix.so.0.0.7 libpkgconf.so.3 -> libpkgconf.so.3.0.0 libargon2.so.1 -> libargon2.so.1 libsasl2.so.3 -> libsasl2.so.3.0.0 libkeyutils.so.1 -> libkeyutils.so.1.9 libgssapi_krb5.so.2 -> libgssapi_krb5.so.2.2 libjson-c.so.5 -> libjson-c.so.5.1.0 libtirpc.so.3 -> libtirpc.so.3.0.0 libgdbm_compat.so.4 -> libgdbm_compat.so.4.0.0 libgdbm.so.6 -> libgdbm.so.6.0.0 libkrb5support.so.0 -> libkrb5support.so.0.1 libudev.so.1 -> libudev.so.1.7.3 p11-kit-trust.so -> libnssckbi.so libseccomp.so.2 -> libseccomp.so.2.5.3 libexpat.so.1 -> libexpat.so.1.8.1 libfdisk.so.1 -> libfdisk.so.1.1.0 libarchive.so.13 -> libarchive.so.13.5.2 libssh.so.4 -> libssh_threads.so.4.8.7 libkrad.so.0 -> libkrad.so.0.0 libssl.so.1.1 -> libssl.so.1.1.1l libcrypto.so.1.1 -> libcrypto.so.1.1.1l libldap_r-2.4.so.2 -> libldap_r-2.4.so.2.11.7 libssl.so.3 -> libssl.so.3.0.0 libpcre.so.1 -> libpcre.so.1.2.13 libsigsegv.so.2 -> libsigsegv.so.2.0.6 libxml2.so.2 -> libxml2.so.2.9.12 libmount.so.1 -> libmount.so.1.1.0 libsystemd.so.0 -> libsystemd.so.0.33.0 libnsl.so.3 -> libnsl.so.3.0.0 libk5crypto.so.3 -> libk5crypto.so.3.1 libutempter.so.0 -> libutempter.so.1.2.1 libcurl.so.4 -> libcurl.so.4.7.0 libkrb5.so.3 -> libkrb5.so.3.3 libblkid.so.1 -> libblkid.so.1.1.0 libsemanage.so.2 -> libsemanage.so.2 libslapi-2.4.so.2 -> libslapi-2.4.so.2.11.7 libselinux.so.1 -> libselinux.so.1 libcrack.so.2 -> libcrack.so.2.9.0 libnss_systemd.so.2 -> libnss_systemd.so.2 libtasn1.so.6 -> libtasn1.so.6.6.2 libidn2.so.0 -> libidn2.so.0.3.7 libmpfr.so.6 -> libmpfr.so.6.1.0 libauparse.so.0 -> libauparse.so.0.0.0 libdrop_ambient.so.0 -> libdrop_ambient.so.0.0.0 libcap-ng.so.0 -> libcap-ng.so.0.0.0 liblz4.so.1 -> liblz4.so.1.9.3 libcom_err.so.2 -> libcom_err.so.2.1 libaudit.so.1 -> libaudit.so.1.0.0 libreadline.so.8 -> libreadline.so.8.1 libhistory.so.8 -> libhistory.so.8.1 libpsx.so.2 -> libpsx.so.2.48 libunistring.so.2 -> libunistring.so.2.1.0 liblua-5.4.so -> liblua-5.4.so liblzma.so.5 -> liblzma.so.5.2.5 libsmartcols.so.1 -> libsmartcols.so.1.1.0 libtinfo.so.6 -> libtinfo.so.6.2 libtic.so.6 -> libtic.so.6.2 libpanelw.so.6 -> libpanelw.so.6.2 libpanel.so.6 -> libpanel.so.6.2 libncursesw.so.6 -> libncursesw.so.6.2 libpcprofile.so -> libpcprofile.so libmemusage.so -> libmemusage.so libutil.so.1 -> libutil.so.1 libthread_db.so.1 -> libthread_db.so.1 librt.so.1 -> librt.so.1 libresolv.so.2 -> libresolv.so.2 libpthread.so.0 -> libpthread.so.0 libnss_files.so.2 -> libnss_files.so.2 libnss_dns.so.2 -> libnss_dns.so.2 libnss_compat.so.2 -> libnss_compat.so.2 libm.so.6 -> libm.so.6 libdl.so.2 -> libdl.so.2 libc_malloc_debug.so.0 -> libc_malloc_debug.so.0 libc.so.6 -> libc.so.6 libanl.so.1 -> libanl.so.1 libSegFault.so -> libSegFault.so libBrokenLocale.so.1 -> libBrokenLocale.so.1 ld-linux-armhf.so.3 -> ld-linux-armhf.so.3 libpsl.so.5 -> libpsl.so.5.3.3 libacl.so.1 -> libacl.so.1.1.2301 libmagic.so.1 -> libmagic.so.1.0.0 libattr.so.1 -> libattr.so.1.1.2501 libsepol.so.2 -> libsepol.so.2 libgmp.so.10 -> libgmp.so.10.4.1 libuuid.so.1 -> libuuid.so.1.3.0 libcap.so.2 -> libcap.so.2.48 libpcre2-posix.so.3 -> libpcre2-posix.so.3.0.1 libzstd.so.1 -> libzstd.so.1.5.0 libelf.so.1 -> libelf-0.186.so libbz2.so.1 -> libbz2.so.1.0.8 libpcre2-8.so.0 -> libpcre2-8.so.0.10.4 libz.so.1 -> libz.so.1.2.11 libncurses.so.6 -> libncurses.so.6.2 libmenuw.so.6 -> libmenuw.so.6.2 libmenu.so.6 -> libmenu.so.6.2 libformw.so.6 -> libformw.so.6.2 libform.so.6 -> libform.so.6.2 libgcc_s.so.1 -> libgcc_s-11-20211203.so.1 libstdc++.so.6 -> libstdc++.so.6.0.29 liblua-5.3.so -> liblua-5.3.so libsqlite3.so.0 -> libsqlite3.so.0.8.6 libpopt.so.0 -> libpopt.so.0.0.1 libcrypt.so.2 -> libcrypt.so.2.0.0 ldconfig: Can't create temporary cache file /etc/ld.so.cache~: Permission denied <mock-chroot> sh-5.1$ ldconfig -p <mock-chroot> sh-5.1$ With root: <mock-chroot> sh-5.1# ldconfig -p <mock-chroot> sh-5.1# ldconfig -v ldconfig: Path `/usr/lib' given more than once (from <builtin>:0 and <builtin>:0) /lib: (from <builtin>:0) libxxhash.so.0 -> libxxhash.so.0.8.1 librpmbuild.so.9 -> librpmbuild.so.9.2.0 libopcodes-2.37-22.fc36.so -> libopcodes-2.37-22.fc36.so libctf.so.0 -> libctf.so.0.0.0 libctf-nobfd.so.0 -> libctf-nobfd.so.0.0.0 libbfd-2.37-22.fc36.so -> libbfd-2.37-22.fc36.so libdw.so.1 -> libdw-0.186.so libasm.so.1 -> libasm-0.186.so libdebuginfod.so.1 -> libdebuginfod-0.186.so libdevmapper.so.1.02 -> libdevmapper.so.1.02 libcryptsetup.so.12 -> libcryptsetup.so.12.7.0 libauthselect.so.3 -> libauthselect.so.3.1.2 libpamc.so.0 -> libpamc.so.0.82.1 libpam_misc.so.0 -> libpam_misc.so.0.82.1 libpam.so.0 -> libpam.so.0.85.1 libpwquality.so.1 -> libpwquality.so.1.0.2 librpmio.so.9 -> librpmio.so.9.2.0 libnghttp2.so.14 -> libnghttp2.so.14.21.1 libkmod.so.2 -> libkmod.so.2.3.7 libgcrypt.so.20 -> libgcrypt.so.20.3.4 libverto.so.1 -> libverto.so.1.0.0 libgpg-error.so.0 -> libgpg-error.so.0.32.1 librpm.so.9 -> librpm.so.9.2.0 libnss_resolve.so.2 -> libnss_resolve.so.2 libgomp.so.1 -> libgomp.so.1.0.0 libp11-kit.so.0 -> libp11-kit.so.0.3.0 libgssrpc.so.4 -> libgssrpc.so.4.2 libffi.so.6 -> libffi.so.6.0.2 libeconf.so.0 -> libeconf.so.0.4.0 liblber-2.4.so.2 -> liblber-2.4.so.2.11.7 libnss_myhostname.so.2 -> libnss_myhostname.so.2 libdb-5.3.so -> libdb-5.3.so libkdb5.so.10 -> libkdb5.so.10.0 libbrotlienc.so.1 -> libbrotlienc.so.1.0.9 libbrotlidec.so.1 -> libbrotlidec.so.1.0.9 libbrotlicommon.so.1 -> libbrotlicommon.so.1.0.9 libcrypto.so.3 -> libcrypto.so.3.0.0 libpcreposix.so.0 -> libpcreposix.so.0.0.7 libpkgconf.so.3 -> libpkgconf.so.3.0.0 libargon2.so.1 -> libargon2.so.1 libsasl2.so.3 -> libsasl2.so.3.0.0 libkeyutils.so.1 -> libkeyutils.so.1.9 libgssapi_krb5.so.2 -> libgssapi_krb5.so.2.2 libjson-c.so.5 -> libjson-c.so.5.1.0 libtirpc.so.3 -> libtirpc.so.3.0.0 libgdbm_compat.so.4 -> libgdbm_compat.so.4.0.0 libgdbm.so.6 -> libgdbm.so.6.0.0 libkrb5support.so.0 -> libkrb5support.so.0.1 libudev.so.1 -> libudev.so.1.7.3 p11-kit-trust.so -> libnssckbi.so libseccomp.so.2 -> libseccomp.so.2.5.3 libexpat.so.1 -> libexpat.so.1.8.1 libfdisk.so.1 -> libfdisk.so.1.1.0 libarchive.so.13 -> libarchive.so.13.5.2 libssh.so.4 -> libssh_threads.so.4.8.7 libkrad.so.0 -> libkrad.so.0.0 libssl.so.1.1 -> libssl.so.1.1.1l libcrypto.so.1.1 -> libcrypto.so.1.1.1l libldap_r-2.4.so.2 -> libldap_r-2.4.so.2.11.7 libssl.so.3 -> libssl.so.3.0.0 libpcre.so.1 -> libpcre.so.1.2.13 libsigsegv.so.2 -> libsigsegv.so.2.0.6 libxml2.so.2 -> libxml2.so.2.9.12 libmount.so.1 -> libmount.so.1.1.0 libsystemd.so.0 -> libsystemd.so.0.33.0 libnsl.so.3 -> libnsl.so.3.0.0 libk5crypto.so.3 -> libk5crypto.so.3.1 libutempter.so.0 -> libutempter.so.1.2.1 libcurl.so.4 -> libcurl.so.4.7.0 libkrb5.so.3 -> libkrb5.so.3.3 libblkid.so.1 -> libblkid.so.1.1.0 libsemanage.so.2 -> libsemanage.so.2 libslapi-2.4.so.2 -> libslapi-2.4.so.2.11.7 libselinux.so.1 -> libselinux.so.1 libcrack.so.2 -> libcrack.so.2.9.0 libnss_systemd.so.2 -> libnss_systemd.so.2 libtasn1.so.6 -> libtasn1.so.6.6.2 libidn2.so.0 -> libidn2.so.0.3.7 libmpfr.so.6 -> libmpfr.so.6.1.0 libauparse.so.0 -> libauparse.so.0.0.0 libdrop_ambient.so.0 -> libdrop_ambient.so.0.0.0 libcap-ng.so.0 -> libcap-ng.so.0.0.0 liblz4.so.1 -> liblz4.so.1.9.3 libcom_err.so.2 -> libcom_err.so.2.1 libaudit.so.1 -> libaudit.so.1.0.0 libreadline.so.8 -> libreadline.so.8.1 libhistory.so.8 -> libhistory.so.8.1 libpsx.so.2 -> libpsx.so.2.48 libunistring.so.2 -> libunistring.so.2.1.0 liblua-5.4.so -> liblua-5.4.so liblzma.so.5 -> liblzma.so.5.2.5 libsmartcols.so.1 -> libsmartcols.so.1.1.0 libtinfo.so.6 -> libtinfo.so.6.2 libtic.so.6 -> libtic.so.6.2 libpanelw.so.6 -> libpanelw.so.6.2 libpanel.so.6 -> libpanel.so.6.2 libncursesw.so.6 -> libncursesw.so.6.2 libpcprofile.so -> libpcprofile.so libmemusage.so -> libmemusage.so libutil.so.1 -> libutil.so.1 libthread_db.so.1 -> libthread_db.so.1 librt.so.1 -> librt.so.1 libresolv.so.2 -> libresolv.so.2 libpthread.so.0 -> libpthread.so.0 libnss_files.so.2 -> libnss_files.so.2 libnss_dns.so.2 -> libnss_dns.so.2 libnss_compat.so.2 -> libnss_compat.so.2 libm.so.6 -> libm.so.6 libdl.so.2 -> libdl.so.2 libc_malloc_debug.so.0 -> libc_malloc_debug.so.0 libc.so.6 -> libc.so.6 libanl.so.1 -> libanl.so.1 libSegFault.so -> libSegFault.so libBrokenLocale.so.1 -> libBrokenLocale.so.1 ld-linux-armhf.so.3 -> ld-linux-armhf.so.3 libpsl.so.5 -> libpsl.so.5.3.3 libacl.so.1 -> libacl.so.1.1.2301 libmagic.so.1 -> libmagic.so.1.0.0 libattr.so.1 -> libattr.so.1.1.2501 libsepol.so.2 -> libsepol.so.2 libgmp.so.10 -> libgmp.so.10.4.1 libuuid.so.1 -> libuuid.so.1.3.0 libcap.so.2 -> libcap.so.2.48 libpcre2-posix.so.3 -> libpcre2-posix.so.3.0.1 libzstd.so.1 -> libzstd.so.1.5.0 libelf.so.1 -> libelf-0.186.so libbz2.so.1 -> libbz2.so.1.0.8 libpcre2-8.so.0 -> libpcre2-8.so.0.10.4 libz.so.1 -> libz.so.1.2.11 libncurses.so.6 -> libncurses.so.6.2 libmenuw.so.6 -> libmenuw.so.6.2 libmenu.so.6 -> libmenu.so.6.2 libformw.so.6 -> libformw.so.6.2 libform.so.6 -> libform.so.6.2 libgcc_s.so.1 -> libgcc_s-11-20211203.so.1 libstdc++.so.6 -> libstdc++.so.6.0.29 liblua-5.3.so -> liblua-5.3.so libsqlite3.so.0 -> libsqlite3.so.0.8.6 libpopt.so.0 -> libpopt.so.0.0.1 libcrypt.so.2 -> libcrypt.so.2.0.0 <mock-chroot> sh-5.1# ldconfig -p <mock-chroot> sh-5.1#
(my laptop)$ strace ldconfig -p ... openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY) = 3 newfstatat(3, "", {st_mode=S_IFREG|0644, st_size=246095, ...}, AT_EMPTY_PATH) = 0 mmap(NULL, 246095, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f638283b000 openat(AT_FDCWD, "/usr/share/locale/en_US.UTF-8/LC_MESSAGES/libc.mo", O_RDONLY) = -1 ENOENT (No such file or directory) openat(AT_FDCWD, "/usr/share/locale/en_US.utf8/LC_MESSAGES/libc.mo", O_RDONLY) = -1 ENOENT (No such file or directory) openat(AT_FDCWD, "/usr/share/locale/en_US/LC_MESSAGES/libc.mo", O_RDONLY) = -1 ENOENT (No such file or directory) openat(AT_FDCWD, "/usr/share/locale/en.UTF-8/LC_MESSAGES/libc.mo", O_RDONLY) = -1 ENOENT (No such file or directory) openat(AT_FDCWD, "/usr/share/locale/en.utf8/LC_MESSAGES/libc.mo", O_RDONLY) = -1 ENOENT (No such file or directory) openat(AT_FDCWD, "/usr/share/locale/en/LC_MESSAGES/libc.mo", O_RDONLY) = -1 ENOENT (No such file or directory) newfstatat(1, "", {st_mode=S_IFIFO|0600, st_size=0, ...}, AT_EMPTY_PATH) = 0 write(1, "4788 libs found in cache `/etc/l"..., 40964788 libs found in cache `/etc/ld.so.cache' p11-kit-trust.so (libc6,x86-64) => /lib64/p11-kit-trust.so lib3mf.so.2 (libc6,x86-64) => /lib64/lib3mf.so.2 libz3.so.4.8 (libc6,x86-64) => /lib64/libz3.so.4.8 libzzipwrap-0.so.13 (libc6,x86-64) => /lib64/libzzipwrap-0.so.13 libzzipmmapped-0.so.13 (libc6,x86-64) => /lib64/libzzipmmapped-0.so.13 libzzipfseeko-0.so.13 (libc6,x86-64) => /lib64/libzzipfseeko-0.so.13 (mock-fedora-rawhide-armh7vl)$ strace ldconfig -p ... openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_LARGEFILE) = 3 statx(3, "", AT_STATX_SYNC_AS_STAT|AT_NO_AUTOMOUNT|AT_EMPTY_PATH, STATX_BASIC_STATS, {stx_mask=STATX_BASIC_STATS|STATX_MNT_ID, stx_attributes=0, stx_mode=S_IFREG|0644, stx_size=14189, ...}) = 0 close(3) = 0 exit_group(0) = ? +++ exited with 0 +++ $ ls -l /etc/ld.so.cache -rw-r--r--. 1 root root 14189 Dec 28 19:14 /etc/ld.so.cache
On i386, I see this for ldconfig -p: openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_LARGEFILE) = 3 statx(3, "", AT_STATX_SYNC_AS_STAT|AT_NO_AUTOMOUNT|AT_EMPTY_PATH, STATX_BASIC_STATS, {stx_mask=STATX_BASIC_STATS|STATX_MNT_ID, stx_attributes=0, stx_mode=S_IFREG|0644, stx_size=6669, ...}) = 0 mmap2(NULL, 0, PROT_READ, MAP_PRIVATE, 3, 0) = -1 EINVAL (Invalid argument) (glibc-2.34.9000-33.fc36.i686) It looks like something goes wrong inside glibc between the statx system call and the struct stat64 result from fstat64.
I'll revert this commit: commit a6d2f948b71adcb5ea395cb04833bc645eab45e6 Author: Adhemerval Zanella <adhemerval.zanella> Date: Thu May 20 14:20:18 2021 -0300 Use LFS and 64 bit time for installed programs (BZ #15333) Raised here upstream: https://sourceware.org/pipermail/libc-alpha/2021-December/134716.html
Many thanks for the quick revert!
I suppose this was fixed.
On the subject of ldconfig -p output being empty on armv7, I found this bug. It was broken with glibc-2.34.9000-33.fc36.armv7hl but it appears to be fixed with glibc-2.34.9000-34.fc36.armv7hl
Sounds like we're OK here now. Re-open if anything remains.